[PATCH for-10.0 v2 00/54] accel/tcg: Convert victim tlb to IntervalTree

[PATCH for-10.0 v2 00/54] accel/tcg: Convert victim tlb to IntervalTree

[Richard Henderson]

v1: 20241009150855.804605-1-richard.henderson@linaro.org

The initial idea was: how much can we do with an intelligent data
structure for the same cost as a linear search through an array?


r~


Richard Henderson (54):
  util/interval-tree: Introduce interval_tree_free_nodes
  accel/tcg: Split out tlbfast_flush_locked
  accel/tcg: Split out tlbfast_{index,entry}
  accel/tcg: Split out tlbfast_flush_range_locked
  accel/tcg: Fix flags usage in mmu_lookup1, atomic_mmu_lookup
  accel/tcg: Assert non-zero length in tlb_flush_range_by_mmuidx*
  accel/tcg: Assert bits in range in tlb_flush_range_by_mmuidx*
  accel/tcg: Flush entire tlb when a masked range wraps
  accel/tcg: Add IntervalTreeRoot to CPUTLBDesc
  accel/tcg: Populate IntervalTree in tlb_set_page_full
  accel/tcg: Remove IntervalTree entry in tlb_flush_page_locked
  accel/tcg: Remove IntervalTree entries in tlb_flush_range_locked
  accel/tcg: Process IntervalTree entries in tlb_reset_dirty
  accel/tcg: Process IntervalTree entries in tlb_set_dirty
  accel/tcg: Use tlb_hit_page in victim_tlb_hit
  accel/tcg: Pass full addr to victim_tlb_hit
  accel/tcg: Replace victim_tlb_hit with tlbtree_hit
  accel/tcg: Remove the victim tlb
  accel/tcg: Remove tlb_n_used_entries_inc
  include/exec/tlb-common: Move CPUTLBEntryFull from hw/core/cpu.h
  accel/tcg: Delay plugin adjustment in probe_access_internal
  accel/tcg: Call cpu_ld*_code_mmu from cpu_ld*_code
  accel/tcg: Check original prot bits for read in atomic_mmu_lookup
  accel/tcg: Preserve tlb flags in tlb_set_compare
  accel/tcg: Return CPUTLBEntryFull not pointer in probe_access_full_mmu
  accel/tcg: Return CPUTLBEntryFull not pointer in probe_access_full
  accel/tcg: Return CPUTLBEntryFull not pointer in probe_access_internal
  accel/tcg: Introduce tlb_lookup
  accel/tcg: Partially unify MMULookupPageData and TLBLookupOutput
  accel/tcg: Merge mmu_lookup1 into mmu_lookup
  accel/tcg: Always use IntervalTree for code lookups
  accel/tcg: Link CPUTLBEntry to CPUTLBEntryTree
  accel/tcg: Remove CPUTLBDesc.fulltlb
  target/alpha: Convert to TCGCPUOps.tlb_fill_align
  target/avr: Convert to TCGCPUOps.tlb_fill_align
  target/i386: Convert to TCGCPUOps.tlb_fill_align
  target/loongarch: Convert to TCGCPUOps.tlb_fill_align
  target/m68k: Convert to TCGCPUOps.tlb_fill_align
  target/m68k: Do not call tlb_set_page in helper_ptest
  target/microblaze: Convert to TCGCPUOps.tlb_fill_align
  target/mips: Convert to TCGCPUOps.tlb_fill_align
  target/openrisc: Convert to TCGCPUOps.tlb_fill_align
  target/ppc: Convert to TCGCPUOps.tlb_fill_align
  target/riscv: Convert to TCGCPUOps.tlb_fill_align
  target/rx: Convert to TCGCPUOps.tlb_fill_align
  target/s390x: Convert to TCGCPUOps.tlb_fill_align
  target/sh4: Convert to TCGCPUOps.tlb_fill_align
  target/sparc: Convert to TCGCPUOps.tlb_fill_align
  target/tricore: Convert to TCGCPUOps.tlb_fill_align
  target/xtensa: Convert to TCGCPUOps.tlb_fill_align
  accel/tcg: Drop TCGCPUOps.tlb_fill
  accel/tcg: Unexport tlb_set_page*
  accel/tcg: Merge tlb_fill_align into callers
  accel/tcg: Return CPUTLBEntryTree from tlb_set_page_full

 include/exec/cpu-all.h               |   3 +
 include/exec/exec-all.h              |  65 +-
 include/exec/tlb-common.h            |  68 +-
 include/hw/core/cpu.h                |  75 +-
 include/hw/core/tcg-cpu-ops.h        |  10 -
 include/qemu/interval-tree.h         |  11 +
 target/alpha/cpu.h                   |   6 +-
 target/avr/cpu.h                     |   7 +-
 target/i386/tcg/helper-tcg.h         |   6 +-
 target/loongarch/internals.h         |   7 +-
 target/m68k/cpu.h                    |   7 +-
 target/microblaze/cpu.h              |   7 +-
 target/mips/tcg/tcg-internal.h       |   6 +-
 target/openrisc/cpu.h                |   8 +-
 target/ppc/internal.h                |   7 +-
 target/riscv/cpu.h                   |   8 +-
 target/s390x/s390x-internal.h        |   7 +-
 target/sh4/cpu.h                     |   8 +-
 target/sparc/cpu.h                   |   8 +-
 target/tricore/cpu.h                 |   7 +-
 target/xtensa/cpu.h                  |   8 +-
 accel/tcg/cputlb.c                   | 994 +++++++++++++--------------
 target/alpha/cpu.c                   |   2 +-
 target/alpha/helper.c                |  23 +-
 target/arm/ptw.c                     |  10 +-
 target/arm/tcg/helper-a64.c          |   4 +-
 target/arm/tcg/mte_helper.c          |  15 +-
 target/arm/tcg/sve_helper.c          |   6 +-
 target/avr/cpu.c                     |   2 +-
 target/avr/helper.c                  |  19 +-
 target/i386/tcg/sysemu/excp_helper.c |  36 +-
 target/i386/tcg/tcg-cpu.c            |   2 +-
 target/loongarch/cpu.c               |   2 +-
 target/loongarch/tcg/tlb_helper.c    |  17 +-
 target/m68k/cpu.c                    |   2 +-
 target/m68k/helper.c                 |  32 +-
 target/microblaze/cpu.c              |   2 +-
 target/microblaze/helper.c           |  33 +-
 target/mips/cpu.c                    |   2 +-
 target/mips/tcg/sysemu/tlb_helper.c  |  29 +-
 target/openrisc/cpu.c                |   2 +-
 target/openrisc/mmu.c                |  39 +-
 target/ppc/cpu_init.c                |   2 +-
 target/ppc/mmu_helper.c              |  21 +-
 target/riscv/cpu_helper.c            |  22 +-
 target/riscv/tcg/tcg-cpu.c           |   2 +-
 target/rx/cpu.c                      |  19 +-
 target/s390x/cpu.c                   |   4 +-
 target/s390x/tcg/excp_helper.c       |  23 +-
 target/sh4/cpu.c                     |   2 +-
 target/sh4/helper.c                  |  24 +-
 target/sparc/cpu.c                   |   2 +-
 target/sparc/mmu_helper.c            |  44 +-
 target/tricore/cpu.c                 |   2 +-
 target/tricore/helper.c              |  19 +-
 target/xtensa/cpu.c                  |   2 +-
 target/xtensa/helper.c               |  28 +-
 util/interval-tree.c                 |  20 +
 util/selfmap.c                       |  13 +-
 59 files changed, 938 insertions(+), 923 deletions(-)

-- 
2.43.0

[PATCH v2 01/54] util/interval-tree: Introduce interval_tree_free_nodes

[Richard Henderson]

Provide a general-purpose release-all-nodes operation, that allows
for the IntervalTreeNode to be embeded within a larger structure.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 include/qemu/interval-tree.h | 11 +++++++++++
 util/interval-tree.c         | 20 ++++++++++++++++++++
 util/selfmap.c               | 13 +------------
 3 files changed, 32 insertions(+), 12 deletions(-)

diff --git a/include/qemu/interval-tree.h b/include/qemu/interval-tree.h
index 25006debe8..d90ea6d17f 100644
--- a/include/qemu/interval-tree.h
+++ b/include/qemu/interval-tree.h
@@ -96,4 +96,15 @@ IntervalTreeNode *interval_tree_iter_first(IntervalTreeRoot *root,
 IntervalTreeNode *interval_tree_iter_next(IntervalTreeNode *node,
                                           uint64_t start, uint64_t last);
 
+/**
+ * interval_tree_free_nodes:
+ * @root: root of the tree
+ * @it_offset: offset from outermost type to IntervalTreeNode
+ *
+ * Free, via g_free, all nodes under @root.  IntervalTreeNode may
+ * not be the true type of the nodes allocated; @it_offset gives
+ * the offset from the outermost type to the IntervalTreeNode member.
+ */
+void interval_tree_free_nodes(IntervalTreeRoot *root, size_t it_offset);
+
 #endif /* QEMU_INTERVAL_TREE_H */
diff --git a/util/interval-tree.c b/util/interval-tree.c
index 53465182e6..663d3ec222 100644
--- a/util/interval-tree.c
+++ b/util/interval-tree.c
@@ -639,6 +639,16 @@ static void rb_erase_augmented_cached(RBNode *node, RBRootLeftCached *root,
     rb_erase_augmented(node, &root->rb_root, augment);
 }
 
+static void rb_node_free(RBNode *rb, size_t rb_offset)
+{
+    if (rb->rb_left) {
+        rb_node_free(rb->rb_left, rb_offset);
+    }
+    if (rb->rb_right) {
+        rb_node_free(rb->rb_right, rb_offset);
+    }
+    g_free((void *)rb - rb_offset);
+}
 
 /*
  * Interval trees.
@@ -870,6 +880,16 @@ IntervalTreeNode *interval_tree_iter_next(IntervalTreeNode *node,
     }
 }
 
+void interval_tree_free_nodes(IntervalTreeRoot *root, size_t it_offset)
+{
+    if (root && root->rb_root.rb_node) {
+        rb_node_free(root->rb_root.rb_node,
+                     it_offset + offsetof(IntervalTreeNode, rb));
+        root->rb_root.rb_node = NULL;
+        root->rb_leftmost = NULL;
+    }
+}
+
 /* Occasionally useful for calling from within the debugger. */
 #if 0
 static void debug_interval_tree_int(IntervalTreeNode *node,
diff --git a/util/selfmap.c b/util/selfmap.c
index 483cb617e2..d2b86da301 100644
--- a/util/selfmap.c
+++ b/util/selfmap.c
@@ -87,23 +87,12 @@ IntervalTreeRoot *read_self_maps(void)
  * @root: an interval tree
  *
  * Free a tree of MapInfo structures.
- * Since we allocated each MapInfo in one chunk, we need not consider the
- * contents and can simply free each RBNode.
  */
 
-static void free_rbnode(RBNode *n)
-{
-    if (n) {
-        free_rbnode(n->rb_left);
-        free_rbnode(n->rb_right);
-        g_free(n);
-    }
-}
-
 void free_self_maps(IntervalTreeRoot *root)
 {
     if (root) {
-        free_rbnode(root->rb_root.rb_node);
+        interval_tree_free_nodes(root, offsetof(MapInfo, itree));
         g_free(root);
     }
 }
-- 
2.43.0

[PATCH v2 02/54] accel/tcg: Split out tlbfast_flush_locked

[Richard Henderson]

We will have a need to flush only the "fast" portion
of the tlb, allowing re-fill from the "full" portion.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index b76a4eac4e..c1838412e8 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -284,13 +284,18 @@ static void tlb_mmu_resize_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast,
     }
 }
 
-static void tlb_mmu_flush_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast)
+static void tlbfast_flush_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast)
 {
     desc->n_used_entries = 0;
+    memset(fast->table, -1, sizeof_tlb(fast));
+}
+
+static void tlb_mmu_flush_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast)
+{
+    tlbfast_flush_locked(desc, fast);
     desc->large_page_addr = -1;
     desc->large_page_mask = -1;
     desc->vindex = 0;
-    memset(fast->table, -1, sizeof_tlb(fast));
     memset(desc->vtable, -1, sizeof(desc->vtable));
 }
 
-- 
2.43.0

[PATCH v2 03/54] accel/tcg: Split out tlbfast_{index,entry}

[Richard Henderson]

Often we already have the CPUTLBDescFast structure pointer.
Allows future code simplification.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index c1838412e8..e37af24525 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -131,20 +131,28 @@ static inline uint64_t tlb_addr_write(const CPUTLBEntry *entry)
     return tlb_read_idx(entry, MMU_DATA_STORE);
 }
 
+static inline uintptr_t tlbfast_index(CPUTLBDescFast *fast, vaddr addr)
+{
+    return (addr >> TARGET_PAGE_BITS) & (fast->mask >> CPU_TLB_ENTRY_BITS);
+}
+
+static inline CPUTLBEntry *tlbfast_entry(CPUTLBDescFast *fast, vaddr addr)
+{
+    return fast->table + tlbfast_index(fast, addr);
+}
+
 /* Find the TLB index corresponding to the mmu_idx + address pair.  */
 static inline uintptr_t tlb_index(CPUState *cpu, uintptr_t mmu_idx,
                                   vaddr addr)
 {
-    uintptr_t size_mask = cpu->neg.tlb.f[mmu_idx].mask >> CPU_TLB_ENTRY_BITS;
-
-    return (addr >> TARGET_PAGE_BITS) & size_mask;
+    return tlbfast_index(&cpu->neg.tlb.f[mmu_idx], addr);
 }
 
 /* Find the TLB entry corresponding to the mmu_idx + address pair.  */
 static inline CPUTLBEntry *tlb_entry(CPUState *cpu, uintptr_t mmu_idx,
                                      vaddr addr)
 {
-    return &cpu->neg.tlb.f[mmu_idx].table[tlb_index(cpu, mmu_idx, addr)];
+    return tlbfast_entry(&cpu->neg.tlb.f[mmu_idx], addr);
 }
 
 static void tlb_window_reset(CPUTLBDesc *desc, int64_t ns,
-- 
2.43.0

[PATCH v2 04/54] accel/tcg: Split out tlbfast_flush_range_locked

[Richard Henderson]

While this may at present be overly complicated for use
by single page flushes, do so with the expectation that
this will eventually allow simplification of large pages.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 66 +++++++++++++++++++++++-----------------------
 1 file changed, 33 insertions(+), 33 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index e37af24525..46fa0ae802 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -493,11 +493,6 @@ static bool tlb_flush_entry_mask_locked(CPUTLBEntry *tlb_entry,
     return false;
 }
 
-static inline bool tlb_flush_entry_locked(CPUTLBEntry *tlb_entry, vaddr page)
-{
-    return tlb_flush_entry_mask_locked(tlb_entry, page, -1);
-}
-
 /* Called with tlb_c.lock held */
 static void tlb_flush_vtlb_page_mask_locked(CPUState *cpu, int mmu_idx,
                                             vaddr page,
@@ -520,10 +515,37 @@ static inline void tlb_flush_vtlb_page_locked(CPUState *cpu, int mmu_idx,
     tlb_flush_vtlb_page_mask_locked(cpu, mmu_idx, page, -1);
 }
 
+static void tlbfast_flush_range_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast,
+                                       vaddr addr, vaddr len, vaddr mask)
+{
+    /*
+     * If @mask is smaller than the tlb size, there may be multiple entries
+     * within the TLB; for now, just flush the entire TLB.
+     * Otherwise all addresses that match under @mask hit the same TLB entry.
+     *
+     * If @len is larger than the tlb size, then it will take longer to
+     * test all of the entries in the TLB than it will to flush it all.
+     */
+    if (mask < fast->mask || len > fast->mask) {
+        tlbfast_flush_locked(desc, fast);
+        return;
+    }
+
+    for (vaddr i = 0; i < len; i += TARGET_PAGE_SIZE) {
+        vaddr page = addr + i;
+        CPUTLBEntry *entry = tlbfast_entry(fast, page);
+
+        if (tlb_flush_entry_mask_locked(entry, page, mask)) {
+            desc->n_used_entries--;
+        }
+    }
+}
+
 static void tlb_flush_page_locked(CPUState *cpu, int midx, vaddr page)
 {
-    vaddr lp_addr = cpu->neg.tlb.d[midx].large_page_addr;
-    vaddr lp_mask = cpu->neg.tlb.d[midx].large_page_mask;
+    CPUTLBDesc *desc = &cpu->neg.tlb.d[midx];
+    vaddr lp_addr = desc->large_page_addr;
+    vaddr lp_mask = desc->large_page_mask;
 
     /* Check if we need to flush due to large pages.  */
     if ((page & lp_mask) == lp_addr) {
@@ -532,9 +554,8 @@ static void tlb_flush_page_locked(CPUState *cpu, int midx, vaddr page)
                   midx, lp_addr, lp_mask);
         tlb_flush_one_mmuidx_locked(cpu, midx, get_clock_realtime());
     } else {
-        if (tlb_flush_entry_locked(tlb_entry(cpu, midx, page), page)) {
-            tlb_n_used_entries_dec(cpu, midx);
-        }
+        tlbfast_flush_range_locked(desc, &cpu->neg.tlb.f[midx],
+                                   page, TARGET_PAGE_SIZE, -1);
         tlb_flush_vtlb_page_locked(cpu, midx, page);
     }
 }
@@ -689,24 +710,6 @@ static void tlb_flush_range_locked(CPUState *cpu, int midx,
     CPUTLBDescFast *f = &cpu->neg.tlb.f[midx];
     vaddr mask = MAKE_64BIT_MASK(0, bits);
 
-    /*
-     * If @bits is smaller than the tlb size, there may be multiple entries
-     * within the TLB; otherwise all addresses that match under @mask hit
-     * the same TLB entry.
-     * TODO: Perhaps allow bits to be a few bits less than the size.
-     * For now, just flush the entire TLB.
-     *
-     * If @len is larger than the tlb size, then it will take longer to
-     * test all of the entries in the TLB than it will to flush it all.
-     */
-    if (mask < f->mask || len > f->mask) {
-        tlb_debug("forcing full flush midx %d ("
-                  "%016" VADDR_PRIx "/%016" VADDR_PRIx "+%016" VADDR_PRIx ")\n",
-                  midx, addr, mask, len);
-        tlb_flush_one_mmuidx_locked(cpu, midx, get_clock_realtime());
-        return;
-    }
-
     /*
      * Check if we need to flush due to large pages.
      * Because large_page_mask contains all 1's from the msb,
@@ -720,13 +723,10 @@ static void tlb_flush_range_locked(CPUState *cpu, int midx,
         return;
     }
 
+    tlbfast_flush_range_locked(d, f, addr, len, mask);
+
     for (vaddr i = 0; i < len; i += TARGET_PAGE_SIZE) {
         vaddr page = addr + i;
-        CPUTLBEntry *entry = tlb_entry(cpu, midx, page);
-
-        if (tlb_flush_entry_mask_locked(entry, page, mask)) {
-            tlb_n_used_entries_dec(cpu, midx);
-        }
         tlb_flush_vtlb_page_mask_locked(cpu, midx, page, mask);
     }
 }
-- 
2.43.0

[PATCH v2 05/54] accel/tcg: Fix flags usage in mmu_lookup1, atomic_mmu_lookup

[Richard Henderson]

The INVALID bit should only be auto-cleared when we have
just called tlb_fill, not along the victim_tlb_hit path.

In atomic_mmu_lookup, rename tlb_addr to flags, as that
is what we're actually carrying around.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 33 ++++++++++++++++++++++-----------
 1 file changed, 22 insertions(+), 11 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 46fa0ae802..77b972fd93 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -1652,7 +1652,7 @@ static bool mmu_lookup1(CPUState *cpu, MMULookupPageData *data, MemOp memop,
     uint64_t tlb_addr = tlb_read_idx(entry, access_type);
     bool maybe_resized = false;
     CPUTLBEntryFull *full;
-    int flags;
+    int flags = TLB_FLAGS_MASK & ~TLB_FORCE_SLOW;
 
     /* If the TLB entry is for a different page, reload and try again.  */
     if (!tlb_hit(tlb_addr, addr)) {
@@ -1663,8 +1663,14 @@ static bool mmu_lookup1(CPUState *cpu, MMULookupPageData *data, MemOp memop,
             maybe_resized = true;
             index = tlb_index(cpu, mmu_idx, addr);
             entry = tlb_entry(cpu, mmu_idx, addr);
+            /*
+             * With PAGE_WRITE_INV, we set TLB_INVALID_MASK immediately,
+             * to force the next access through tlb_fill.  We've just
+             * called tlb_fill, so we know that this entry *is* valid.
+             */
+            flags &= ~TLB_INVALID_MASK;
         }
-        tlb_addr = tlb_read_idx(entry, access_type) & ~TLB_INVALID_MASK;
+        tlb_addr = tlb_read_idx(entry, access_type);
     }
 
     full = &cpu->neg.tlb.d[mmu_idx].fulltlb[index];
@@ -1814,10 +1820,10 @@ static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
     MemOp mop = get_memop(oi);
     uintptr_t index;
     CPUTLBEntry *tlbe;
-    vaddr tlb_addr;
     void *hostaddr;
     CPUTLBEntryFull *full;
     bool did_tlb_fill = false;
+    int flags;
 
     tcg_debug_assert(mmu_idx < NB_MMU_MODES);
 
@@ -1828,8 +1834,8 @@ static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
     tlbe = tlb_entry(cpu, mmu_idx, addr);
 
     /* Check TLB entry and enforce page permissions.  */
-    tlb_addr = tlb_addr_write(tlbe);
-    if (!tlb_hit(tlb_addr, addr)) {
+    flags = TLB_FLAGS_MASK;
+    if (!tlb_hit(tlb_addr_write(tlbe), addr)) {
         if (!victim_tlb_hit(cpu, mmu_idx, index, MMU_DATA_STORE,
                             addr & TARGET_PAGE_MASK)) {
             tlb_fill_align(cpu, addr, MMU_DATA_STORE, mmu_idx,
@@ -1837,8 +1843,13 @@ static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
             did_tlb_fill = true;
             index = tlb_index(cpu, mmu_idx, addr);
             tlbe = tlb_entry(cpu, mmu_idx, addr);
+            /*
+             * With PAGE_WRITE_INV, we set TLB_INVALID_MASK immediately,
+             * to force the next access through tlb_fill.  We've just
+             * called tlb_fill, so we know that this entry *is* valid.
+             */
+            flags &= ~TLB_INVALID_MASK;
         }
-        tlb_addr = tlb_addr_write(tlbe) & ~TLB_INVALID_MASK;
     }
 
     /*
@@ -1874,11 +1885,11 @@ static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
         goto stop_the_world;
     }
 
-    /* Collect tlb flags for read. */
-    tlb_addr |= tlbe->addr_read;
+    /* Collect tlb flags for read and write. */
+    flags &= tlbe->addr_read | tlb_addr_write(tlbe);
 
     /* Notice an IO access or a needs-MMU-lookup access */
-    if (unlikely(tlb_addr & (TLB_MMIO | TLB_DISCARD_WRITE))) {
+    if (unlikely(flags & (TLB_MMIO | TLB_DISCARD_WRITE))) {
         /* There's really nothing that can be done to
            support this apart from stop-the-world.  */
         goto stop_the_world;
@@ -1887,11 +1898,11 @@ static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
     hostaddr = (void *)((uintptr_t)addr + tlbe->addend);
     full = &cpu->neg.tlb.d[mmu_idx].fulltlb[index];
 
-    if (unlikely(tlb_addr & TLB_NOTDIRTY)) {
+    if (unlikely(flags & TLB_NOTDIRTY)) {
         notdirty_write(cpu, addr, size, full, retaddr);
     }
 
-    if (unlikely(tlb_addr & TLB_FORCE_SLOW)) {
+    if (unlikely(flags & TLB_FORCE_SLOW)) {
         int wp_flags = 0;
 
         if (full->slow_flags[MMU_DATA_STORE] & TLB_WATCHPOINT) {
-- 
2.43.0

[PATCH v2 06/54] accel/tcg: Assert non-zero length in tlb_flush_range_by_mmuidx*

[Richard Henderson]

Next patches will assume non-zero length.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 77b972fd93..1346a26d90 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -791,6 +791,7 @@ void tlb_flush_range_by_mmuidx(CPUState *cpu, vaddr addr,
     TLBFlushRangeData d;
 
     assert_cpu_is_self(cpu);
+    assert(len != 0);
 
     /*
      * If all bits are significant, and len is small,
@@ -830,6 +831,8 @@ void tlb_flush_range_by_mmuidx_all_cpus_synced(CPUState *src_cpu,
     TLBFlushRangeData d, *p;
     CPUState *dst_cpu;
 
+    assert(len != 0);
+
     /*
      * If all bits are significant, and len is small,
      * this devolves to tlb_flush_page.
-- 
2.43.0

[PATCH v2 07/54] accel/tcg: Assert bits in range in tlb_flush_range_by_mmuidx*

[Richard Henderson]

The only target that does not use TARGET_LONG_BITS is Arm, which
only reduces bits based on TBI.  There is no point in handling
odd combinations of parameters.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 16 ++++------------
 1 file changed, 4 insertions(+), 12 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 1346a26d90..5510f40333 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -792,20 +792,16 @@ void tlb_flush_range_by_mmuidx(CPUState *cpu, vaddr addr,
 
     assert_cpu_is_self(cpu);
     assert(len != 0);
+    assert(bits > TARGET_PAGE_BITS && bits <= TARGET_LONG_BITS);
 
     /*
      * If all bits are significant, and len is small,
      * this devolves to tlb_flush_page.
      */
-    if (bits >= TARGET_LONG_BITS && len <= TARGET_PAGE_SIZE) {
+    if (bits == TARGET_LONG_BITS && len <= TARGET_PAGE_SIZE) {
         tlb_flush_page_by_mmuidx(cpu, addr, idxmap);
         return;
     }
-    /* If no page bits are significant, this devolves to tlb_flush. */
-    if (bits < TARGET_PAGE_BITS) {
-        tlb_flush_by_mmuidx(cpu, idxmap);
-        return;
-    }
 
     /* This should already be page aligned */
     d.addr = addr & TARGET_PAGE_MASK;
@@ -832,20 +828,16 @@ void tlb_flush_range_by_mmuidx_all_cpus_synced(CPUState *src_cpu,
     CPUState *dst_cpu;
 
     assert(len != 0);
+    assert(bits > TARGET_PAGE_BITS && bits <= TARGET_LONG_BITS);
 
     /*
      * If all bits are significant, and len is small,
      * this devolves to tlb_flush_page.
      */
-    if (bits >= TARGET_LONG_BITS && len <= TARGET_PAGE_SIZE) {
+    if (bits == TARGET_LONG_BITS && len <= TARGET_PAGE_SIZE) {
         tlb_flush_page_by_mmuidx_all_cpus_synced(src_cpu, addr, idxmap);
         return;
     }
-    /* If no page bits are significant, this devolves to tlb_flush. */
-    if (bits < TARGET_PAGE_BITS) {
-        tlb_flush_by_mmuidx_all_cpus_synced(src_cpu, idxmap);
-        return;
-    }
 
     /* This should already be page aligned */
     d.addr = addr & TARGET_PAGE_MASK;
-- 
2.43.0

[PATCH v2 08/54] accel/tcg: Flush entire tlb when a masked range wraps

[Richard Henderson]

We expect masked address spaces to be quite large, e.g. 56 bits
for AArch64 top-byte-ignore mode.  We do not expect addr+len to
wrap around, but it is possible with AArch64 guest flush range
instructions.

Convert this unlikely case to a full tlb flush.  This can simplify
the subroutines actually performing the range flush.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 5510f40333..31c45a6213 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -802,6 +802,11 @@ void tlb_flush_range_by_mmuidx(CPUState *cpu, vaddr addr,
         tlb_flush_page_by_mmuidx(cpu, addr, idxmap);
         return;
     }
+    /* If addr+len wraps in len bits, fall back to full flush. */
+    if (bits < TARGET_LONG_BITS && ((addr ^ (addr + len - 1)) >> bits)) {
+        tlb_flush_by_mmuidx(cpu, idxmap);
+        return;
+    }
 
     /* This should already be page aligned */
     d.addr = addr & TARGET_PAGE_MASK;
@@ -838,6 +843,11 @@ void tlb_flush_range_by_mmuidx_all_cpus_synced(CPUState *src_cpu,
         tlb_flush_page_by_mmuidx_all_cpus_synced(src_cpu, addr, idxmap);
         return;
     }
+    /* If addr+len wraps in len bits, fall back to full flush. */
+    if (bits < TARGET_LONG_BITS && ((addr ^ (addr + len - 1)) >> bits)) {
+        tlb_flush_by_mmuidx_all_cpus_synced(src_cpu, idxmap);
+        return;
+    }
 
     /* This should already be page aligned */
     d.addr = addr & TARGET_PAGE_MASK;
-- 
2.43.0

[PATCH v2 09/54] accel/tcg: Add IntervalTreeRoot to CPUTLBDesc

[Richard Henderson]

Add the data structures for tracking softmmu pages via
a balanced interval tree.  So far, only initialize and
destroy the data structure.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 include/hw/core/cpu.h |  3 +++
 accel/tcg/cputlb.c    | 11 +++++++++++
 2 files changed, 14 insertions(+)

diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
index db8a6fbc6e..1ebc999a73 100644
--- a/include/hw/core/cpu.h
+++ b/include/hw/core/cpu.h
@@ -35,6 +35,7 @@
 #include "qemu/queue.h"
 #include "qemu/lockcnt.h"
 #include "qemu/thread.h"
+#include "qemu/interval-tree.h"
 #include "qom/object.h"
 
 typedef int (*WriteCoreDumpFunction)(const void *buf, size_t size,
@@ -290,6 +291,8 @@ typedef struct CPUTLBDesc {
     CPUTLBEntry vtable[CPU_VTLB_SIZE];
     CPUTLBEntryFull vfulltlb[CPU_VTLB_SIZE];
     CPUTLBEntryFull *fulltlb;
+    /* All active tlb entries for this address space. */
+    IntervalTreeRoot iroot;
 } CPUTLBDesc;
 
 /*
diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 31c45a6213..aa51fc1d26 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -89,6 +89,13 @@ QEMU_BUILD_BUG_ON(sizeof(vaddr) > sizeof(run_on_cpu_data));
 QEMU_BUILD_BUG_ON(NB_MMU_MODES > 16);
 #define ALL_MMUIDX_BITS ((1 << NB_MMU_MODES) - 1)
 
+/* Extra data required to manage CPUTLBEntryFull within an interval tree. */
+typedef struct CPUTLBEntryTree {
+    IntervalTreeNode itree;
+    CPUTLBEntry copy;
+    CPUTLBEntryFull full;
+} CPUTLBEntryTree;
+
 static inline size_t tlb_n_entries(CPUTLBDescFast *fast)
 {
     return (fast->mask >> CPU_TLB_ENTRY_BITS) + 1;
@@ -305,6 +312,7 @@ static void tlb_mmu_flush_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast)
     desc->large_page_mask = -1;
     desc->vindex = 0;
     memset(desc->vtable, -1, sizeof(desc->vtable));
+    interval_tree_free_nodes(&desc->iroot, offsetof(CPUTLBEntryTree, itree));
 }
 
 static void tlb_flush_one_mmuidx_locked(CPUState *cpu, int mmu_idx,
@@ -326,6 +334,7 @@ static void tlb_mmu_init(CPUTLBDesc *desc, CPUTLBDescFast *fast, int64_t now)
     fast->mask = (n_entries - 1) << CPU_TLB_ENTRY_BITS;
     fast->table = g_new(CPUTLBEntry, n_entries);
     desc->fulltlb = g_new(CPUTLBEntryFull, n_entries);
+    memset(&desc->iroot, 0, sizeof(desc->iroot));
     tlb_mmu_flush_locked(desc, fast);
 }
 
@@ -365,6 +374,8 @@ void tlb_destroy(CPUState *cpu)
 
         g_free(fast->table);
         g_free(desc->fulltlb);
+        interval_tree_free_nodes(&cpu->neg.tlb.d[i].iroot,
+                                 offsetof(CPUTLBEntryTree, itree));
     }
 }
 
-- 
2.43.0

[PATCH v2 10/54] accel/tcg: Populate IntervalTree in tlb_set_page_full

[Richard Henderson]

Add or replace an entry in the IntervalTree for each
page installed into softmmu.  We do not yet use the
tree for anything else.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 34 ++++++++++++++++++++++++++++------
 1 file changed, 28 insertions(+), 6 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index aa51fc1d26..ea6a5177de 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -305,6 +305,17 @@ static void tlbfast_flush_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast)
     memset(fast->table, -1, sizeof_tlb(fast));
 }
 
+static CPUTLBEntryTree *tlbtree_lookup_range(CPUTLBDesc *desc, vaddr s, vaddr l)
+{
+    IntervalTreeNode *i = interval_tree_iter_first(&desc->iroot, s, l);
+    return i ? container_of(i, CPUTLBEntryTree, itree) : NULL;
+}
+
+static CPUTLBEntryTree *tlbtree_lookup_addr(CPUTLBDesc *desc, vaddr addr)
+{
+    return tlbtree_lookup_range(desc, addr, addr);
+}
+
 static void tlb_mmu_flush_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast)
 {
     tlbfast_flush_locked(desc, fast);
@@ -1072,7 +1083,8 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
     MemoryRegionSection *section;
     unsigned int index, read_flags, write_flags;
     uintptr_t addend;
-    CPUTLBEntry *te, tn;
+    CPUTLBEntry *te;
+    CPUTLBEntryTree *node;
     hwaddr iotlb, xlat, sz, paddr_page;
     vaddr addr_page;
     int asidx, wp_flags, prot;
@@ -1180,6 +1192,15 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
         tlb_n_used_entries_dec(cpu, mmu_idx);
     }
 
+    /* Replace an old IntervalTree entry, or create a new one. */
+    node = tlbtree_lookup_addr(desc, addr_page);
+    if (!node) {
+        node = g_new(CPUTLBEntryTree, 1);
+        node->itree.start = addr_page;
+        node->itree.last = addr_page + TARGET_PAGE_SIZE - 1;
+        interval_tree_insert(&node->itree, &desc->iroot);
+    }
+
     /* refill the tlb */
     /*
      * When memory region is ram, iotlb contains a TARGET_PAGE_BITS
@@ -1201,15 +1222,15 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
     full->phys_addr = paddr_page;
 
     /* Now calculate the new entry */
-    tn.addend = addend - addr_page;
+    node->copy.addend = addend - addr_page;
 
-    tlb_set_compare(full, &tn, addr_page, read_flags,
+    tlb_set_compare(full, &node->copy, addr_page, read_flags,
                     MMU_INST_FETCH, prot & PAGE_EXEC);
 
     if (wp_flags & BP_MEM_READ) {
         read_flags |= TLB_WATCHPOINT;
     }
-    tlb_set_compare(full, &tn, addr_page, read_flags,
+    tlb_set_compare(full, &node->copy, addr_page, read_flags,
                     MMU_DATA_LOAD, prot & PAGE_READ);
 
     if (prot & PAGE_WRITE_INV) {
@@ -1218,10 +1239,11 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
     if (wp_flags & BP_MEM_WRITE) {
         write_flags |= TLB_WATCHPOINT;
     }
-    tlb_set_compare(full, &tn, addr_page, write_flags,
+    tlb_set_compare(full, &node->copy, addr_page, write_flags,
                     MMU_DATA_STORE, prot & PAGE_WRITE);
 
-    copy_tlb_helper_locked(te, &tn);
+    node->full = *full;
+    copy_tlb_helper_locked(te, &node->copy);
     tlb_n_used_entries_inc(cpu, mmu_idx);
     qemu_spin_unlock(&tlb->c.lock);
 }
-- 
2.43.0

[PATCH v2 11/54] accel/tcg: Remove IntervalTree entry in tlb_flush_page_locked

[Richard Henderson]

Flush a page from the IntervalTree cache.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index ea6a5177de..d532d69083 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -568,6 +568,7 @@ static void tlb_flush_page_locked(CPUState *cpu, int midx, vaddr page)
     CPUTLBDesc *desc = &cpu->neg.tlb.d[midx];
     vaddr lp_addr = desc->large_page_addr;
     vaddr lp_mask = desc->large_page_mask;
+    CPUTLBEntryTree *node;
 
     /* Check if we need to flush due to large pages.  */
     if ((page & lp_mask) == lp_addr) {
@@ -575,10 +576,17 @@ static void tlb_flush_page_locked(CPUState *cpu, int midx, vaddr page)
                   VADDR_PRIx "/%016" VADDR_PRIx ")\n",
                   midx, lp_addr, lp_mask);
         tlb_flush_one_mmuidx_locked(cpu, midx, get_clock_realtime());
-    } else {
-        tlbfast_flush_range_locked(desc, &cpu->neg.tlb.f[midx],
-                                   page, TARGET_PAGE_SIZE, -1);
-        tlb_flush_vtlb_page_locked(cpu, midx, page);
+        return;
+    }
+
+    tlbfast_flush_range_locked(desc, &cpu->neg.tlb.f[midx],
+                               page, TARGET_PAGE_SIZE, -1);
+    tlb_flush_vtlb_page_locked(cpu, midx, page);
+
+    node = tlbtree_lookup_addr(desc, page);
+    if (node) {
+        interval_tree_remove(&node->itree, &desc->iroot);
+        g_free(node);
     }
 }
 
-- 
2.43.0

[PATCH v2 12/54] accel/tcg: Remove IntervalTree entries in tlb_flush_range_locked

[Richard Henderson]

Flush a masked range of pages from the IntervalTree cache.
When the mask is not used there is a redundant comparison,
but that is better than duplicating code at this point.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index d532d69083..e2c855f147 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -311,6 +311,13 @@ static CPUTLBEntryTree *tlbtree_lookup_range(CPUTLBDesc *desc, vaddr s, vaddr l)
     return i ? container_of(i, CPUTLBEntryTree, itree) : NULL;
 }
 
+static CPUTLBEntryTree *tlbtree_lookup_range_next(CPUTLBEntryTree *prev,
+                                                  vaddr s, vaddr l)
+{
+    IntervalTreeNode *i = interval_tree_iter_next(&prev->itree, s, l);
+    return i ? container_of(i, CPUTLBEntryTree, itree) : NULL;
+}
+
 static CPUTLBEntryTree *tlbtree_lookup_addr(CPUTLBDesc *desc, vaddr addr)
 {
     return tlbtree_lookup_range(desc, addr, addr);
@@ -739,6 +746,8 @@ static void tlb_flush_range_locked(CPUState *cpu, int midx,
     CPUTLBDesc *d = &cpu->neg.tlb.d[midx];
     CPUTLBDescFast *f = &cpu->neg.tlb.f[midx];
     vaddr mask = MAKE_64BIT_MASK(0, bits);
+    CPUTLBEntryTree *node;
+    vaddr addr_mask, last_mask, last_imask;
 
     /*
      * Check if we need to flush due to large pages.
@@ -759,6 +768,22 @@ static void tlb_flush_range_locked(CPUState *cpu, int midx,
         vaddr page = addr + i;
         tlb_flush_vtlb_page_mask_locked(cpu, midx, page, mask);
     }
+
+    addr_mask = addr & mask;
+    last_mask = addr_mask + len - 1;
+    last_imask = last_mask | ~mask;
+    node = tlbtree_lookup_range(d, addr_mask, last_imask);
+    while (node) {
+        CPUTLBEntryTree *next =
+            tlbtree_lookup_range_next(node, addr_mask, last_imask);
+        vaddr page_mask = node->itree.start & mask;
+
+        if (page_mask >= addr_mask && page_mask < last_mask) {
+            interval_tree_remove(&node->itree, &d->iroot);
+            g_free(node);
+        }
+        node = next;
+    }
 }
 
 typedef struct {
-- 
2.43.0

[PATCH v2 13/54] accel/tcg: Process IntervalTree entries in tlb_reset_dirty

[Richard Henderson]

Update the addr_write copy within each interval tree node.
Tidy the iteration within the other two loops as well.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index e2c855f147..0c9f834cbe 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -1010,17 +1010,20 @@ void tlb_reset_dirty(CPUState *cpu, ram_addr_t start1, ram_addr_t length)
 
     qemu_spin_lock(&cpu->neg.tlb.c.lock);
     for (mmu_idx = 0; mmu_idx < NB_MMU_MODES; mmu_idx++) {
-        unsigned int i;
-        unsigned int n = tlb_n_entries(&cpu->neg.tlb.f[mmu_idx]);
+        CPUTLBDesc *desc = &cpu->neg.tlb.d[mmu_idx];
+        CPUTLBDescFast *fast = &cpu->neg.tlb.f[mmu_idx];
 
-        for (i = 0; i < n; i++) {
-            tlb_reset_dirty_range_locked(&cpu->neg.tlb.f[mmu_idx].table[i],
-                                         start1, length);
+        for (size_t i = 0, n = tlb_n_entries(fast); i < n; i++) {
+            tlb_reset_dirty_range_locked(&fast->table[i], start1, length);
         }
 
-        for (i = 0; i < CPU_VTLB_SIZE; i++) {
-            tlb_reset_dirty_range_locked(&cpu->neg.tlb.d[mmu_idx].vtable[i],
-                                         start1, length);
+        for (size_t i = 0; i < CPU_VTLB_SIZE; i++) {
+            tlb_reset_dirty_range_locked(&desc->vtable[i], start1, length);
+        }
+
+        for (CPUTLBEntryTree *t = tlbtree_lookup_range(desc, 0, -1); t;
+             t = tlbtree_lookup_range_next(t, 0, -1)) {
+            tlb_reset_dirty_range_locked(&t->copy, start1, length);
         }
     }
     qemu_spin_unlock(&cpu->neg.tlb.c.lock);
-- 
2.43.0

[PATCH v2 14/54] accel/tcg: Process IntervalTree entries in tlb_set_dirty

[Richard Henderson]

Update the addr_write copy within an interval tree node.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 0c9f834cbe..eb85e96ee2 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -1049,13 +1049,18 @@ static void tlb_set_dirty(CPUState *cpu, vaddr addr)
     addr &= TARGET_PAGE_MASK;
     qemu_spin_lock(&cpu->neg.tlb.c.lock);
     for (mmu_idx = 0; mmu_idx < NB_MMU_MODES; mmu_idx++) {
-        tlb_set_dirty1_locked(tlb_entry(cpu, mmu_idx, addr), addr);
-    }
+        CPUTLBDesc *desc = &cpu->neg.tlb.d[mmu_idx];
+        CPUTLBEntryTree *node;
 
-    for (mmu_idx = 0; mmu_idx < NB_MMU_MODES; mmu_idx++) {
-        int k;
-        for (k = 0; k < CPU_VTLB_SIZE; k++) {
-            tlb_set_dirty1_locked(&cpu->neg.tlb.d[mmu_idx].vtable[k], addr);
+        tlb_set_dirty1_locked(tlb_entry(cpu, mmu_idx, addr), addr);
+
+        for (int k = 0; k < CPU_VTLB_SIZE; k++) {
+            tlb_set_dirty1_locked(&desc->vtable[k], addr);
+        }
+
+        node = tlbtree_lookup_addr(desc, addr);
+        if (node) {
+            tlb_set_dirty1_locked(&node->copy, addr);
         }
     }
     qemu_spin_unlock(&cpu->neg.tlb.c.lock);
-- 
2.43.0

[PATCH v2 15/54] accel/tcg: Use tlb_hit_page in victim_tlb_hit

[Richard Henderson]

This is clearer than directly comparing the
page address and the comparator.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index eb85e96ee2..7ecd327297 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -1394,9 +1394,8 @@ static bool victim_tlb_hit(CPUState *cpu, size_t mmu_idx, size_t index,
     assert_cpu_is_self(cpu);
     for (vidx = 0; vidx < CPU_VTLB_SIZE; ++vidx) {
         CPUTLBEntry *vtlb = &cpu->neg.tlb.d[mmu_idx].vtable[vidx];
-        uint64_t cmp = tlb_read_idx(vtlb, access_type);
 
-        if (cmp == page) {
+        if (tlb_hit_page(tlb_read_idx(vtlb, access_type), page)) {
             /* Found entry in victim tlb, swap tlb and iotlb.  */
             CPUTLBEntry tmptlb, *tlb = &cpu->neg.tlb.f[mmu_idx].table[index];
 
-- 
2.43.0

[PATCH v2 16/54] accel/tcg: Pass full addr to victim_tlb_hit

[Richard Henderson]

Do not mask the address to the page in these calls.
It is easy enough to use a different helper instead.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 15 ++++++---------
 1 file changed, 6 insertions(+), 9 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 7ecd327297..3aab72ea82 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -1387,7 +1387,7 @@ static void io_failed(CPUState *cpu, CPUTLBEntryFull *full, vaddr addr,
 /* Return true if ADDR is present in the victim tlb, and has been copied
    back to the main tlb.  */
 static bool victim_tlb_hit(CPUState *cpu, size_t mmu_idx, size_t index,
-                           MMUAccessType access_type, vaddr page)
+                           MMUAccessType access_type, vaddr addr)
 {
     size_t vidx;
 
@@ -1395,7 +1395,7 @@ static bool victim_tlb_hit(CPUState *cpu, size_t mmu_idx, size_t index,
     for (vidx = 0; vidx < CPU_VTLB_SIZE; ++vidx) {
         CPUTLBEntry *vtlb = &cpu->neg.tlb.d[mmu_idx].vtable[vidx];
 
-        if (tlb_hit_page(tlb_read_idx(vtlb, access_type), page)) {
+        if (tlb_hit(tlb_read_idx(vtlb, access_type), addr)) {
             /* Found entry in victim tlb, swap tlb and iotlb.  */
             CPUTLBEntry tmptlb, *tlb = &cpu->neg.tlb.f[mmu_idx].table[index];
 
@@ -1448,13 +1448,12 @@ static int probe_access_internal(CPUState *cpu, vaddr addr,
     uintptr_t index = tlb_index(cpu, mmu_idx, addr);
     CPUTLBEntry *entry = tlb_entry(cpu, mmu_idx, addr);
     uint64_t tlb_addr = tlb_read_idx(entry, access_type);
-    vaddr page_addr = addr & TARGET_PAGE_MASK;
     int flags = TLB_FLAGS_MASK & ~TLB_FORCE_SLOW;
     bool force_mmio = check_mem_cbs && cpu_plugin_mem_cbs_enabled(cpu);
     CPUTLBEntryFull *full;
 
-    if (!tlb_hit_page(tlb_addr, page_addr)) {
-        if (!victim_tlb_hit(cpu, mmu_idx, index, access_type, page_addr)) {
+    if (!tlb_hit(tlb_addr, addr)) {
+        if (!victim_tlb_hit(cpu, mmu_idx, index, access_type, addr)) {
             if (!tlb_fill_align(cpu, addr, access_type, mmu_idx,
                                 0, fault_size, nonfault, retaddr)) {
                 /* Non-faulting page table read failed.  */
@@ -1734,8 +1733,7 @@ static bool mmu_lookup1(CPUState *cpu, MMULookupPageData *data, MemOp memop,
 
     /* If the TLB entry is for a different page, reload and try again.  */
     if (!tlb_hit(tlb_addr, addr)) {
-        if (!victim_tlb_hit(cpu, mmu_idx, index, access_type,
-                            addr & TARGET_PAGE_MASK)) {
+        if (!victim_tlb_hit(cpu, mmu_idx, index, access_type, addr)) {
             tlb_fill_align(cpu, addr, access_type, mmu_idx,
                            memop, data->size, false, ra);
             maybe_resized = true;
@@ -1914,8 +1912,7 @@ static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
     /* Check TLB entry and enforce page permissions.  */
     flags = TLB_FLAGS_MASK;
     if (!tlb_hit(tlb_addr_write(tlbe), addr)) {
-        if (!victim_tlb_hit(cpu, mmu_idx, index, MMU_DATA_STORE,
-                            addr & TARGET_PAGE_MASK)) {
+        if (!victim_tlb_hit(cpu, mmu_idx, index, MMU_DATA_STORE, addr)) {
             tlb_fill_align(cpu, addr, MMU_DATA_STORE, mmu_idx,
                            mop, size, false, retaddr);
             did_tlb_fill = true;
-- 
2.43.0

[PATCH v2 17/54] accel/tcg: Replace victim_tlb_hit with tlbtree_hit

[Richard Henderson]

Change from a linear search on the victim tlb
to a balanced binary tree search on the interval tree.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 59 ++++++++++++++++++++++++----------------------
 1 file changed, 31 insertions(+), 28 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 3aab72ea82..ea4b78866b 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -1384,35 +1384,38 @@ static void io_failed(CPUState *cpu, CPUTLBEntryFull *full, vaddr addr,
     }
 }
 
-/* Return true if ADDR is present in the victim tlb, and has been copied
-   back to the main tlb.  */
-static bool victim_tlb_hit(CPUState *cpu, size_t mmu_idx, size_t index,
-                           MMUAccessType access_type, vaddr addr)
+/*
+ * Return true if ADDR is present in the interval tree,
+ * and has been copied back to the main tlb.
+ */
+static bool tlbtree_hit(CPUState *cpu, int mmu_idx,
+                        MMUAccessType access_type, vaddr addr)
 {
-    size_t vidx;
+    CPUTLBDesc *desc = &cpu->neg.tlb.d[mmu_idx];
+    CPUTLBDescFast *fast = &cpu->neg.tlb.f[mmu_idx];
+    CPUTLBEntryTree *node;
+    size_t index;
 
     assert_cpu_is_self(cpu);
-    for (vidx = 0; vidx < CPU_VTLB_SIZE; ++vidx) {
-        CPUTLBEntry *vtlb = &cpu->neg.tlb.d[mmu_idx].vtable[vidx];
-
-        if (tlb_hit(tlb_read_idx(vtlb, access_type), addr)) {
-            /* Found entry in victim tlb, swap tlb and iotlb.  */
-            CPUTLBEntry tmptlb, *tlb = &cpu->neg.tlb.f[mmu_idx].table[index];
-
-            qemu_spin_lock(&cpu->neg.tlb.c.lock);
-            copy_tlb_helper_locked(&tmptlb, tlb);
-            copy_tlb_helper_locked(tlb, vtlb);
-            copy_tlb_helper_locked(vtlb, &tmptlb);
-            qemu_spin_unlock(&cpu->neg.tlb.c.lock);
-
-            CPUTLBEntryFull *f1 = &cpu->neg.tlb.d[mmu_idx].fulltlb[index];
-            CPUTLBEntryFull *f2 = &cpu->neg.tlb.d[mmu_idx].vfulltlb[vidx];
-            CPUTLBEntryFull tmpf;
-            tmpf = *f1; *f1 = *f2; *f2 = tmpf;
-            return true;
-        }
+    node = tlbtree_lookup_addr(desc, addr);
+    if (!node) {
+        /* There is no cached mapping for this page. */
+        return false;
     }
-    return false;
+
+    if (!tlb_hit(tlb_read_idx(&node->copy, access_type), addr)) {
+        /* This access is not permitted. */
+        return false;
+    }
+
+    /* Install the cached entry. */
+    index = tlbfast_index(fast, addr);
+    qemu_spin_lock(&cpu->neg.tlb.c.lock);
+    copy_tlb_helper_locked(&fast->table[index], &node->copy);
+    qemu_spin_unlock(&cpu->neg.tlb.c.lock);
+
+    desc->fulltlb[index] = node->full;
+    return true;
 }
 
 static void notdirty_write(CPUState *cpu, vaddr mem_vaddr, unsigned size,
@@ -1453,7 +1456,7 @@ static int probe_access_internal(CPUState *cpu, vaddr addr,
     CPUTLBEntryFull *full;
 
     if (!tlb_hit(tlb_addr, addr)) {
-        if (!victim_tlb_hit(cpu, mmu_idx, index, access_type, addr)) {
+        if (!tlbtree_hit(cpu, mmu_idx, access_type, addr)) {
             if (!tlb_fill_align(cpu, addr, access_type, mmu_idx,
                                 0, fault_size, nonfault, retaddr)) {
                 /* Non-faulting page table read failed.  */
@@ -1733,7 +1736,7 @@ static bool mmu_lookup1(CPUState *cpu, MMULookupPageData *data, MemOp memop,
 
     /* If the TLB entry is for a different page, reload and try again.  */
     if (!tlb_hit(tlb_addr, addr)) {
-        if (!victim_tlb_hit(cpu, mmu_idx, index, access_type, addr)) {
+        if (!tlbtree_hit(cpu, mmu_idx, access_type, addr)) {
             tlb_fill_align(cpu, addr, access_type, mmu_idx,
                            memop, data->size, false, ra);
             maybe_resized = true;
@@ -1912,7 +1915,7 @@ static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
     /* Check TLB entry and enforce page permissions.  */
     flags = TLB_FLAGS_MASK;
     if (!tlb_hit(tlb_addr_write(tlbe), addr)) {
-        if (!victim_tlb_hit(cpu, mmu_idx, index, MMU_DATA_STORE, addr)) {
+        if (!tlbtree_hit(cpu, mmu_idx, MMU_DATA_STORE, addr)) {
             tlb_fill_align(cpu, addr, MMU_DATA_STORE, mmu_idx,
                            mop, size, false, retaddr);
             did_tlb_fill = true;
-- 
2.43.0

[PATCH v2 18/54] accel/tcg: Remove the victim tlb

[Richard Henderson]

This has been functionally replaced by the IntervalTree.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 include/hw/core/cpu.h |  8 -----
 accel/tcg/cputlb.c    | 74 -------------------------------------------
 2 files changed, 82 deletions(-)

diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
index 1ebc999a73..8eda0574b2 100644
--- a/include/hw/core/cpu.h
+++ b/include/hw/core/cpu.h
@@ -201,9 +201,6 @@ struct CPUClass {
  */
 #define NB_MMU_MODES 16
 
-/* Use a fully associative victim tlb of 8 entries. */
-#define CPU_VTLB_SIZE 8
-
 /*
  * The full TLB entry, which is not accessed by generated TCG code,
  * so the layout is not as critical as that of CPUTLBEntry. This is
@@ -285,11 +282,6 @@ typedef struct CPUTLBDesc {
     /* maximum number of entries observed in the window */
     size_t window_max_entries;
     size_t n_used_entries;
-    /* The next index to use in the tlb victim table.  */
-    size_t vindex;
-    /* The tlb victim table, in two parts.  */
-    CPUTLBEntry vtable[CPU_VTLB_SIZE];
-    CPUTLBEntryFull vfulltlb[CPU_VTLB_SIZE];
     CPUTLBEntryFull *fulltlb;
     /* All active tlb entries for this address space. */
     IntervalTreeRoot iroot;
diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index ea4b78866b..8caa8c0f1d 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -328,8 +328,6 @@ static void tlb_mmu_flush_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast)
     tlbfast_flush_locked(desc, fast);
     desc->large_page_addr = -1;
     desc->large_page_mask = -1;
-    desc->vindex = 0;
-    memset(desc->vtable, -1, sizeof(desc->vtable));
     interval_tree_free_nodes(&desc->iroot, offsetof(CPUTLBEntryTree, itree));
 }
 
@@ -361,11 +359,6 @@ static inline void tlb_n_used_entries_inc(CPUState *cpu, uintptr_t mmu_idx)
     cpu->neg.tlb.d[mmu_idx].n_used_entries++;
 }
 
-static inline void tlb_n_used_entries_dec(CPUState *cpu, uintptr_t mmu_idx)
-{
-    cpu->neg.tlb.d[mmu_idx].n_used_entries--;
-}
-
 void tlb_init(CPUState *cpu)
 {
     int64_t now = get_clock_realtime();
@@ -496,20 +489,6 @@ static bool tlb_hit_page_mask_anyprot(CPUTLBEntry *tlb_entry,
             page == (tlb_entry->addr_code & mask));
 }
 
-static inline bool tlb_hit_page_anyprot(CPUTLBEntry *tlb_entry, vaddr page)
-{
-    return tlb_hit_page_mask_anyprot(tlb_entry, page, -1);
-}
-
-/**
- * tlb_entry_is_empty - return true if the entry is not in use
- * @te: pointer to CPUTLBEntry
- */
-static inline bool tlb_entry_is_empty(const CPUTLBEntry *te)
-{
-    return te->addr_read == -1 && te->addr_write == -1 && te->addr_code == -1;
-}
-
 /* Called with tlb_c.lock held */
 static bool tlb_flush_entry_mask_locked(CPUTLBEntry *tlb_entry,
                                         vaddr page,
@@ -522,28 +501,6 @@ static bool tlb_flush_entry_mask_locked(CPUTLBEntry *tlb_entry,
     return false;
 }
 
-/* Called with tlb_c.lock held */
-static void tlb_flush_vtlb_page_mask_locked(CPUState *cpu, int mmu_idx,
-                                            vaddr page,
-                                            vaddr mask)
-{
-    CPUTLBDesc *d = &cpu->neg.tlb.d[mmu_idx];
-    int k;
-
-    assert_cpu_is_self(cpu);
-    for (k = 0; k < CPU_VTLB_SIZE; k++) {
-        if (tlb_flush_entry_mask_locked(&d->vtable[k], page, mask)) {
-            tlb_n_used_entries_dec(cpu, mmu_idx);
-        }
-    }
-}
-
-static inline void tlb_flush_vtlb_page_locked(CPUState *cpu, int mmu_idx,
-                                              vaddr page)
-{
-    tlb_flush_vtlb_page_mask_locked(cpu, mmu_idx, page, -1);
-}
-
 static void tlbfast_flush_range_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast,
                                        vaddr addr, vaddr len, vaddr mask)
 {
@@ -588,7 +545,6 @@ static void tlb_flush_page_locked(CPUState *cpu, int midx, vaddr page)
 
     tlbfast_flush_range_locked(desc, &cpu->neg.tlb.f[midx],
                                page, TARGET_PAGE_SIZE, -1);
-    tlb_flush_vtlb_page_locked(cpu, midx, page);
 
     node = tlbtree_lookup_addr(desc, page);
     if (node) {
@@ -764,11 +720,6 @@ static void tlb_flush_range_locked(CPUState *cpu, int midx,
 
     tlbfast_flush_range_locked(d, f, addr, len, mask);
 
-    for (vaddr i = 0; i < len; i += TARGET_PAGE_SIZE) {
-        vaddr page = addr + i;
-        tlb_flush_vtlb_page_mask_locked(cpu, midx, page, mask);
-    }
-
     addr_mask = addr & mask;
     last_mask = addr_mask + len - 1;
     last_imask = last_mask | ~mask;
@@ -1017,10 +968,6 @@ void tlb_reset_dirty(CPUState *cpu, ram_addr_t start1, ram_addr_t length)
             tlb_reset_dirty_range_locked(&fast->table[i], start1, length);
         }
 
-        for (size_t i = 0; i < CPU_VTLB_SIZE; i++) {
-            tlb_reset_dirty_range_locked(&desc->vtable[i], start1, length);
-        }
-
         for (CPUTLBEntryTree *t = tlbtree_lookup_range(desc, 0, -1); t;
              t = tlbtree_lookup_range_next(t, 0, -1)) {
             tlb_reset_dirty_range_locked(&t->copy, start1, length);
@@ -1054,10 +1001,6 @@ static void tlb_set_dirty(CPUState *cpu, vaddr addr)
 
         tlb_set_dirty1_locked(tlb_entry(cpu, mmu_idx, addr), addr);
 
-        for (int k = 0; k < CPU_VTLB_SIZE; k++) {
-            tlb_set_dirty1_locked(&desc->vtable[k], addr);
-        }
-
         node = tlbtree_lookup_addr(desc, addr);
         if (node) {
             tlb_set_dirty1_locked(&node->copy, addr);
@@ -1216,23 +1159,6 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
     /* Note that the tlb is no longer clean.  */
     tlb->c.dirty |= 1 << mmu_idx;
 
-    /* Make sure there's no cached translation for the new page.  */
-    tlb_flush_vtlb_page_locked(cpu, mmu_idx, addr_page);
-
-    /*
-     * Only evict the old entry to the victim tlb if it's for a
-     * different page; otherwise just overwrite the stale data.
-     */
-    if (!tlb_hit_page_anyprot(te, addr_page) && !tlb_entry_is_empty(te)) {
-        unsigned vidx = desc->vindex++ % CPU_VTLB_SIZE;
-        CPUTLBEntry *tv = &desc->vtable[vidx];
-
-        /* Evict the old entry into the victim tlb.  */
-        copy_tlb_helper_locked(tv, te);
-        desc->vfulltlb[vidx] = desc->fulltlb[index];
-        tlb_n_used_entries_dec(cpu, mmu_idx);
-    }
-
     /* Replace an old IntervalTree entry, or create a new one. */
     node = tlbtree_lookup_addr(desc, addr_page);
     if (!node) {
-- 
2.43.0

[PATCH v2 19/54] accel/tcg: Remove tlb_n_used_entries_inc

[Richard Henderson]

Expand the function into its only caller, using the
existing CPUTLBDesc local pointer.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 8caa8c0f1d..3e24529f4f 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -354,11 +354,6 @@ static void tlb_mmu_init(CPUTLBDesc *desc, CPUTLBDescFast *fast, int64_t now)
     tlb_mmu_flush_locked(desc, fast);
 }
 
-static inline void tlb_n_used_entries_inc(CPUState *cpu, uintptr_t mmu_idx)
-{
-    cpu->neg.tlb.d[mmu_idx].n_used_entries++;
-}
-
 void tlb_init(CPUState *cpu)
 {
     int64_t now = get_clock_realtime();
@@ -1211,7 +1206,7 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
 
     node->full = *full;
     copy_tlb_helper_locked(te, &node->copy);
-    tlb_n_used_entries_inc(cpu, mmu_idx);
+    desc->n_used_entries++;
     qemu_spin_unlock(&tlb->c.lock);
 }
 
-- 
2.43.0

[PATCH v2 20/54] include/exec/tlb-common: Move CPUTLBEntryFull from hw/core/cpu.h

[Richard Henderson]

CPUTLBEntryFull structures are no longer directly included within
the CPUState structure.  Move the structure definition out of cpu.h
to reduce visibility.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 include/exec/tlb-common.h | 63 +++++++++++++++++++++++++++++++++++++++
 include/hw/core/cpu.h     | 63 ---------------------------------------
 2 files changed, 63 insertions(+), 63 deletions(-)

diff --git a/include/exec/tlb-common.h b/include/exec/tlb-common.h
index dc5a5faa0b..300f9fae67 100644
--- a/include/exec/tlb-common.h
+++ b/include/exec/tlb-common.h
@@ -53,4 +53,67 @@ typedef struct CPUTLBDescFast {
     CPUTLBEntry *table;
 } CPUTLBDescFast QEMU_ALIGNED(2 * sizeof(void *));
 
+/*
+ * The full TLB entry, which is not accessed by generated TCG code,
+ * so the layout is not as critical as that of CPUTLBEntry. This is
+ * also why we don't want to combine the two structs.
+ */
+struct CPUTLBEntryFull {
+    /*
+     * @xlat_section contains:
+     *  - in the lower TARGET_PAGE_BITS, a physical section number
+     *  - with the lower TARGET_PAGE_BITS masked off, an offset which
+     *    must be added to the virtual address to obtain:
+     *     + the ram_addr_t of the target RAM (if the physical section
+     *       number is PHYS_SECTION_NOTDIRTY or PHYS_SECTION_ROM)
+     *     + the offset within the target MemoryRegion (otherwise)
+     */
+    hwaddr xlat_section;
+
+    /*
+     * @phys_addr contains the physical address in the address space
+     * given by cpu_asidx_from_attrs(cpu, @attrs).
+     */
+    hwaddr phys_addr;
+
+    /* @attrs contains the memory transaction attributes for the page. */
+    MemTxAttrs attrs;
+
+    /* @prot contains the complete protections for the page. */
+    uint8_t prot;
+
+    /* @lg_page_size contains the log2 of the page size. */
+    uint8_t lg_page_size;
+
+    /* Additional tlb flags requested by tlb_fill. */
+    uint8_t tlb_fill_flags;
+
+    /*
+     * Additional tlb flags for use by the slow path. If non-zero,
+     * the corresponding CPUTLBEntry comparator must have TLB_FORCE_SLOW.
+     */
+    uint8_t slow_flags[MMU_ACCESS_COUNT];
+
+    /*
+     * Allow target-specific additions to this structure.
+     * This may be used to cache items from the guest cpu
+     * page tables for later use by the implementation.
+     */
+    union {
+        /*
+         * Cache the attrs and shareability fields from the page table entry.
+         *
+         * For ARMMMUIdx_Stage2*, pte_attrs is the S2 descriptor bits [5:2].
+         * Otherwise, pte_attrs is the same as the MAIR_EL1 8-bit format.
+         * For shareability and guarded, as in the SH and GP fields respectively
+         * of the VMSAv8-64 PTEs.
+         */
+        struct {
+            uint8_t pte_attrs;
+            uint8_t shareability;
+            bool guarded;
+        } arm;
+    } extra;
+};
+
 #endif /* EXEC_TLB_COMMON_H */
diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
index 8eda0574b2..4364ddb1db 100644
--- a/include/hw/core/cpu.h
+++ b/include/hw/core/cpu.h
@@ -201,69 +201,6 @@ struct CPUClass {
  */
 #define NB_MMU_MODES 16
 
-/*
- * The full TLB entry, which is not accessed by generated TCG code,
- * so the layout is not as critical as that of CPUTLBEntry. This is
- * also why we don't want to combine the two structs.
- */
-struct CPUTLBEntryFull {
-    /*
-     * @xlat_section contains:
-     *  - in the lower TARGET_PAGE_BITS, a physical section number
-     *  - with the lower TARGET_PAGE_BITS masked off, an offset which
-     *    must be added to the virtual address to obtain:
-     *     + the ram_addr_t of the target RAM (if the physical section
-     *       number is PHYS_SECTION_NOTDIRTY or PHYS_SECTION_ROM)
-     *     + the offset within the target MemoryRegion (otherwise)
-     */
-    hwaddr xlat_section;
-
-    /*
-     * @phys_addr contains the physical address in the address space
-     * given by cpu_asidx_from_attrs(cpu, @attrs).
-     */
-    hwaddr phys_addr;
-
-    /* @attrs contains the memory transaction attributes for the page. */
-    MemTxAttrs attrs;
-
-    /* @prot contains the complete protections for the page. */
-    uint8_t prot;
-
-    /* @lg_page_size contains the log2 of the page size. */
-    uint8_t lg_page_size;
-
-    /* Additional tlb flags requested by tlb_fill. */
-    uint8_t tlb_fill_flags;
-
-    /*
-     * Additional tlb flags for use by the slow path. If non-zero,
-     * the corresponding CPUTLBEntry comparator must have TLB_FORCE_SLOW.
-     */
-    uint8_t slow_flags[MMU_ACCESS_COUNT];
-
-    /*
-     * Allow target-specific additions to this structure.
-     * This may be used to cache items from the guest cpu
-     * page tables for later use by the implementation.
-     */
-    union {
-        /*
-         * Cache the attrs and shareability fields from the page table entry.
-         *
-         * For ARMMMUIdx_Stage2*, pte_attrs is the S2 descriptor bits [5:2].
-         * Otherwise, pte_attrs is the same as the MAIR_EL1 8-bit format.
-         * For shareability and guarded, as in the SH and GP fields respectively
-         * of the VMSAv8-64 PTEs.
-         */
-        struct {
-            uint8_t pte_attrs;
-            uint8_t shareability;
-            bool guarded;
-        } arm;
-    } extra;
-};
-
 /*
  * Data elements that are per MMU mode, minus the bits accessed by
  * the TCG fast path.
-- 
2.43.0

[PATCH v2 21/54] accel/tcg: Delay plugin adjustment in probe_access_internal

[Richard Henderson]

Remove force_mmio and place the expression into the IF
expression, behind the short-circuit logic expressions
that might eliminate its computation.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 3e24529f4f..a4c69bcbf1 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -1373,7 +1373,6 @@ static int probe_access_internal(CPUState *cpu, vaddr addr,
     CPUTLBEntry *entry = tlb_entry(cpu, mmu_idx, addr);
     uint64_t tlb_addr = tlb_read_idx(entry, access_type);
     int flags = TLB_FLAGS_MASK & ~TLB_FORCE_SLOW;
-    bool force_mmio = check_mem_cbs && cpu_plugin_mem_cbs_enabled(cpu);
     CPUTLBEntryFull *full;
 
     if (!tlb_hit(tlb_addr, addr)) {
@@ -1404,9 +1403,14 @@ static int probe_access_internal(CPUState *cpu, vaddr addr,
     *pfull = full = &cpu->neg.tlb.d[mmu_idx].fulltlb[index];
     flags |= full->slow_flags[access_type];
 
-    /* Fold all "mmio-like" bits into TLB_MMIO.  This is not RAM.  */
-    if (unlikely(flags & ~(TLB_WATCHPOINT | TLB_NOTDIRTY | TLB_CHECK_ALIGNED))
-        || (access_type != MMU_INST_FETCH && force_mmio)) {
+    /*
+     * Fold all "mmio-like" bits, and required plugin callbacks, to TLB_MMIO.
+     * These cannot be treated as RAM.
+     */
+    if ((flags & ~(TLB_WATCHPOINT | TLB_NOTDIRTY | TLB_CHECK_ALIGNED))
+        || (access_type != MMU_INST_FETCH
+            && check_mem_cbs
+            && cpu_plugin_mem_cbs_enabled(cpu))) {
         *phost = NULL;
         return TLB_MMIO;
     }
-- 
2.43.0

[PATCH v2 22/54] accel/tcg: Call cpu_ld*_code_mmu from cpu_ld*_code

[Richard Henderson]

Ensure a common entry point for all code lookups.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index a4c69bcbf1..c975dd2322 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -2924,28 +2924,28 @@ uint32_t cpu_ldub_code(CPUArchState *env, abi_ptr addr)
 {
     CPUState *cs = env_cpu(env);
     MemOpIdx oi = make_memop_idx(MO_UB, cpu_mmu_index(cs, true));
-    return do_ld1_mmu(cs, addr, oi, 0, MMU_INST_FETCH);
+    return cpu_ldb_code_mmu(env, addr, oi, 0);
 }
 
 uint32_t cpu_lduw_code(CPUArchState *env, abi_ptr addr)
 {
     CPUState *cs = env_cpu(env);
     MemOpIdx oi = make_memop_idx(MO_TEUW, cpu_mmu_index(cs, true));
-    return do_ld2_mmu(cs, addr, oi, 0, MMU_INST_FETCH);
+    return cpu_ldw_code_mmu(env, addr, oi, 0);
 }
 
 uint32_t cpu_ldl_code(CPUArchState *env, abi_ptr addr)
 {
     CPUState *cs = env_cpu(env);
     MemOpIdx oi = make_memop_idx(MO_TEUL, cpu_mmu_index(cs, true));
-    return do_ld4_mmu(cs, addr, oi, 0, MMU_INST_FETCH);
+    return cpu_ldl_code_mmu(env, addr, oi, 0);
 }
 
 uint64_t cpu_ldq_code(CPUArchState *env, abi_ptr addr)
 {
     CPUState *cs = env_cpu(env);
     MemOpIdx oi = make_memop_idx(MO_TEUQ, cpu_mmu_index(cs, true));
-    return do_ld8_mmu(cs, addr, oi, 0, MMU_INST_FETCH);
+    return cpu_ldq_code_mmu(env, addr, oi, 0);
 }
 
 uint8_t cpu_ldb_code_mmu(CPUArchState *env, abi_ptr addr,
-- 
2.43.0

[PATCH v2 23/54] accel/tcg: Check original prot bits for read in atomic_mmu_lookup

[Richard Henderson]

In the mist before CPUTLBEntryFull existed, we had to be
clever to detect write-only pages.  Now we can directly
test the saved prot bits, which is clearer.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index c975dd2322..ae3a99eb47 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -1854,14 +1854,13 @@ static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
             flags &= ~TLB_INVALID_MASK;
         }
     }
+    full = &cpu->neg.tlb.d[mmu_idx].fulltlb[index];
 
     /*
      * Let the guest notice RMW on a write-only page.
      * We have just verified that the page is writable.
-     * Subpage lookups may have left TLB_INVALID_MASK set,
-     * but addr_read will only be -1 if PAGE_READ was unset.
      */
-    if (unlikely(tlbe->addr_read == -1)) {
+    if (unlikely(!(full->prot & PAGE_READ))) {
         tlb_fill_align(cpu, addr, MMU_DATA_LOAD, mmu_idx,
                        0, size, false, retaddr);
         /*
@@ -1899,7 +1898,6 @@ static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
     }
 
     hostaddr = (void *)((uintptr_t)addr + tlbe->addend);
-    full = &cpu->neg.tlb.d[mmu_idx].fulltlb[index];
 
     if (unlikely(flags & TLB_NOTDIRTY)) {
         notdirty_write(cpu, addr, size, full, retaddr);
-- 
2.43.0

[PATCH v2 24/54] accel/tcg: Preserve tlb flags in tlb_set_compare

[Richard Henderson]

Before, if !enable, we squashed the entire address comparator to -1.
This works because TLB_INVALID_MASK is set.  It seemed natural, because
the tlb is cleared with memset of 0xff.

With this patch, we retain all of the other TLB_* bits even when
the page is not enabled.  This works because TLB_INVALID_MASK is set.
This will be used in a subsequent patch; the addr_read comparator
contains the flags for pages that are executable but not readable.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 16 +++++++---------
 1 file changed, 7 insertions(+), 9 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index ae3a99eb47..585f4171cc 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -1032,15 +1032,13 @@ static inline void tlb_set_compare(CPUTLBEntryFull *full, CPUTLBEntry *ent,
                                    vaddr address, int flags,
                                    MMUAccessType access_type, bool enable)
 {
-    if (enable) {
-        address |= flags & TLB_FLAGS_MASK;
-        flags &= TLB_SLOW_FLAGS_MASK;
-        if (flags) {
-            address |= TLB_FORCE_SLOW;
-        }
-    } else {
-        address = -1;
-        flags = 0;
+    if (!enable) {
+        address = TLB_INVALID_MASK;
+    }
+    address |= flags & TLB_FLAGS_MASK;
+    flags &= TLB_SLOW_FLAGS_MASK;
+    if (flags) {
+        address |= TLB_FORCE_SLOW;
     }
     ent->addr_idx[access_type] = address;
     full->slow_flags[access_type] = flags;
-- 
2.43.0

[PATCH v2 25/54] accel/tcg: Return CPUTLBEntryFull not pointer in probe_access_full_mmu

[Richard Henderson]

Return a copy of the structure, not a pointer.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 include/exec/exec-all.h              |  2 +-
 accel/tcg/cputlb.c                   | 13 ++++++++-----
 target/arm/ptw.c                     | 10 +++++-----
 target/i386/tcg/sysemu/excp_helper.c |  8 ++++----
 4 files changed, 18 insertions(+), 15 deletions(-)

diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
index 2e4c4cc4b4..df7d0b5ad0 100644
--- a/include/exec/exec-all.h
+++ b/include/exec/exec-all.h
@@ -393,7 +393,7 @@ int probe_access_full(CPUArchState *env, vaddr addr, int size,
  */
 int probe_access_full_mmu(CPUArchState *env, vaddr addr, int size,
                           MMUAccessType access_type, int mmu_idx,
-                          void **phost, CPUTLBEntryFull **pfull);
+                          void **phost, CPUTLBEntryFull *pfull);
 
 #endif /* !CONFIG_USER_ONLY */
 #endif /* CONFIG_TCG */
diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 585f4171cc..81135524eb 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -1439,25 +1439,28 @@ int probe_access_full(CPUArchState *env, vaddr addr, int size,
 
 int probe_access_full_mmu(CPUArchState *env, vaddr addr, int size,
                           MMUAccessType access_type, int mmu_idx,
-                          void **phost, CPUTLBEntryFull **pfull)
+                          void **phost, CPUTLBEntryFull *pfull)
 {
     void *discard_phost;
-    CPUTLBEntryFull *discard_tlb;
+    CPUTLBEntryFull *full;
 
     /* privately handle users that don't need full results */
     phost = phost ? phost : &discard_phost;
-    pfull = pfull ? pfull : &discard_tlb;
 
     int flags = probe_access_internal(env_cpu(env), addr, size, access_type,
-                                      mmu_idx, true, phost, pfull, 0, false);
+                                      mmu_idx, true, phost, &full, 0, false);
 
     /* Handle clean RAM pages.  */
     if (unlikely(flags & TLB_NOTDIRTY)) {
         int dirtysize = size == 0 ? 1 : size;
-        notdirty_write(env_cpu(env), addr, dirtysize, *pfull, 0);
+        notdirty_write(env_cpu(env), addr, dirtysize, full, 0);
         flags &= ~TLB_NOTDIRTY;
     }
 
+    if (pfull) {
+        *pfull = *full;
+    }
+
     return flags;
 }
 
diff --git a/target/arm/ptw.c b/target/arm/ptw.c
index 9849949508..3ae5f524de 100644
--- a/target/arm/ptw.c
+++ b/target/arm/ptw.c
@@ -592,7 +592,7 @@ static bool S1_ptw_translate(CPUARMState *env, S1Translate *ptw,
         ptw->out_space = s2.f.attrs.space;
     } else {
 #ifdef CONFIG_TCG
-        CPUTLBEntryFull *full;
+        CPUTLBEntryFull full;
         int flags;
 
         env->tlb_fi = fi;
@@ -604,10 +604,10 @@ static bool S1_ptw_translate(CPUARMState *env, S1Translate *ptw,
         if (unlikely(flags & TLB_INVALID_MASK)) {
             goto fail;
         }
-        ptw->out_phys = full->phys_addr | (addr & ~TARGET_PAGE_MASK);
-        ptw->out_rw = full->prot & PAGE_WRITE;
-        pte_attrs = full->extra.arm.pte_attrs;
-        ptw->out_space = full->attrs.space;
+        ptw->out_phys = full.phys_addr | (addr & ~TARGET_PAGE_MASK);
+        ptw->out_rw = full.prot & PAGE_WRITE;
+        pte_attrs = full.extra.arm.pte_attrs;
+        ptw->out_space = full.attrs.space;
 #else
         g_assert_not_reached();
 #endif
diff --git a/target/i386/tcg/sysemu/excp_helper.c b/target/i386/tcg/sysemu/excp_helper.c
index 02d3486421..168ff8e5f3 100644
--- a/target/i386/tcg/sysemu/excp_helper.c
+++ b/target/i386/tcg/sysemu/excp_helper.c
@@ -436,7 +436,7 @@ do_check_protect_pse36:
      * addresses) using the address with the A20 bit set.
      */
     if (in->ptw_idx == MMU_NESTED_IDX) {
-        CPUTLBEntryFull *full;
+        CPUTLBEntryFull full;
         int flags, nested_page_size;
 
         flags = probe_access_full_mmu(env, paddr, 0, access_type,
@@ -451,7 +451,7 @@ do_check_protect_pse36:
         }
 
         /* Merge stage1 & stage2 protection bits. */
-        prot &= full->prot;
+        prot &= full.prot;
 
         /* Re-verify resulting protection. */
         if ((prot & (1 << access_type)) == 0) {
@@ -459,8 +459,8 @@ do_check_protect_pse36:
         }
 
         /* Merge stage1 & stage2 addresses to final physical address. */
-        nested_page_size = 1 << full->lg_page_size;
-        paddr = (full->phys_addr & ~(nested_page_size - 1))
+        nested_page_size = 1 << full.lg_page_size;
+        paddr = (full.phys_addr & ~(nested_page_size - 1))
               | (paddr & (nested_page_size - 1));
 
         /*
-- 
2.43.0

[PATCH v2 26/54] accel/tcg: Return CPUTLBEntryFull not pointer in probe_access_full

[Richard Henderson]

Return a copy of the structure, not a pointer.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 include/exec/exec-all.h     |  6 +-----
 accel/tcg/cputlb.c          |  8 +++++---
 target/arm/tcg/helper-a64.c |  4 ++--
 target/arm/tcg/mte_helper.c | 15 ++++++---------
 target/arm/tcg/sve_helper.c |  6 +++---
 5 files changed, 17 insertions(+), 22 deletions(-)

diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
index df7d0b5ad0..69bdb77584 100644
--- a/include/exec/exec-all.h
+++ b/include/exec/exec-all.h
@@ -365,10 +365,6 @@ int probe_access_flags(CPUArchState *env, vaddr addr, int size,
  * probe_access_full:
  * Like probe_access_flags, except also return into @pfull.
  *
- * The CPUTLBEntryFull structure returned via @pfull is transient
- * and must be consumed or copied immediately, before any further
- * access or changes to TLB @mmu_idx.
- *
  * This function will not fault if @nonfault is set, but will
  * return TLB_INVALID_MASK if the page is not mapped, or is not
  * accessible with @access_type.
@@ -379,7 +375,7 @@ int probe_access_flags(CPUArchState *env, vaddr addr, int size,
 int probe_access_full(CPUArchState *env, vaddr addr, int size,
                       MMUAccessType access_type, int mmu_idx,
                       bool nonfault, void **phost,
-                      CPUTLBEntryFull **pfull, uintptr_t retaddr);
+                      CPUTLBEntryFull *pfull, uintptr_t retaddr);
 
 /**
  * probe_access_full_mmu:
diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 81135524eb..84e7e633e3 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -1420,20 +1420,22 @@ static int probe_access_internal(CPUState *cpu, vaddr addr,
 
 int probe_access_full(CPUArchState *env, vaddr addr, int size,
                       MMUAccessType access_type, int mmu_idx,
-                      bool nonfault, void **phost, CPUTLBEntryFull **pfull,
+                      bool nonfault, void **phost, CPUTLBEntryFull *pfull,
                       uintptr_t retaddr)
 {
+    CPUTLBEntryFull *full;
     int flags = probe_access_internal(env_cpu(env), addr, size, access_type,
-                                      mmu_idx, nonfault, phost, pfull, retaddr,
+                                      mmu_idx, nonfault, phost, &full, retaddr,
                                       true);
 
     /* Handle clean RAM pages.  */
     if (unlikely(flags & TLB_NOTDIRTY)) {
         int dirtysize = size == 0 ? 1 : size;
-        notdirty_write(env_cpu(env), addr, dirtysize, *pfull, retaddr);
+        notdirty_write(env_cpu(env), addr, dirtysize, full, retaddr);
         flags &= ~TLB_NOTDIRTY;
     }
 
+    *pfull = *full;
     return flags;
 }
 
diff --git a/target/arm/tcg/helper-a64.c b/target/arm/tcg/helper-a64.c
index 8f42a28d07..783864d6db 100644
--- a/target/arm/tcg/helper-a64.c
+++ b/target/arm/tcg/helper-a64.c
@@ -1883,14 +1883,14 @@ static bool is_guarded_page(CPUARMState *env, target_ulong addr, uintptr_t ra)
 #ifdef CONFIG_USER_ONLY
     return page_get_flags(addr) & PAGE_BTI;
 #else
-    CPUTLBEntryFull *full;
+    CPUTLBEntryFull full;
     void *host;
     int mmu_idx = cpu_mmu_index(env_cpu(env), true);
     int flags = probe_access_full(env, addr, 0, MMU_INST_FETCH, mmu_idx,
                                   false, &host, &full, ra);
 
     assert(!(flags & TLB_INVALID_MASK));
-    return full->extra.arm.guarded;
+    return full.extra.arm.guarded;
 #endif
 }
 
diff --git a/target/arm/tcg/mte_helper.c b/target/arm/tcg/mte_helper.c
index 9d2ba287ee..870b2875af 100644
--- a/target/arm/tcg/mte_helper.c
+++ b/target/arm/tcg/mte_helper.c
@@ -83,8 +83,7 @@ uint8_t *allocation_tag_mem_probe(CPUARMState *env, int ptr_mmu_idx,
                       TARGET_PAGE_BITS - LOG2_TAG_GRANULE - 1);
     return tags + index;
 #else
-    CPUTLBEntryFull *full;
-    MemTxAttrs attrs;
+    CPUTLBEntryFull full;
     int in_page, flags;
     hwaddr ptr_paddr, tag_paddr, xlat;
     MemoryRegion *mr;
@@ -110,7 +109,7 @@ uint8_t *allocation_tag_mem_probe(CPUARMState *env, int ptr_mmu_idx,
     assert(!(flags & TLB_INVALID_MASK));
 
     /* If the virtual page MemAttr != Tagged, access unchecked. */
-    if (full->extra.arm.pte_attrs != 0xf0) {
+    if (full.extra.arm.pte_attrs != 0xf0) {
         return NULL;
     }
 
@@ -129,9 +128,7 @@ uint8_t *allocation_tag_mem_probe(CPUARMState *env, int ptr_mmu_idx,
      * Remember these values across the second lookup below,
      * which may invalidate this pointer via tlb resize.
      */
-    ptr_paddr = full->phys_addr | (ptr & ~TARGET_PAGE_MASK);
-    attrs = full->attrs;
-    full = NULL;
+    ptr_paddr = full.phys_addr | (ptr & ~TARGET_PAGE_MASK);
 
     /*
      * The Normal memory access can extend to the next page.  E.g. a single
@@ -150,17 +147,17 @@ uint8_t *allocation_tag_mem_probe(CPUARMState *env, int ptr_mmu_idx,
     if (!probe && unlikely(flags & TLB_WATCHPOINT)) {
         int wp = ptr_access == MMU_DATA_LOAD ? BP_MEM_READ : BP_MEM_WRITE;
         assert(ra != 0);
-        cpu_check_watchpoint(env_cpu(env), ptr, ptr_size, attrs, wp, ra);
+        cpu_check_watchpoint(env_cpu(env), ptr, ptr_size, full.attrs, wp, ra);
     }
 
     /* Convert to the physical address in tag space.  */
     tag_paddr = ptr_paddr >> (LOG2_TAG_GRANULE + 1);
 
     /* Look up the address in tag space. */
-    tag_asi = attrs.secure ? ARMASIdx_TagS : ARMASIdx_TagNS;
+    tag_asi = full.attrs.secure ? ARMASIdx_TagS : ARMASIdx_TagNS;
     tag_as = cpu_get_address_space(env_cpu(env), tag_asi);
     mr = address_space_translate(tag_as, tag_paddr, &xlat, NULL,
-                                 tag_access == MMU_DATA_STORE, attrs);
+                                 tag_access == MMU_DATA_STORE, full.attrs);
 
     /*
      * Note that @mr will never be NULL.  If there is nothing in the address
diff --git a/target/arm/tcg/sve_helper.c b/target/arm/tcg/sve_helper.c
index f1ee0e060f..dad0d5e518 100644
--- a/target/arm/tcg/sve_helper.c
+++ b/target/arm/tcg/sve_helper.c
@@ -5357,7 +5357,7 @@ bool sve_probe_page(SVEHostPage *info, bool nofault, CPUARMState *env,
     flags = probe_access_flags(env, addr, 0, access_type, mmu_idx, nofault,
                                &info->host, retaddr);
 #else
-    CPUTLBEntryFull *full;
+    CPUTLBEntryFull full;
     flags = probe_access_full(env, addr, 0, access_type, mmu_idx, nofault,
                               &info->host, &full, retaddr);
 #endif
@@ -5373,8 +5373,8 @@ bool sve_probe_page(SVEHostPage *info, bool nofault, CPUARMState *env,
     /* Require both ANON and MTE; see allocation_tag_mem(). */
     info->tagged = (flags & PAGE_ANON) && (flags & PAGE_MTE);
 #else
-    info->attrs = full->attrs;
-    info->tagged = full->extra.arm.pte_attrs == 0xf0;
+    info->attrs = full.attrs;
+    info->tagged = full.extra.arm.pte_attrs == 0xf0;
 #endif
 
     /* Ensure that info->host[] is relative to addr, not addr + mem_off. */
-- 
2.43.0

[PATCH v2 27/54] accel/tcg: Return CPUTLBEntryFull not pointer in probe_access_internal

[Richard Henderson]

Return a copy of the structure, not a pointer.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 40 ++++++++++++++++++----------------------
 1 file changed, 18 insertions(+), 22 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 84e7e633e3..41b2f76cc9 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -1364,7 +1364,7 @@ static void notdirty_write(CPUState *cpu, vaddr mem_vaddr, unsigned size,
 static int probe_access_internal(CPUState *cpu, vaddr addr,
                                  int fault_size, MMUAccessType access_type,
                                  int mmu_idx, bool nonfault,
-                                 void **phost, CPUTLBEntryFull **pfull,
+                                 void **phost, CPUTLBEntryFull *pfull,
                                  uintptr_t retaddr, bool check_mem_cbs)
 {
     uintptr_t index = tlb_index(cpu, mmu_idx, addr);
@@ -1379,7 +1379,7 @@ static int probe_access_internal(CPUState *cpu, vaddr addr,
                                 0, fault_size, nonfault, retaddr)) {
                 /* Non-faulting page table read failed.  */
                 *phost = NULL;
-                *pfull = NULL;
+                memset(pfull, 0, sizeof(*pfull));
                 return TLB_INVALID_MASK;
             }
 
@@ -1398,8 +1398,9 @@ static int probe_access_internal(CPUState *cpu, vaddr addr,
     }
     flags &= tlb_addr;
 
-    *pfull = full = &cpu->neg.tlb.d[mmu_idx].fulltlb[index];
+    full = &cpu->neg.tlb.d[mmu_idx].fulltlb[index];
     flags |= full->slow_flags[access_type];
+    *pfull = *full;
 
     /*
      * Fold all "mmio-like" bits, and required plugin callbacks, to TLB_MMIO.
@@ -1423,19 +1424,17 @@ int probe_access_full(CPUArchState *env, vaddr addr, int size,
                       bool nonfault, void **phost, CPUTLBEntryFull *pfull,
                       uintptr_t retaddr)
 {
-    CPUTLBEntryFull *full;
     int flags = probe_access_internal(env_cpu(env), addr, size, access_type,
-                                      mmu_idx, nonfault, phost, &full, retaddr,
+                                      mmu_idx, nonfault, phost, pfull, retaddr,
                                       true);
 
     /* Handle clean RAM pages.  */
     if (unlikely(flags & TLB_NOTDIRTY)) {
         int dirtysize = size == 0 ? 1 : size;
-        notdirty_write(env_cpu(env), addr, dirtysize, full, retaddr);
+        notdirty_write(env_cpu(env), addr, dirtysize, pfull, retaddr);
         flags &= ~TLB_NOTDIRTY;
     }
 
-    *pfull = *full;
     return flags;
 }
 
@@ -1444,25 +1443,22 @@ int probe_access_full_mmu(CPUArchState *env, vaddr addr, int size,
                           void **phost, CPUTLBEntryFull *pfull)
 {
     void *discard_phost;
-    CPUTLBEntryFull *full;
+    CPUTLBEntryFull discard_full;
 
     /* privately handle users that don't need full results */
     phost = phost ? phost : &discard_phost;
+    pfull = pfull ? pfull : &discard_full;
 
     int flags = probe_access_internal(env_cpu(env), addr, size, access_type,
-                                      mmu_idx, true, phost, &full, 0, false);
+                                      mmu_idx, true, phost, pfull, 0, false);
 
     /* Handle clean RAM pages.  */
     if (unlikely(flags & TLB_NOTDIRTY)) {
         int dirtysize = size == 0 ? 1 : size;
-        notdirty_write(env_cpu(env), addr, dirtysize, full, 0);
+        notdirty_write(env_cpu(env), addr, dirtysize, pfull, 0);
         flags &= ~TLB_NOTDIRTY;
     }
 
-    if (pfull) {
-        *pfull = *full;
-    }
-
     return flags;
 }
 
@@ -1470,7 +1466,7 @@ int probe_access_flags(CPUArchState *env, vaddr addr, int size,
                        MMUAccessType access_type, int mmu_idx,
                        bool nonfault, void **phost, uintptr_t retaddr)
 {
-    CPUTLBEntryFull *full;
+    CPUTLBEntryFull full;
     int flags;
 
     g_assert(-(addr | TARGET_PAGE_MASK) >= size);
@@ -1482,7 +1478,7 @@ int probe_access_flags(CPUArchState *env, vaddr addr, int size,
     /* Handle clean RAM pages. */
     if (unlikely(flags & TLB_NOTDIRTY)) {
         int dirtysize = size == 0 ? 1 : size;
-        notdirty_write(env_cpu(env), addr, dirtysize, full, retaddr);
+        notdirty_write(env_cpu(env), addr, dirtysize, &full, retaddr);
         flags &= ~TLB_NOTDIRTY;
     }
 
@@ -1492,7 +1488,7 @@ int probe_access_flags(CPUArchState *env, vaddr addr, int size,
 void *probe_access(CPUArchState *env, vaddr addr, int size,
                    MMUAccessType access_type, int mmu_idx, uintptr_t retaddr)
 {
-    CPUTLBEntryFull *full;
+    CPUTLBEntryFull full;
     void *host;
     int flags;
 
@@ -1513,12 +1509,12 @@ void *probe_access(CPUArchState *env, vaddr addr, int size,
             int wp_access = (access_type == MMU_DATA_STORE
                              ? BP_MEM_WRITE : BP_MEM_READ);
             cpu_check_watchpoint(env_cpu(env), addr, size,
-                                 full->attrs, wp_access, retaddr);
+                                 full.attrs, wp_access, retaddr);
         }
 
         /* Handle clean RAM pages.  */
         if (flags & TLB_NOTDIRTY) {
-            notdirty_write(env_cpu(env), addr, size, full, retaddr);
+            notdirty_write(env_cpu(env), addr, size, &full, retaddr);
         }
     }
 
@@ -1528,7 +1524,7 @@ void *probe_access(CPUArchState *env, vaddr addr, int size,
 void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr addr,
                         MMUAccessType access_type, int mmu_idx)
 {
-    CPUTLBEntryFull *full;
+    CPUTLBEntryFull full;
     void *host;
     int flags;
 
@@ -1552,7 +1548,7 @@ void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr addr,
 tb_page_addr_t get_page_addr_code_hostp(CPUArchState *env, vaddr addr,
                                         void **hostp)
 {
-    CPUTLBEntryFull *full;
+    CPUTLBEntryFull full;
     void *p;
 
     (void)probe_access_internal(env_cpu(env), addr, 1, MMU_INST_FETCH,
@@ -1562,7 +1558,7 @@ tb_page_addr_t get_page_addr_code_hostp(CPUArchState *env, vaddr addr,
         return -1;
     }
 
-    if (full->lg_page_size < TARGET_PAGE_BITS) {
+    if (full.lg_page_size < TARGET_PAGE_BITS) {
         return -1;
     }
 
-- 
2.43.0

[PATCH v2 28/54] accel/tcg: Introduce tlb_lookup

[Richard Henderson]

Unify 3 instances of tlb lookup, through tlb_hit, tlbtree_hit,
and tlb_full_align.  Use structures to avoid too many arguments.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 369 ++++++++++++++++++++++-----------------------
 1 file changed, 178 insertions(+), 191 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 41b2f76cc9..a33bebf55a 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -1271,6 +1271,118 @@ static inline void cpu_unaligned_access(CPUState *cpu, vaddr addr,
                                           mmu_idx, retaddr);
 }
 
+typedef struct TLBLookupInput {
+    vaddr addr;
+    uintptr_t ra;
+    int memop_probe           : 16;
+    unsigned int size         : 8;
+    MMUAccessType access_type : 4;
+    unsigned int mmu_idx      : 4;
+} TLBLookupInput;
+
+typedef struct TLBLookupOutput {
+    CPUTLBEntryFull full;
+    void *haddr;
+    int flags;
+    bool did_tlb_fill;
+} TLBLookupOutput;
+
+static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
+                       const TLBLookupInput *i)
+{
+    CPUTLBDesc *desc = &cpu->neg.tlb.d[i->mmu_idx];
+    CPUTLBDescFast *fast = &cpu->neg.tlb.f[i->mmu_idx];
+    vaddr addr = i->addr;
+    MMUAccessType access_type = i->access_type;
+    CPUTLBEntryFull *full;
+    CPUTLBEntryTree *node;
+    CPUTLBEntry *entry;
+    uint64_t cmp;
+    bool probe = i->memop_probe < 0;
+    MemOp memop = probe ? 0 : i->memop_probe;
+    int flags = TLB_FLAGS_MASK & ~TLB_FORCE_SLOW;
+
+    assert_cpu_is_self(cpu);
+    o->did_tlb_fill = false;
+
+    /* Primary lookup in the fast tlb. */
+    entry = tlbfast_entry(fast, addr);
+    full = &desc->fulltlb[tlbfast_index(fast, addr)];
+    cmp = tlb_read_idx(entry, access_type);
+    if (tlb_hit(cmp, addr)) {
+        goto found;
+    }
+
+    /* Secondary lookup in the IntervalTree. */
+    node = tlbtree_lookup_addr(desc, addr);
+    if (node) {
+        cmp = tlb_read_idx(&node->copy, access_type);
+        if (tlb_hit(cmp, addr)) {
+            /* Install the cached entry. */
+            qemu_spin_lock(&cpu->neg.tlb.c.lock);
+            copy_tlb_helper_locked(entry, &node->copy);
+            qemu_spin_unlock(&cpu->neg.tlb.c.lock);
+            *full = node->full;
+            goto found;
+        }
+    }
+
+    /* Finally, query the target hook. */
+    if (!tlb_fill_align(cpu, addr, access_type, i->mmu_idx,
+                        memop, i->size, probe, i->ra)) {
+        tcg_debug_assert(probe);
+        return false;
+    }
+
+    o->did_tlb_fill = true;
+
+    entry = tlbfast_entry(fast, addr);
+    full = &desc->fulltlb[tlbfast_index(fast, addr)];
+    cmp = tlb_read_idx(entry, access_type);
+    /*
+     * With PAGE_WRITE_INV, we set TLB_INVALID_MASK immediately,
+     * to force the next access through tlb_fill_align.  We've just
+     * called tlb_fill_align, so we know that this entry *is* valid.
+     */
+    flags &= ~TLB_INVALID_MASK;
+    goto done;
+
+ found:
+    /* Alignment has not been checked by tlb_fill_align. */
+    {
+        int a_bits = memop_alignment_bits(memop);
+
+        /*
+         * The TLB_CHECK_ALIGNED check differs from the normal alignment
+         * check, in that this is based on the atomicity of the operation.
+         * The intended use case is the ARM memory type field of each PTE,
+         * where access to pages with Device memory type require alignment.
+         */
+        if (unlikely(flags & TLB_CHECK_ALIGNED)) {
+            int at_bits = memop_atomicity_bits(memop);
+            a_bits = MAX(a_bits, at_bits);
+        }
+        if (unlikely(addr & ((1 << a_bits) - 1))) {
+            cpu_unaligned_access(cpu, addr, access_type, i->mmu_idx, i->ra);
+        }
+    }
+
+ done:
+    flags &= cmp;
+    flags |= full->slow_flags[access_type];
+    o->flags = flags;
+    o->full = *full;
+    o->haddr = (void *)((uintptr_t)addr + entry->addend);
+    return true;
+}
+
+static void tlb_lookup_nofail(CPUState *cpu, TLBLookupOutput *o,
+                              const TLBLookupInput *i)
+{
+    bool ok = tlb_lookup(cpu, o, i);
+    tcg_debug_assert(ok);
+}
+
 static MemoryRegionSection *
 io_prepare(hwaddr *out_offset, CPUState *cpu, hwaddr xlat,
            MemTxAttrs attrs, vaddr addr, uintptr_t retaddr)
@@ -1303,40 +1415,6 @@ static void io_failed(CPUState *cpu, CPUTLBEntryFull *full, vaddr addr,
     }
 }
 
-/*
- * Return true if ADDR is present in the interval tree,
- * and has been copied back to the main tlb.
- */
-static bool tlbtree_hit(CPUState *cpu, int mmu_idx,
-                        MMUAccessType access_type, vaddr addr)
-{
-    CPUTLBDesc *desc = &cpu->neg.tlb.d[mmu_idx];
-    CPUTLBDescFast *fast = &cpu->neg.tlb.f[mmu_idx];
-    CPUTLBEntryTree *node;
-    size_t index;
-
-    assert_cpu_is_self(cpu);
-    node = tlbtree_lookup_addr(desc, addr);
-    if (!node) {
-        /* There is no cached mapping for this page. */
-        return false;
-    }
-
-    if (!tlb_hit(tlb_read_idx(&node->copy, access_type), addr)) {
-        /* This access is not permitted. */
-        return false;
-    }
-
-    /* Install the cached entry. */
-    index = tlbfast_index(fast, addr);
-    qemu_spin_lock(&cpu->neg.tlb.c.lock);
-    copy_tlb_helper_locked(&fast->table[index], &node->copy);
-    qemu_spin_unlock(&cpu->neg.tlb.c.lock);
-
-    desc->fulltlb[index] = node->full;
-    return true;
-}
-
 static void notdirty_write(CPUState *cpu, vaddr mem_vaddr, unsigned size,
                            CPUTLBEntryFull *full, uintptr_t retaddr)
 {
@@ -1367,40 +1445,26 @@ static int probe_access_internal(CPUState *cpu, vaddr addr,
                                  void **phost, CPUTLBEntryFull *pfull,
                                  uintptr_t retaddr, bool check_mem_cbs)
 {
-    uintptr_t index = tlb_index(cpu, mmu_idx, addr);
-    CPUTLBEntry *entry = tlb_entry(cpu, mmu_idx, addr);
-    uint64_t tlb_addr = tlb_read_idx(entry, access_type);
-    int flags = TLB_FLAGS_MASK & ~TLB_FORCE_SLOW;
-    CPUTLBEntryFull *full;
+    TLBLookupInput i = {
+        .addr = addr,
+        .ra = retaddr,
+        .access_type = access_type,
+        .size = fault_size,
+        .memop_probe = nonfault ? -1 : 0,
+        .mmu_idx = mmu_idx,
+    };
+    TLBLookupOutput o;
+    int flags;
 
-    if (!tlb_hit(tlb_addr, addr)) {
-        if (!tlbtree_hit(cpu, mmu_idx, access_type, addr)) {
-            if (!tlb_fill_align(cpu, addr, access_type, mmu_idx,
-                                0, fault_size, nonfault, retaddr)) {
-                /* Non-faulting page table read failed.  */
-                *phost = NULL;
-                memset(pfull, 0, sizeof(*pfull));
-                return TLB_INVALID_MASK;
-            }
-
-            /* TLB resize via tlb_fill_align may have moved the entry.  */
-            index = tlb_index(cpu, mmu_idx, addr);
-            entry = tlb_entry(cpu, mmu_idx, addr);
-
-            /*
-             * With PAGE_WRITE_INV, we set TLB_INVALID_MASK immediately,
-             * to force the next access through tlb_fill_align.  We've just
-             * called tlb_fill_align, so we know that this entry *is* valid.
-             */
-            flags &= ~TLB_INVALID_MASK;
-        }
-        tlb_addr = tlb_read_idx(entry, access_type);
+    if (!tlb_lookup(cpu, &o, &i)) {
+        /* Non-faulting page table read failed.  */
+        *phost = NULL;
+        memset(pfull, 0, sizeof(*pfull));
+        return TLB_INVALID_MASK;
     }
-    flags &= tlb_addr;
 
-    full = &cpu->neg.tlb.d[mmu_idx].fulltlb[index];
-    flags |= full->slow_flags[access_type];
-    *pfull = *full;
+    *pfull = o.full;
+    flags = o.flags;
 
     /*
      * Fold all "mmio-like" bits, and required plugin callbacks, to TLB_MMIO.
@@ -1415,7 +1479,7 @@ static int probe_access_internal(CPUState *cpu, vaddr addr,
     }
 
     /* Everything else is RAM. */
-    *phost = (void *)((uintptr_t)addr + entry->addend);
+    *phost = o.haddr;
     return flags;
 }
 
@@ -1625,6 +1689,7 @@ typedef struct MMULookupPageData {
     vaddr addr;
     int flags;
     int size;
+    TLBLookupOutput o;
 } MMULookupPageData;
 
 typedef struct MMULookupLocals {
@@ -1644,67 +1709,25 @@ typedef struct MMULookupLocals {
  *
  * Resolve the translation for the one page at @data.addr, filling in
  * the rest of @data with the results.  If the translation fails,
- * tlb_fill_align will longjmp out.  Return true if the softmmu tlb for
- * @mmu_idx may have resized.
+ * tlb_fill_align will longjmp out.
  */
-static bool mmu_lookup1(CPUState *cpu, MMULookupPageData *data, MemOp memop,
+static void mmu_lookup1(CPUState *cpu, MMULookupPageData *data, MemOp memop,
                         int mmu_idx, MMUAccessType access_type, uintptr_t ra)
 {
-    vaddr addr = data->addr;
-    uintptr_t index = tlb_index(cpu, mmu_idx, addr);
-    CPUTLBEntry *entry = tlb_entry(cpu, mmu_idx, addr);
-    uint64_t tlb_addr = tlb_read_idx(entry, access_type);
-    bool maybe_resized = false;
-    CPUTLBEntryFull *full;
-    int flags = TLB_FLAGS_MASK & ~TLB_FORCE_SLOW;
+    TLBLookupInput i = {
+        .addr = data->addr,
+        .ra = ra,
+        .access_type = access_type,
+        .memop_probe = memop,
+        .size = data->size,
+        .mmu_idx = mmu_idx,
+    };
 
-    /* If the TLB entry is for a different page, reload and try again.  */
-    if (!tlb_hit(tlb_addr, addr)) {
-        if (!tlbtree_hit(cpu, mmu_idx, access_type, addr)) {
-            tlb_fill_align(cpu, addr, access_type, mmu_idx,
-                           memop, data->size, false, ra);
-            maybe_resized = true;
-            index = tlb_index(cpu, mmu_idx, addr);
-            entry = tlb_entry(cpu, mmu_idx, addr);
-            /*
-             * With PAGE_WRITE_INV, we set TLB_INVALID_MASK immediately,
-             * to force the next access through tlb_fill.  We've just
-             * called tlb_fill, so we know that this entry *is* valid.
-             */
-            flags &= ~TLB_INVALID_MASK;
-        }
-        tlb_addr = tlb_read_idx(entry, access_type);
-    }
+    tlb_lookup_nofail(cpu, &data->o, &i);
 
-    full = &cpu->neg.tlb.d[mmu_idx].fulltlb[index];
-    flags = tlb_addr & (TLB_FLAGS_MASK & ~TLB_FORCE_SLOW);
-    flags |= full->slow_flags[access_type];
-
-    if (likely(!maybe_resized)) {
-        /* Alignment has not been checked by tlb_fill_align. */
-        int a_bits = memop_alignment_bits(memop);
-
-        /*
-         * This alignment check differs from the one above, in that this is
-         * based on the atomicity of the operation. The intended use case is
-         * the ARM memory type field of each PTE, where access to pages with
-         * Device memory type require alignment.
-         */
-        if (unlikely(flags & TLB_CHECK_ALIGNED)) {
-            int at_bits = memop_atomicity_bits(memop);
-            a_bits = MAX(a_bits, at_bits);
-        }
-        if (unlikely(addr & ((1 << a_bits) - 1))) {
-            cpu_unaligned_access(cpu, addr, access_type, mmu_idx, ra);
-        }
-    }
-
-    data->full = full;
-    data->flags = flags;
-    /* Compute haddr speculatively; depending on flags it might be invalid. */
-    data->haddr = (void *)((uintptr_t)addr + entry->addend);
-
-    return maybe_resized;
+    data->full = &data->o.full;
+    data->flags = data->o.flags;
+    data->haddr = data->o.haddr;
 }
 
 /**
@@ -1785,15 +1808,9 @@ static bool mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
         l->page[1].size = l->page[0].size - size0;
         l->page[0].size = size0;
 
-        /*
-         * Lookup both pages, recognizing exceptions from either.  If the
-         * second lookup potentially resized, refresh first CPUTLBEntryFull.
-         */
+        /* Lookup both pages, recognizing exceptions from either. */
         mmu_lookup1(cpu, &l->page[0], l->memop, l->mmu_idx, type, ra);
-        if (mmu_lookup1(cpu, &l->page[1], 0, l->mmu_idx, type, ra)) {
-            uintptr_t index = tlb_index(cpu, l->mmu_idx, addr);
-            l->page[0].full = &cpu->neg.tlb.d[l->mmu_idx].fulltlb[index];
-        }
+        mmu_lookup1(cpu, &l->page[1], 0, l->mmu_idx, type, ra);
 
         flags = l->page[0].flags | l->page[1].flags;
         if (unlikely(flags & (TLB_WATCHPOINT | TLB_NOTDIRTY))) {
@@ -1819,49 +1836,26 @@ static bool mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
 static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
                                int size, uintptr_t retaddr)
 {
-    uintptr_t mmu_idx = get_mmuidx(oi);
-    MemOp mop = get_memop(oi);
-    uintptr_t index;
-    CPUTLBEntry *tlbe;
-    void *hostaddr;
-    CPUTLBEntryFull *full;
-    bool did_tlb_fill = false;
-    int flags;
+    TLBLookupInput i = {
+        .addr = addr,
+        .ra = retaddr - GETPC_ADJ,
+        .access_type = MMU_DATA_STORE,
+        .memop_probe = get_memop(oi),
+        .mmu_idx = get_mmuidx(oi),
+    };
+    TLBLookupOutput o;
+    int flags, wp_flags;
 
-    tcg_debug_assert(mmu_idx < NB_MMU_MODES);
-
-    /* Adjust the given return address.  */
-    retaddr -= GETPC_ADJ;
-
-    index = tlb_index(cpu, mmu_idx, addr);
-    tlbe = tlb_entry(cpu, mmu_idx, addr);
-
-    /* Check TLB entry and enforce page permissions.  */
-    flags = TLB_FLAGS_MASK;
-    if (!tlb_hit(tlb_addr_write(tlbe), addr)) {
-        if (!tlbtree_hit(cpu, mmu_idx, MMU_DATA_STORE, addr)) {
-            tlb_fill_align(cpu, addr, MMU_DATA_STORE, mmu_idx,
-                           mop, size, false, retaddr);
-            did_tlb_fill = true;
-            index = tlb_index(cpu, mmu_idx, addr);
-            tlbe = tlb_entry(cpu, mmu_idx, addr);
-            /*
-             * With PAGE_WRITE_INV, we set TLB_INVALID_MASK immediately,
-             * to force the next access through tlb_fill.  We've just
-             * called tlb_fill, so we know that this entry *is* valid.
-             */
-            flags &= ~TLB_INVALID_MASK;
-        }
-    }
-    full = &cpu->neg.tlb.d[mmu_idx].fulltlb[index];
+    i.size = memop_size(i.memop_probe);
+    tlb_lookup_nofail(cpu, &o, &i);
 
     /*
      * Let the guest notice RMW on a write-only page.
      * We have just verified that the page is writable.
      */
-    if (unlikely(!(full->prot & PAGE_READ))) {
-        tlb_fill_align(cpu, addr, MMU_DATA_LOAD, mmu_idx,
-                       0, size, false, retaddr);
+    if (unlikely(!(o.full.prot & PAGE_READ))) {
+        tlb_fill_align(cpu, addr, MMU_DATA_LOAD, i.mmu_idx,
+                       0, i.size, false, i.ra);
         /*
          * Since we don't support reads and writes to different
          * addresses, and we do have the proper page loaded for
@@ -1871,12 +1865,13 @@ static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
     }
 
     /* Enforce guest required alignment, if not handled by tlb_fill_align. */
-    if (!did_tlb_fill && (addr & ((1 << memop_alignment_bits(mop)) - 1))) {
-        cpu_unaligned_access(cpu, addr, MMU_DATA_STORE, mmu_idx, retaddr);
+    if (!o.did_tlb_fill
+        && (addr & ((1 << memop_alignment_bits(i.memop_probe)) - 1))) {
+        cpu_unaligned_access(cpu, addr, MMU_DATA_STORE, i.mmu_idx, i.ra);
     }
 
     /* Enforce qemu required alignment.  */
-    if (unlikely(addr & (size - 1))) {
+    if (unlikely(addr & (i.size - 1))) {
         /*
          * We get here if guest alignment was not requested, or was not
          * enforced by cpu_unaligned_access or tlb_fill_align above.
@@ -1886,41 +1881,33 @@ static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
         goto stop_the_world;
     }
 
-    /* Collect tlb flags for read and write. */
-    flags &= tlbe->addr_read | tlb_addr_write(tlbe);
-
     /* Notice an IO access or a needs-MMU-lookup access */
+    flags = o.flags;
     if (unlikely(flags & (TLB_MMIO | TLB_DISCARD_WRITE))) {
         /* There's really nothing that can be done to
            support this apart from stop-the-world.  */
         goto stop_the_world;
     }
 
-    hostaddr = (void *)((uintptr_t)addr + tlbe->addend);
-
     if (unlikely(flags & TLB_NOTDIRTY)) {
-        notdirty_write(cpu, addr, size, full, retaddr);
+        notdirty_write(cpu, addr, i.size, &o.full, i.ra);
     }
 
-    if (unlikely(flags & TLB_FORCE_SLOW)) {
-        int wp_flags = 0;
-
-        if (full->slow_flags[MMU_DATA_STORE] & TLB_WATCHPOINT) {
-            wp_flags |= BP_MEM_WRITE;
-        }
-        if (full->slow_flags[MMU_DATA_LOAD] & TLB_WATCHPOINT) {
-            wp_flags |= BP_MEM_READ;
-        }
-        if (wp_flags) {
-            cpu_check_watchpoint(cpu, addr, size,
-                                 full->attrs, wp_flags, retaddr);
-        }
+    wp_flags = 0;
+    if (flags & TLB_WATCHPOINT) {
+        wp_flags |= BP_MEM_WRITE;
+    }
+    if (o.full.slow_flags[MMU_DATA_LOAD] & TLB_WATCHPOINT) {
+        wp_flags |= BP_MEM_READ;
+    }
+    if (unlikely(wp_flags)) {
+        cpu_check_watchpoint(cpu, addr, i.size, o.full.attrs, wp_flags, i.ra);
     }
 
-    return hostaddr;
+    return o.haddr;
 
  stop_the_world:
-    cpu_loop_exit_atomic(cpu, retaddr);
+    cpu_loop_exit_atomic(cpu, i.ra);
 }
 
 /*
-- 
2.43.0

[PATCH v2 29/54] accel/tcg: Partially unify MMULookupPageData and TLBLookupOutput

[Richard Henderson]

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 151 ++++++++++++++++++++++-----------------------
 1 file changed, 74 insertions(+), 77 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index a33bebf55a..8f459be5a8 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -1684,10 +1684,7 @@ bool tlb_plugin_lookup(CPUState *cpu, vaddr addr, int mmu_idx,
  */
 
 typedef struct MMULookupPageData {
-    CPUTLBEntryFull *full;
-    void *haddr;
     vaddr addr;
-    int flags;
     int size;
     TLBLookupOutput o;
 } MMULookupPageData;
@@ -1724,10 +1721,6 @@ static void mmu_lookup1(CPUState *cpu, MMULookupPageData *data, MemOp memop,
     };
 
     tlb_lookup_nofail(cpu, &data->o, &i);
-
-    data->full = &data->o.full;
-    data->flags = data->o.flags;
-    data->haddr = data->o.haddr;
 }
 
 /**
@@ -1743,24 +1736,22 @@ static void mmu_lookup1(CPUState *cpu, MMULookupPageData *data, MemOp memop,
 static void mmu_watch_or_dirty(CPUState *cpu, MMULookupPageData *data,
                                MMUAccessType access_type, uintptr_t ra)
 {
-    CPUTLBEntryFull *full = data->full;
-    vaddr addr = data->addr;
-    int flags = data->flags;
-    int size = data->size;
+    int flags = data->o.flags;
 
     /* On watchpoint hit, this will longjmp out.  */
     if (flags & TLB_WATCHPOINT) {
         int wp = access_type == MMU_DATA_STORE ? BP_MEM_WRITE : BP_MEM_READ;
-        cpu_check_watchpoint(cpu, addr, size, full->attrs, wp, ra);
+        cpu_check_watchpoint(cpu, data->addr, data->size,
+                             data->o.full.attrs, wp, ra);
         flags &= ~TLB_WATCHPOINT;
     }
 
     /* Note that notdirty is only set for writes. */
     if (flags & TLB_NOTDIRTY) {
-        notdirty_write(cpu, addr, size, full, ra);
+        notdirty_write(cpu, data->addr, data->size, &data->o.full, ra);
         flags &= ~TLB_NOTDIRTY;
     }
-    data->flags = flags;
+    data->o.flags = flags;
 }
 
 /**
@@ -1795,7 +1786,7 @@ static bool mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
     if (likely(!crosspage)) {
         mmu_lookup1(cpu, &l->page[0], l->memop, l->mmu_idx, type, ra);
 
-        flags = l->page[0].flags;
+        flags = l->page[0].o.flags;
         if (unlikely(flags & (TLB_WATCHPOINT | TLB_NOTDIRTY))) {
             mmu_watch_or_dirty(cpu, &l->page[0], type, ra);
         }
@@ -1812,7 +1803,7 @@ static bool mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
         mmu_lookup1(cpu, &l->page[0], l->memop, l->mmu_idx, type, ra);
         mmu_lookup1(cpu, &l->page[1], 0, l->mmu_idx, type, ra);
 
-        flags = l->page[0].flags | l->page[1].flags;
+        flags = l->page[0].o.flags | l->page[1].o.flags;
         if (unlikely(flags & (TLB_WATCHPOINT | TLB_NOTDIRTY))) {
             mmu_watch_or_dirty(cpu, &l->page[0], type, ra);
             mmu_watch_or_dirty(cpu, &l->page[1], type, ra);
@@ -2029,7 +2020,7 @@ static Int128 do_ld16_mmio_beN(CPUState *cpu, CPUTLBEntryFull *full,
  */
 static uint64_t do_ld_bytes_beN(MMULookupPageData *p, uint64_t ret_be)
 {
-    uint8_t *haddr = p->haddr;
+    uint8_t *haddr = p->o.haddr;
     int i, size = p->size;
 
     for (i = 0; i < size; i++) {
@@ -2047,7 +2038,7 @@ static uint64_t do_ld_bytes_beN(MMULookupPageData *p, uint64_t ret_be)
  */
 static uint64_t do_ld_parts_beN(MMULookupPageData *p, uint64_t ret_be)
 {
-    void *haddr = p->haddr;
+    void *haddr = p->o.haddr;
     int size = p->size;
 
     do {
@@ -2097,7 +2088,7 @@ static uint64_t do_ld_parts_beN(MMULookupPageData *p, uint64_t ret_be)
 static uint64_t do_ld_whole_be4(MMULookupPageData *p, uint64_t ret_be)
 {
     int o = p->addr & 3;
-    uint32_t x = load_atomic4(p->haddr - o);
+    uint32_t x = load_atomic4(p->o.haddr - o);
 
     x = cpu_to_be32(x);
     x <<= o * 8;
@@ -2117,7 +2108,7 @@ static uint64_t do_ld_whole_be8(CPUState *cpu, uintptr_t ra,
                                 MMULookupPageData *p, uint64_t ret_be)
 {
     int o = p->addr & 7;
-    uint64_t x = load_atomic8_or_exit(cpu, ra, p->haddr - o);
+    uint64_t x = load_atomic8_or_exit(cpu, ra, p->o.haddr - o);
 
     x = cpu_to_be64(x);
     x <<= o * 8;
@@ -2137,7 +2128,7 @@ static Int128 do_ld_whole_be16(CPUState *cpu, uintptr_t ra,
                                MMULookupPageData *p, uint64_t ret_be)
 {
     int o = p->addr & 15;
-    Int128 x, y = load_atomic16_or_exit(cpu, ra, p->haddr - o);
+    Int128 x, y = load_atomic16_or_exit(cpu, ra, p->o.haddr - o);
     int size = p->size;
 
     if (!HOST_BIG_ENDIAN) {
@@ -2160,8 +2151,8 @@ static uint64_t do_ld_beN(CPUState *cpu, MMULookupPageData *p,
     MemOp atom;
     unsigned tmp, half_size;
 
-    if (unlikely(p->flags & TLB_MMIO)) {
-        return do_ld_mmio_beN(cpu, p->full, ret_be, p->addr, p->size,
+    if (unlikely(p->o.flags & TLB_MMIO)) {
+        return do_ld_mmio_beN(cpu, &p->o.full, ret_be, p->addr, p->size,
                               mmu_idx, type, ra);
     }
 
@@ -2210,8 +2201,9 @@ static Int128 do_ld16_beN(CPUState *cpu, MMULookupPageData *p,
     uint64_t b;
     MemOp atom;
 
-    if (unlikely(p->flags & TLB_MMIO)) {
-        return do_ld16_mmio_beN(cpu, p->full, a, p->addr, size, mmu_idx, ra);
+    if (unlikely(p->o.flags & TLB_MMIO)) {
+        return do_ld16_mmio_beN(cpu, &p->o.full, a, p->addr,
+                                size, mmu_idx, ra);
     }
 
     /*
@@ -2223,7 +2215,7 @@ static Int128 do_ld16_beN(CPUState *cpu, MMULookupPageData *p,
     case MO_ATOM_SUBALIGN:
         p->size = size - 8;
         a = do_ld_parts_beN(p, a);
-        p->haddr += size - 8;
+        p->o.haddr += size - 8;
         p->size = 8;
         b = do_ld_parts_beN(p, 0);
         break;
@@ -2242,7 +2234,7 @@ static Int128 do_ld16_beN(CPUState *cpu, MMULookupPageData *p,
     case MO_ATOM_NONE:
         p->size = size - 8;
         a = do_ld_bytes_beN(p, a);
-        b = ldq_be_p(p->haddr + size - 8);
+        b = ldq_be_p(p->o.haddr + size - 8);
         break;
 
     default:
@@ -2255,10 +2247,11 @@ static Int128 do_ld16_beN(CPUState *cpu, MMULookupPageData *p,
 static uint8_t do_ld_1(CPUState *cpu, MMULookupPageData *p, int mmu_idx,
                        MMUAccessType type, uintptr_t ra)
 {
-    if (unlikely(p->flags & TLB_MMIO)) {
-        return do_ld_mmio_beN(cpu, p->full, 0, p->addr, 1, mmu_idx, type, ra);
+    if (unlikely(p->o.flags & TLB_MMIO)) {
+        return do_ld_mmio_beN(cpu, &p->o.full, 0, p->addr, 1,
+                              mmu_idx, type, ra);
     } else {
-        return *(uint8_t *)p->haddr;
+        return *(uint8_t *)p->o.haddr;
     }
 }
 
@@ -2267,14 +2260,15 @@ static uint16_t do_ld_2(CPUState *cpu, MMULookupPageData *p, int mmu_idx,
 {
     uint16_t ret;
 
-    if (unlikely(p->flags & TLB_MMIO)) {
-        ret = do_ld_mmio_beN(cpu, p->full, 0, p->addr, 2, mmu_idx, type, ra);
+    if (unlikely(p->o.flags & TLB_MMIO)) {
+        ret = do_ld_mmio_beN(cpu, &p->o.full, 0, p->addr, 2,
+                             mmu_idx, type, ra);
         if ((memop & MO_BSWAP) == MO_LE) {
             ret = bswap16(ret);
         }
     } else {
         /* Perform the load host endian, then swap if necessary. */
-        ret = load_atom_2(cpu, ra, p->haddr, memop);
+        ret = load_atom_2(cpu, ra, p->o.haddr, memop);
         if (memop & MO_BSWAP) {
             ret = bswap16(ret);
         }
@@ -2287,14 +2281,15 @@ static uint32_t do_ld_4(CPUState *cpu, MMULookupPageData *p, int mmu_idx,
 {
     uint32_t ret;
 
-    if (unlikely(p->flags & TLB_MMIO)) {
-        ret = do_ld_mmio_beN(cpu, p->full, 0, p->addr, 4, mmu_idx, type, ra);
+    if (unlikely(p->o.flags & TLB_MMIO)) {
+        ret = do_ld_mmio_beN(cpu, &p->o.full, 0, p->addr, 4,
+                             mmu_idx, type, ra);
         if ((memop & MO_BSWAP) == MO_LE) {
             ret = bswap32(ret);
         }
     } else {
         /* Perform the load host endian. */
-        ret = load_atom_4(cpu, ra, p->haddr, memop);
+        ret = load_atom_4(cpu, ra, p->o.haddr, memop);
         if (memop & MO_BSWAP) {
             ret = bswap32(ret);
         }
@@ -2307,14 +2302,15 @@ static uint64_t do_ld_8(CPUState *cpu, MMULookupPageData *p, int mmu_idx,
 {
     uint64_t ret;
 
-    if (unlikely(p->flags & TLB_MMIO)) {
-        ret = do_ld_mmio_beN(cpu, p->full, 0, p->addr, 8, mmu_idx, type, ra);
+    if (unlikely(p->o.flags & TLB_MMIO)) {
+        ret = do_ld_mmio_beN(cpu, &p->o.full, 0, p->addr, 8,
+                             mmu_idx, type, ra);
         if ((memop & MO_BSWAP) == MO_LE) {
             ret = bswap64(ret);
         }
     } else {
         /* Perform the load host endian. */
-        ret = load_atom_8(cpu, ra, p->haddr, memop);
+        ret = load_atom_8(cpu, ra, p->o.haddr, memop);
         if (memop & MO_BSWAP) {
             ret = bswap64(ret);
         }
@@ -2414,15 +2410,15 @@ static Int128 do_ld16_mmu(CPUState *cpu, vaddr addr,
     cpu_req_mo(TCG_MO_LD_LD | TCG_MO_ST_LD);
     crosspage = mmu_lookup(cpu, addr, oi, ra, MMU_DATA_LOAD, &l);
     if (likely(!crosspage)) {
-        if (unlikely(l.page[0].flags & TLB_MMIO)) {
-            ret = do_ld16_mmio_beN(cpu, l.page[0].full, 0, addr, 16,
+        if (unlikely(l.page[0].o.flags & TLB_MMIO)) {
+            ret = do_ld16_mmio_beN(cpu, &l.page[0].o.full, 0, addr, 16,
                                    l.mmu_idx, ra);
             if ((l.memop & MO_BSWAP) == MO_LE) {
                 ret = bswap128(ret);
             }
         } else {
             /* Perform the load host endian. */
-            ret = load_atom_16(cpu, ra, l.page[0].haddr, l.memop);
+            ret = load_atom_16(cpu, ra, l.page[0].o.haddr, l.memop);
             if (l.memop & MO_BSWAP) {
                 ret = bswap128(ret);
             }
@@ -2568,10 +2564,10 @@ static uint64_t do_st_leN(CPUState *cpu, MMULookupPageData *p,
     MemOp atom;
     unsigned tmp, half_size;
 
-    if (unlikely(p->flags & TLB_MMIO)) {
-        return do_st_mmio_leN(cpu, p->full, val_le, p->addr,
+    if (unlikely(p->o.flags & TLB_MMIO)) {
+        return do_st_mmio_leN(cpu, &p->o.full, val_le, p->addr,
                               p->size, mmu_idx, ra);
-    } else if (unlikely(p->flags & TLB_DISCARD_WRITE)) {
+    } else if (unlikely(p->o.flags & TLB_DISCARD_WRITE)) {
         return val_le >> (p->size * 8);
     }
 
@@ -2582,7 +2578,7 @@ static uint64_t do_st_leN(CPUState *cpu, MMULookupPageData *p,
     atom = mop & MO_ATOM_MASK;
     switch (atom) {
     case MO_ATOM_SUBALIGN:
-        return store_parts_leN(p->haddr, p->size, val_le);
+        return store_parts_leN(p->o.haddr, p->size, val_le);
 
     case MO_ATOM_IFALIGN_PAIR:
     case MO_ATOM_WITHIN16_PAIR:
@@ -2593,9 +2589,9 @@ static uint64_t do_st_leN(CPUState *cpu, MMULookupPageData *p,
             ? p->size == half_size
             : p->size >= half_size) {
             if (!HAVE_al8_fast && p->size <= 4) {
-                return store_whole_le4(p->haddr, p->size, val_le);
+                return store_whole_le4(p->o.haddr, p->size, val_le);
             } else if (HAVE_al8) {
-                return store_whole_le8(p->haddr, p->size, val_le);
+                return store_whole_le8(p->o.haddr, p->size, val_le);
             } else {
                 cpu_loop_exit_atomic(cpu, ra);
             }
@@ -2605,7 +2601,7 @@ static uint64_t do_st_leN(CPUState *cpu, MMULookupPageData *p,
     case MO_ATOM_IFALIGN:
     case MO_ATOM_WITHIN16:
     case MO_ATOM_NONE:
-        return store_bytes_leN(p->haddr, p->size, val_le);
+        return store_bytes_leN(p->o.haddr, p->size, val_le);
 
     default:
         g_assert_not_reached();
@@ -2622,10 +2618,10 @@ static uint64_t do_st16_leN(CPUState *cpu, MMULookupPageData *p,
     int size = p->size;
     MemOp atom;
 
-    if (unlikely(p->flags & TLB_MMIO)) {
-        return do_st16_mmio_leN(cpu, p->full, val_le, p->addr,
+    if (unlikely(p->o.flags & TLB_MMIO)) {
+        return do_st16_mmio_leN(cpu, &p->o.full, val_le, p->addr,
                                 size, mmu_idx, ra);
-    } else if (unlikely(p->flags & TLB_DISCARD_WRITE)) {
+    } else if (unlikely(p->o.flags & TLB_DISCARD_WRITE)) {
         return int128_gethi(val_le) >> ((size - 8) * 8);
     }
 
@@ -2636,8 +2632,8 @@ static uint64_t do_st16_leN(CPUState *cpu, MMULookupPageData *p,
     atom = mop & MO_ATOM_MASK;
     switch (atom) {
     case MO_ATOM_SUBALIGN:
-        store_parts_leN(p->haddr, 8, int128_getlo(val_le));
-        return store_parts_leN(p->haddr + 8, p->size - 8,
+        store_parts_leN(p->o.haddr, 8, int128_getlo(val_le));
+        return store_parts_leN(p->o.haddr + 8, p->size - 8,
                                int128_gethi(val_le));
 
     case MO_ATOM_WITHIN16_PAIR:
@@ -2645,7 +2641,7 @@ static uint64_t do_st16_leN(CPUState *cpu, MMULookupPageData *p,
         if (!HAVE_CMPXCHG128) {
             cpu_loop_exit_atomic(cpu, ra);
         }
-        return store_whole_le16(p->haddr, p->size, val_le);
+        return store_whole_le16(p->o.haddr, p->size, val_le);
 
     case MO_ATOM_IFALIGN_PAIR:
         /*
@@ -2655,8 +2651,8 @@ static uint64_t do_st16_leN(CPUState *cpu, MMULookupPageData *p,
     case MO_ATOM_IFALIGN:
     case MO_ATOM_WITHIN16:
     case MO_ATOM_NONE:
-        stq_le_p(p->haddr, int128_getlo(val_le));
-        return store_bytes_leN(p->haddr + 8, p->size - 8,
+        stq_le_p(p->o.haddr, int128_getlo(val_le));
+        return store_bytes_leN(p->o.haddr + 8, p->size - 8,
                                int128_gethi(val_le));
 
     default:
@@ -2667,69 +2663,69 @@ static uint64_t do_st16_leN(CPUState *cpu, MMULookupPageData *p,
 static void do_st_1(CPUState *cpu, MMULookupPageData *p, uint8_t val,
                     int mmu_idx, uintptr_t ra)
 {
-    if (unlikely(p->flags & TLB_MMIO)) {
-        do_st_mmio_leN(cpu, p->full, val, p->addr, 1, mmu_idx, ra);
-    } else if (unlikely(p->flags & TLB_DISCARD_WRITE)) {
+    if (unlikely(p->o.flags & TLB_MMIO)) {
+        do_st_mmio_leN(cpu, &p->o.full, val, p->addr, 1, mmu_idx, ra);
+    } else if (unlikely(p->o.flags & TLB_DISCARD_WRITE)) {
         /* nothing */
     } else {
-        *(uint8_t *)p->haddr = val;
+        *(uint8_t *)p->o.haddr = val;
     }
 }
 
 static void do_st_2(CPUState *cpu, MMULookupPageData *p, uint16_t val,
                     int mmu_idx, MemOp memop, uintptr_t ra)
 {
-    if (unlikely(p->flags & TLB_MMIO)) {
+    if (unlikely(p->o.flags & TLB_MMIO)) {
         if ((memop & MO_BSWAP) != MO_LE) {
             val = bswap16(val);
         }
-        do_st_mmio_leN(cpu, p->full, val, p->addr, 2, mmu_idx, ra);
-    } else if (unlikely(p->flags & TLB_DISCARD_WRITE)) {
+        do_st_mmio_leN(cpu, &p->o.full, val, p->addr, 2, mmu_idx, ra);
+    } else if (unlikely(p->o.flags & TLB_DISCARD_WRITE)) {
         /* nothing */
     } else {
         /* Swap to host endian if necessary, then store. */
         if (memop & MO_BSWAP) {
             val = bswap16(val);
         }
-        store_atom_2(cpu, ra, p->haddr, memop, val);
+        store_atom_2(cpu, ra, p->o.haddr, memop, val);
     }
 }
 
 static void do_st_4(CPUState *cpu, MMULookupPageData *p, uint32_t val,
                     int mmu_idx, MemOp memop, uintptr_t ra)
 {
-    if (unlikely(p->flags & TLB_MMIO)) {
+    if (unlikely(p->o.flags & TLB_MMIO)) {
         if ((memop & MO_BSWAP) != MO_LE) {
             val = bswap32(val);
         }
-        do_st_mmio_leN(cpu, p->full, val, p->addr, 4, mmu_idx, ra);
-    } else if (unlikely(p->flags & TLB_DISCARD_WRITE)) {
+        do_st_mmio_leN(cpu, &p->o.full, val, p->addr, 4, mmu_idx, ra);
+    } else if (unlikely(p->o.flags & TLB_DISCARD_WRITE)) {
         /* nothing */
     } else {
         /* Swap to host endian if necessary, then store. */
         if (memop & MO_BSWAP) {
             val = bswap32(val);
         }
-        store_atom_4(cpu, ra, p->haddr, memop, val);
+        store_atom_4(cpu, ra, p->o.haddr, memop, val);
     }
 }
 
 static void do_st_8(CPUState *cpu, MMULookupPageData *p, uint64_t val,
                     int mmu_idx, MemOp memop, uintptr_t ra)
 {
-    if (unlikely(p->flags & TLB_MMIO)) {
+    if (unlikely(p->o.flags & TLB_MMIO)) {
         if ((memop & MO_BSWAP) != MO_LE) {
             val = bswap64(val);
         }
-        do_st_mmio_leN(cpu, p->full, val, p->addr, 8, mmu_idx, ra);
-    } else if (unlikely(p->flags & TLB_DISCARD_WRITE)) {
+        do_st_mmio_leN(cpu, &p->o.full, val, p->addr, 8, mmu_idx, ra);
+    } else if (unlikely(p->o.flags & TLB_DISCARD_WRITE)) {
         /* nothing */
     } else {
         /* Swap to host endian if necessary, then store. */
         if (memop & MO_BSWAP) {
             val = bswap64(val);
         }
-        store_atom_8(cpu, ra, p->haddr, memop, val);
+        store_atom_8(cpu, ra, p->o.haddr, memop, val);
     }
 }
 
@@ -2822,19 +2818,20 @@ static void do_st16_mmu(CPUState *cpu, vaddr addr, Int128 val,
     cpu_req_mo(TCG_MO_LD_ST | TCG_MO_ST_ST);
     crosspage = mmu_lookup(cpu, addr, oi, ra, MMU_DATA_STORE, &l);
     if (likely(!crosspage)) {
-        if (unlikely(l.page[0].flags & TLB_MMIO)) {
+        if (unlikely(l.page[0].o.flags & TLB_MMIO)) {
             if ((l.memop & MO_BSWAP) != MO_LE) {
                 val = bswap128(val);
             }
-            do_st16_mmio_leN(cpu, l.page[0].full, val, addr, 16, l.mmu_idx, ra);
-        } else if (unlikely(l.page[0].flags & TLB_DISCARD_WRITE)) {
+            do_st16_mmio_leN(cpu, &l.page[0].o.full, val, addr,
+                             16, l.mmu_idx, ra);
+        } else if (unlikely(l.page[0].o.flags & TLB_DISCARD_WRITE)) {
             /* nothing */
         } else {
             /* Swap to host endian if necessary, then store. */
             if (l.memop & MO_BSWAP) {
                 val = bswap128(val);
             }
-            store_atom_16(cpu, ra, l.page[0].haddr, l.memop, val);
+            store_atom_16(cpu, ra, l.page[0].o.haddr, l.memop, val);
         }
         return;
     }
-- 
2.43.0

[PATCH v2 30/54] accel/tcg: Merge mmu_lookup1 into mmu_lookup

[Richard Henderson]

Reuse most of TLBLookupInput between calls to tlb_lookup.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 65 ++++++++++++++++++----------------------------
 1 file changed, 25 insertions(+), 40 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 8f459be5a8..981098a6f2 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -1695,34 +1695,6 @@ typedef struct MMULookupLocals {
     int mmu_idx;
 } MMULookupLocals;
 
-/**
- * mmu_lookup1: translate one page
- * @cpu: generic cpu state
- * @data: lookup parameters
- * @memop: memory operation for the access, or 0
- * @mmu_idx: virtual address context
- * @access_type: load/store/code
- * @ra: return address into tcg generated code, or 0
- *
- * Resolve the translation for the one page at @data.addr, filling in
- * the rest of @data with the results.  If the translation fails,
- * tlb_fill_align will longjmp out.
- */
-static void mmu_lookup1(CPUState *cpu, MMULookupPageData *data, MemOp memop,
-                        int mmu_idx, MMUAccessType access_type, uintptr_t ra)
-{
-    TLBLookupInput i = {
-        .addr = data->addr,
-        .ra = ra,
-        .access_type = access_type,
-        .memop_probe = memop,
-        .size = data->size,
-        .mmu_idx = mmu_idx,
-    };
-
-    tlb_lookup_nofail(cpu, &data->o, &i);
-}
-
 /**
  * mmu_watch_or_dirty
  * @cpu: generic cpu state
@@ -1769,26 +1741,36 @@ static void mmu_watch_or_dirty(CPUState *cpu, MMULookupPageData *data,
 static bool mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
                        uintptr_t ra, MMUAccessType type, MMULookupLocals *l)
 {
+    MemOp memop = get_memop(oi);
+    int mmu_idx = get_mmuidx(oi);
+    TLBLookupInput i = {
+        .addr = addr,
+        .ra = ra,
+        .access_type = type,
+        .memop_probe = memop,
+        .size = memop_size(memop),
+        .mmu_idx = mmu_idx,
+    };
     bool crosspage;
     int flags;
 
-    l->memop = get_memop(oi);
-    l->mmu_idx = get_mmuidx(oi);
+    l->memop = memop;
+    l->mmu_idx = mmu_idx;
 
-    tcg_debug_assert(l->mmu_idx < NB_MMU_MODES);
+    tcg_debug_assert(mmu_idx < NB_MMU_MODES);
 
     l->page[0].addr = addr;
-    l->page[0].size = memop_size(l->memop);
-    l->page[1].addr = (addr + l->page[0].size - 1) & TARGET_PAGE_MASK;
+    l->page[0].size = i.size;
+    l->page[1].addr = (addr + i.size - 1) & TARGET_PAGE_MASK;
     l->page[1].size = 0;
     crosspage = (addr ^ l->page[1].addr) & TARGET_PAGE_MASK;
 
     if (likely(!crosspage)) {
-        mmu_lookup1(cpu, &l->page[0], l->memop, l->mmu_idx, type, ra);
+        tlb_lookup_nofail(cpu, &l->page[0].o, &i);
 
         flags = l->page[0].o.flags;
         if (unlikely(flags & (TLB_WATCHPOINT | TLB_NOTDIRTY))) {
-            mmu_watch_or_dirty(cpu, &l->page[0], type, ra);
+            mmu_watch_or_dirty(cpu, &l->page[0], i.access_type, i.ra);
         }
         if (unlikely(flags & TLB_BSWAP)) {
             l->memop ^= MO_BSWAP;
@@ -1796,17 +1778,20 @@ static bool mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
     } else {
         /* Finish compute of page crossing. */
         int size0 = l->page[1].addr - addr;
-        l->page[1].size = l->page[0].size - size0;
+        l->page[1].size = i.size - size0;
         l->page[0].size = size0;
 
         /* Lookup both pages, recognizing exceptions from either. */
-        mmu_lookup1(cpu, &l->page[0], l->memop, l->mmu_idx, type, ra);
-        mmu_lookup1(cpu, &l->page[1], 0, l->mmu_idx, type, ra);
+        i.size = size0;
+        tlb_lookup_nofail(cpu, &l->page[0].o, &i);
+        i.addr = l->page[1].addr;
+        i.size = l->page[1].size;
+        tlb_lookup_nofail(cpu, &l->page[1].o, &i);
 
         flags = l->page[0].o.flags | l->page[1].o.flags;
         if (unlikely(flags & (TLB_WATCHPOINT | TLB_NOTDIRTY))) {
-            mmu_watch_or_dirty(cpu, &l->page[0], type, ra);
-            mmu_watch_or_dirty(cpu, &l->page[1], type, ra);
+            mmu_watch_or_dirty(cpu, &l->page[0], i.access_type, i.ra);
+            mmu_watch_or_dirty(cpu, &l->page[1], i.access_type, i.ra);
         }
 
         /*
-- 
2.43.0

[PATCH v2 31/54] accel/tcg: Always use IntervalTree for code lookups

[Richard Henderson]

Because translation is special, we don't need the speed
of the direct-mapped softmmu tlb.  We cache a lookups in
DisasContextBase within the translator loop anyway.

Drop the addr_code comparator from CPUTLBEntry.
Go directly to the IntervalTree for MMU_INST_FETCH.
Derive exec flags from read flags.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 include/exec/cpu-all.h    |  3 ++
 include/exec/tlb-common.h |  5 ++-
 accel/tcg/cputlb.c        | 76 ++++++++++++++++++++++++---------------
 3 files changed, 52 insertions(+), 32 deletions(-)

diff --git a/include/exec/cpu-all.h b/include/exec/cpu-all.h
index 45e6676938..ad160c328a 100644
--- a/include/exec/cpu-all.h
+++ b/include/exec/cpu-all.h
@@ -339,6 +339,9 @@ static inline int cpu_mmu_index(CPUState *cs, bool ifetch)
     (TLB_INVALID_MASK | TLB_NOTDIRTY | TLB_MMIO \
     | TLB_FORCE_SLOW | TLB_DISCARD_WRITE)
 
+/* Filter read flags to exec flags. */
+#define TLB_EXEC_FLAGS_MASK  (TLB_MMIO)
+
 /*
  * Flags stored in CPUTLBEntryFull.slow_flags[x].
  * TLB_FORCE_SLOW must be set in CPUTLBEntry.addr_idx[x].
diff --git a/include/exec/tlb-common.h b/include/exec/tlb-common.h
index 300f9fae67..feaa471299 100644
--- a/include/exec/tlb-common.h
+++ b/include/exec/tlb-common.h
@@ -26,7 +26,6 @@ typedef union CPUTLBEntry {
     struct {
         uint64_t addr_read;
         uint64_t addr_write;
-        uint64_t addr_code;
         /*
          * Addend to virtual address to get host address.  IO accesses
          * use the corresponding iotlb value.
@@ -35,7 +34,7 @@ typedef union CPUTLBEntry {
     };
     /*
      * Padding to get a power of two size, as well as index
-     * access to addr_{read,write,code}.
+     * access to addr_{read,write}.
      */
     uint64_t addr_idx[(1 << CPU_TLB_ENTRY_BITS) / sizeof(uint64_t)];
 } CPUTLBEntry;
@@ -92,7 +91,7 @@ struct CPUTLBEntryFull {
      * Additional tlb flags for use by the slow path. If non-zero,
      * the corresponding CPUTLBEntry comparator must have TLB_FORCE_SLOW.
      */
-    uint8_t slow_flags[MMU_ACCESS_COUNT];
+    uint8_t slow_flags[2];
 
     /*
      * Allow target-specific additions to this structure.
diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 981098a6f2..be2ea1bc70 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -114,8 +114,9 @@ static inline uint64_t tlb_read_idx(const CPUTLBEntry *entry,
                       MMU_DATA_LOAD * sizeof(uint64_t));
     QEMU_BUILD_BUG_ON(offsetof(CPUTLBEntry, addr_write) !=
                       MMU_DATA_STORE * sizeof(uint64_t));
-    QEMU_BUILD_BUG_ON(offsetof(CPUTLBEntry, addr_code) !=
-                      MMU_INST_FETCH * sizeof(uint64_t));
+
+    tcg_debug_assert(access_type == MMU_DATA_LOAD ||
+                     access_type == MMU_DATA_STORE);
 
 #if TARGET_LONG_BITS == 32
     /* Use qatomic_read, in case of addr_write; only care about low bits. */
@@ -480,8 +481,7 @@ static bool tlb_hit_page_mask_anyprot(CPUTLBEntry *tlb_entry,
     mask &= TARGET_PAGE_MASK | TLB_INVALID_MASK;
 
     return (page == (tlb_entry->addr_read & mask) ||
-            page == (tlb_addr_write(tlb_entry) & mask) ||
-            page == (tlb_entry->addr_code & mask));
+            page == (tlb_addr_write(tlb_entry) & mask));
 }
 
 /* Called with tlb_c.lock held */
@@ -1184,9 +1184,6 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
     /* Now calculate the new entry */
     node->copy.addend = addend - addr_page;
 
-    tlb_set_compare(full, &node->copy, addr_page, read_flags,
-                    MMU_INST_FETCH, prot & PAGE_EXEC);
-
     if (wp_flags & BP_MEM_READ) {
         read_flags |= TLB_WATCHPOINT;
     }
@@ -1308,22 +1305,30 @@ static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
     /* Primary lookup in the fast tlb. */
     entry = tlbfast_entry(fast, addr);
     full = &desc->fulltlb[tlbfast_index(fast, addr)];
-    cmp = tlb_read_idx(entry, access_type);
-    if (tlb_hit(cmp, addr)) {
-        goto found;
+    if (access_type != MMU_INST_FETCH) {
+        cmp = tlb_read_idx(entry, access_type);
+        if (tlb_hit(cmp, addr)) {
+            goto found_data;
+        }
     }
 
     /* Secondary lookup in the IntervalTree. */
     node = tlbtree_lookup_addr(desc, addr);
     if (node) {
-        cmp = tlb_read_idx(&node->copy, access_type);
-        if (tlb_hit(cmp, addr)) {
-            /* Install the cached entry. */
-            qemu_spin_lock(&cpu->neg.tlb.c.lock);
-            copy_tlb_helper_locked(entry, &node->copy);
-            qemu_spin_unlock(&cpu->neg.tlb.c.lock);
-            *full = node->full;
-            goto found;
+        if (access_type == MMU_INST_FETCH) {
+            if (node->full.prot & PAGE_EXEC) {
+                goto found_code;
+            }
+        } else {
+            cmp = tlb_read_idx(&node->copy, access_type);
+            if (tlb_hit(cmp, addr)) {
+                /* Install the cached entry. */
+                qemu_spin_lock(&cpu->neg.tlb.c.lock);
+                copy_tlb_helper_locked(entry, &node->copy);
+                qemu_spin_unlock(&cpu->neg.tlb.c.lock);
+                *full = node->full;
+                goto found_data;
+            }
         }
     }
 
@@ -1333,9 +1338,14 @@ static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
         tcg_debug_assert(probe);
         return false;
     }
-
     o->did_tlb_fill = true;
 
+    if (access_type == MMU_INST_FETCH) {
+        node = tlbtree_lookup_addr(desc, addr);
+        tcg_debug_assert(node);
+        goto found_code;
+    }
+
     entry = tlbfast_entry(fast, addr);
     full = &desc->fulltlb[tlbfast_index(fast, addr)];
     cmp = tlb_read_idx(entry, access_type);
@@ -1345,14 +1355,29 @@ static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
      * called tlb_fill_align, so we know that this entry *is* valid.
      */
     flags &= ~TLB_INVALID_MASK;
+    goto found_data;
+
+ found_data:
+    flags &= cmp;
+    flags |= full->slow_flags[access_type];
+    o->flags = flags;
+    o->full = *full;
+    o->haddr = (void *)((uintptr_t)addr + entry->addend);
     goto done;
 
- found:
-    /* Alignment has not been checked by tlb_fill_align. */
-    {
+ found_code:
+    o->flags = node->copy.addr_read & TLB_EXEC_FLAGS_MASK;
+    o->full = node->full;
+    o->haddr = (void *)((uintptr_t)addr + node->copy.addend);
+    goto done;
+
+ done:
+    if (!o->did_tlb_fill) {
         int a_bits = memop_alignment_bits(memop);
 
         /*
+         * Alignment has not been checked by tlb_fill_align.
+         *
          * The TLB_CHECK_ALIGNED check differs from the normal alignment
          * check, in that this is based on the atomicity of the operation.
          * The intended use case is the ARM memory type field of each PTE,
@@ -1366,13 +1391,6 @@ static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
             cpu_unaligned_access(cpu, addr, access_type, i->mmu_idx, i->ra);
         }
     }
-
- done:
-    flags &= cmp;
-    flags |= full->slow_flags[access_type];
-    o->flags = flags;
-    o->full = *full;
-    o->haddr = (void *)((uintptr_t)addr + entry->addend);
     return true;
 }
 
-- 
2.43.0

[PATCH v2 32/54] accel/tcg: Link CPUTLBEntry to CPUTLBEntryTree

[Richard Henderson]

Link from the fast tlb entry to the interval tree node.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 include/exec/tlb-common.h |  2 ++
 accel/tcg/cputlb.c        | 26 +++++++++++++-------------
 2 files changed, 15 insertions(+), 13 deletions(-)

diff --git a/include/exec/tlb-common.h b/include/exec/tlb-common.h
index feaa471299..3b57d61112 100644
--- a/include/exec/tlb-common.h
+++ b/include/exec/tlb-common.h
@@ -31,6 +31,8 @@ typedef union CPUTLBEntry {
          * use the corresponding iotlb value.
          */
         uintptr_t addend;
+        /* The defining IntervalTree entry. */
+        struct CPUTLBEntryTree *tree;
     };
     /*
      * Padding to get a power of two size, as well as index
diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index be2ea1bc70..3282436752 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -490,7 +490,10 @@ static bool tlb_flush_entry_mask_locked(CPUTLBEntry *tlb_entry,
                                         vaddr mask)
 {
     if (tlb_hit_page_mask_anyprot(tlb_entry, page, mask)) {
-        memset(tlb_entry, -1, sizeof(*tlb_entry));
+        tlb_entry->addr_read = -1;
+        tlb_entry->addr_write = -1;
+        tlb_entry->addend = 0;
+        tlb_entry->tree = NULL;
         return true;
     }
     return false;
@@ -1183,6 +1186,7 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
 
     /* Now calculate the new entry */
     node->copy.addend = addend - addr_page;
+    node->copy.tree = node;
 
     if (wp_flags & BP_MEM_READ) {
         read_flags |= TLB_WATCHPOINT;
@@ -1291,7 +1295,6 @@ static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
     CPUTLBDescFast *fast = &cpu->neg.tlb.f[i->mmu_idx];
     vaddr addr = i->addr;
     MMUAccessType access_type = i->access_type;
-    CPUTLBEntryFull *full;
     CPUTLBEntryTree *node;
     CPUTLBEntry *entry;
     uint64_t cmp;
@@ -1304,9 +1307,9 @@ static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
 
     /* Primary lookup in the fast tlb. */
     entry = tlbfast_entry(fast, addr);
-    full = &desc->fulltlb[tlbfast_index(fast, addr)];
     if (access_type != MMU_INST_FETCH) {
         cmp = tlb_read_idx(entry, access_type);
+        node = entry->tree;
         if (tlb_hit(cmp, addr)) {
             goto found_data;
         }
@@ -1326,7 +1329,6 @@ static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
                 qemu_spin_lock(&cpu->neg.tlb.c.lock);
                 copy_tlb_helper_locked(entry, &node->copy);
                 qemu_spin_unlock(&cpu->neg.tlb.c.lock);
-                *full = node->full;
                 goto found_data;
             }
         }
@@ -1347,8 +1349,8 @@ static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
     }
 
     entry = tlbfast_entry(fast, addr);
-    full = &desc->fulltlb[tlbfast_index(fast, addr)];
     cmp = tlb_read_idx(entry, access_type);
+    node = entry->tree;
     /*
      * With PAGE_WRITE_INV, we set TLB_INVALID_MASK immediately,
      * to force the next access through tlb_fill_align.  We've just
@@ -1359,19 +1361,18 @@ static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
 
  found_data:
     flags &= cmp;
-    flags |= full->slow_flags[access_type];
+    flags |= node->full.slow_flags[access_type];
     o->flags = flags;
-    o->full = *full;
-    o->haddr = (void *)((uintptr_t)addr + entry->addend);
-    goto done;
+    goto found_common;
 
  found_code:
     o->flags = node->copy.addr_read & TLB_EXEC_FLAGS_MASK;
+    goto found_common;
+
+ found_common:
     o->full = node->full;
     o->haddr = (void *)((uintptr_t)addr + node->copy.addend);
-    goto done;
 
- done:
     if (!o->did_tlb_fill) {
         int a_bits = memop_alignment_bits(memop);
 
@@ -1669,7 +1670,6 @@ bool tlb_plugin_lookup(CPUState *cpu, vaddr addr, int mmu_idx,
                        bool is_store, struct qemu_plugin_hwaddr *data)
 {
     CPUTLBEntry *tlbe = tlb_entry(cpu, mmu_idx, addr);
-    uintptr_t index = tlb_index(cpu, mmu_idx, addr);
     MMUAccessType access_type = is_store ? MMU_DATA_STORE : MMU_DATA_LOAD;
     uint64_t tlb_addr = tlb_read_idx(tlbe, access_type);
     CPUTLBEntryFull *full;
@@ -1678,7 +1678,7 @@ bool tlb_plugin_lookup(CPUState *cpu, vaddr addr, int mmu_idx,
         return false;
     }
 
-    full = &cpu->neg.tlb.d[mmu_idx].fulltlb[index];
+    full = &tlbe->tree->full;
     data->phys_addr = full->phys_addr | (addr & ~TARGET_PAGE_MASK);
 
     /* We must have an iotlb entry for MMIO */
-- 
2.43.0

[PATCH v2 33/54] accel/tcg: Remove CPUTLBDesc.fulltlb

[Richard Henderson]

This array is now write-only, and may be removed.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 include/hw/core/cpu.h |  1 -
 accel/tcg/cputlb.c    | 34 +++++++---------------------------
 2 files changed, 7 insertions(+), 28 deletions(-)

diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
index 4364ddb1db..5c069f2a00 100644
--- a/include/hw/core/cpu.h
+++ b/include/hw/core/cpu.h
@@ -219,7 +219,6 @@ typedef struct CPUTLBDesc {
     /* maximum number of entries observed in the window */
     size_t window_max_entries;
     size_t n_used_entries;
-    CPUTLBEntryFull *fulltlb;
     /* All active tlb entries for this address space. */
     IntervalTreeRoot iroot;
 } CPUTLBDesc;
diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 3282436752..7f63dc3fd8 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -149,13 +149,6 @@ static inline CPUTLBEntry *tlbfast_entry(CPUTLBDescFast *fast, vaddr addr)
     return fast->table + tlbfast_index(fast, addr);
 }
 
-/* Find the TLB index corresponding to the mmu_idx + address pair.  */
-static inline uintptr_t tlb_index(CPUState *cpu, uintptr_t mmu_idx,
-                                  vaddr addr)
-{
-    return tlbfast_index(&cpu->neg.tlb.f[mmu_idx], addr);
-}
-
 /* Find the TLB entry corresponding to the mmu_idx + address pair.  */
 static inline CPUTLBEntry *tlb_entry(CPUState *cpu, uintptr_t mmu_idx,
                                      vaddr addr)
@@ -270,22 +263,20 @@ static void tlb_mmu_resize_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast,
     }
 
     g_free(fast->table);
-    g_free(desc->fulltlb);
 
     tlb_window_reset(desc, now, 0);
     /* desc->n_used_entries is cleared by the caller */
     fast->mask = (new_size - 1) << CPU_TLB_ENTRY_BITS;
     fast->table = g_try_new(CPUTLBEntry, new_size);
-    desc->fulltlb = g_try_new(CPUTLBEntryFull, new_size);
 
     /*
-     * If the allocations fail, try smaller sizes. We just freed some
+     * If the allocation fails, try smaller sizes. We just freed some
      * memory, so going back to half of new_size has a good chance of working.
      * Increased memory pressure elsewhere in the system might cause the
      * allocations to fail though, so we progressively reduce the allocation
      * size, aborting if we cannot even allocate the smallest TLB we support.
      */
-    while (fast->table == NULL || desc->fulltlb == NULL) {
+    while (fast->table == NULL) {
         if (new_size == (1 << CPU_TLB_DYN_MIN_BITS)) {
             error_report("%s: %s", __func__, strerror(errno));
             abort();
@@ -294,9 +285,7 @@ static void tlb_mmu_resize_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast,
         fast->mask = (new_size - 1) << CPU_TLB_ENTRY_BITS;
 
         g_free(fast->table);
-        g_free(desc->fulltlb);
         fast->table = g_try_new(CPUTLBEntry, new_size);
-        desc->fulltlb = g_try_new(CPUTLBEntryFull, new_size);
     }
 }
 
@@ -350,7 +339,6 @@ static void tlb_mmu_init(CPUTLBDesc *desc, CPUTLBDescFast *fast, int64_t now)
     desc->n_used_entries = 0;
     fast->mask = (n_entries - 1) << CPU_TLB_ENTRY_BITS;
     fast->table = g_new(CPUTLBEntry, n_entries);
-    desc->fulltlb = g_new(CPUTLBEntryFull, n_entries);
     memset(&desc->iroot, 0, sizeof(desc->iroot));
     tlb_mmu_flush_locked(desc, fast);
 }
@@ -372,15 +360,9 @@ void tlb_init(CPUState *cpu)
 
 void tlb_destroy(CPUState *cpu)
 {
-    int i;
-
     qemu_spin_destroy(&cpu->neg.tlb.c.lock);
-    for (i = 0; i < NB_MMU_MODES; i++) {
-        CPUTLBDesc *desc = &cpu->neg.tlb.d[i];
-        CPUTLBDescFast *fast = &cpu->neg.tlb.f[i];
-
-        g_free(fast->table);
-        g_free(desc->fulltlb);
+    for (int i = 0; i < NB_MMU_MODES; i++) {
+        g_free(cpu->neg.tlb.f[i].table);
         interval_tree_free_nodes(&cpu->neg.tlb.d[i].iroot,
                                  offsetof(CPUTLBEntryTree, itree));
     }
@@ -1061,7 +1043,7 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
     CPUTLB *tlb = &cpu->neg.tlb;
     CPUTLBDesc *desc = &tlb->d[mmu_idx];
     MemoryRegionSection *section;
-    unsigned int index, read_flags, write_flags;
+    unsigned int read_flags, write_flags;
     uintptr_t addend;
     CPUTLBEntry *te;
     CPUTLBEntryTree *node;
@@ -1140,7 +1122,6 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
     wp_flags = cpu_watchpoint_address_matches(cpu, addr_page,
                                               TARGET_PAGE_SIZE);
 
-    index = tlb_index(cpu, mmu_idx, addr_page);
     te = tlb_entry(cpu, mmu_idx, addr_page);
 
     /*
@@ -1179,8 +1160,8 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
      * subtract here is that of the page base, and not the same as the
      * vaddr we add back in io_prepare()/get_page_addr_code().
      */
-    desc->fulltlb[index] = *full;
-    full = &desc->fulltlb[index];
+    node->full = *full;
+    full = &node->full;
     full->xlat_section = iotlb - addr_page;
     full->phys_addr = paddr_page;
 
@@ -1203,7 +1184,6 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
     tlb_set_compare(full, &node->copy, addr_page, write_flags,
                     MMU_DATA_STORE, prot & PAGE_WRITE);
 
-    node->full = *full;
     copy_tlb_helper_locked(te, &node->copy);
     desc->n_used_entries++;
     qemu_spin_unlock(&tlb->c.lock);
-- 
2.43.0

[PATCH v2 34/54] target/alpha: Convert to TCGCPUOps.tlb_fill_align

[Richard Henderson]

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 target/alpha/cpu.h    |  6 +++---
 target/alpha/cpu.c    |  2 +-
 target/alpha/helper.c | 23 +++++++++++++++++------
 3 files changed, 21 insertions(+), 10 deletions(-)

diff --git a/target/alpha/cpu.h b/target/alpha/cpu.h
index 3556d3227f..70331c0b83 100644
--- a/target/alpha/cpu.h
+++ b/target/alpha/cpu.h
@@ -449,9 +449,9 @@ void alpha_cpu_record_sigsegv(CPUState *cs, vaddr address,
 void alpha_cpu_record_sigbus(CPUState *cs, vaddr address,
                              MMUAccessType access_type, uintptr_t retaddr);
 #else
-bool alpha_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                        MMUAccessType access_type, int mmu_idx,
-                        bool probe, uintptr_t retaddr);
+bool alpha_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out, vaddr addr,
+                              MMUAccessType access_type, int mmu_idx,
+                              MemOp memop, int size, bool probe, uintptr_t ra);
 G_NORETURN void alpha_cpu_do_unaligned_access(CPUState *cpu, vaddr addr,
                                               MMUAccessType access_type, int mmu_idx,
                                               uintptr_t retaddr);
diff --git a/target/alpha/cpu.c b/target/alpha/cpu.c
index 5d75c941f7..7bcc48420d 100644
--- a/target/alpha/cpu.c
+++ b/target/alpha/cpu.c
@@ -228,7 +228,7 @@ static const TCGCPUOps alpha_tcg_ops = {
     .record_sigsegv = alpha_cpu_record_sigsegv,
     .record_sigbus = alpha_cpu_record_sigbus,
 #else
-    .tlb_fill = alpha_cpu_tlb_fill,
+    .tlb_fill_align = alpha_cpu_tlb_fill_align,
     .cpu_exec_interrupt = alpha_cpu_exec_interrupt,
     .cpu_exec_halt = alpha_cpu_has_work,
     .do_interrupt = alpha_cpu_do_interrupt,
diff --git a/target/alpha/helper.c b/target/alpha/helper.c
index 2f1000c99f..26eadfe3ca 100644
--- a/target/alpha/helper.c
+++ b/target/alpha/helper.c
@@ -294,14 +294,21 @@ hwaddr alpha_cpu_get_phys_page_debug(CPUState *cs, vaddr addr)
     return (fail >= 0 ? -1 : phys);
 }
 
-bool alpha_cpu_tlb_fill(CPUState *cs, vaddr addr, int size,
-                        MMUAccessType access_type, int mmu_idx,
-                        bool probe, uintptr_t retaddr)
+bool alpha_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out, vaddr addr,
+                              MMUAccessType access_type, int mmu_idx,
+                              MemOp memop, int size, bool probe, uintptr_t ra)
 {
     CPUAlphaState *env = cpu_env(cs);
     target_ulong phys;
     int prot, fail;
 
+    if (addr & ((1 << memop_alignment_bits(memop)) - 1)) {
+        if (probe) {
+            return false;
+        }
+        alpha_cpu_do_unaligned_access(cs, addr, access_type, mmu_idx, ra);
+    }
+
     fail = get_physical_address(env, addr, 1 << access_type,
                                 mmu_idx, &phys, &prot);
     if (unlikely(fail >= 0)) {
@@ -314,11 +321,15 @@ bool alpha_cpu_tlb_fill(CPUState *cs, vaddr addr, int size,
         env->trap_arg2 = (access_type == MMU_DATA_LOAD ? 0ull :
                           access_type == MMU_DATA_STORE ? 1ull :
                           /* access_type == MMU_INST_FETCH */ -1ull);
-        cpu_loop_exit_restore(cs, retaddr);
+        cpu_loop_exit_restore(cs, ra);
     }
 
-    tlb_set_page(cs, addr & TARGET_PAGE_MASK, phys & TARGET_PAGE_MASK,
-                 prot, mmu_idx, TARGET_PAGE_SIZE);
+    memset(out, 0, sizeof(*out));
+    out->phys_addr = phys;
+    out->prot = prot;
+    out->attrs = MEMTXATTRS_UNSPECIFIED;
+    out->lg_page_size = TARGET_PAGE_BITS;
+
     return true;
 }
 
-- 
2.43.0

[PATCH v2 35/54] target/avr: Convert to TCGCPUOps.tlb_fill_align

[Richard Henderson]

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 target/avr/cpu.h    |  7 ++++---
 target/avr/cpu.c    |  2 +-
 target/avr/helper.c | 19 ++++++++++++-------
 3 files changed, 17 insertions(+), 11 deletions(-)

diff --git a/target/avr/cpu.h b/target/avr/cpu.h
index 4725535102..cdd3bcd418 100644
--- a/target/avr/cpu.h
+++ b/target/avr/cpu.h
@@ -23,6 +23,7 @@
 
 #include "cpu-qom.h"
 #include "exec/cpu-defs.h"
+#include "exec/memop.h"
 
 #ifdef CONFIG_USER_ONLY
 #error "AVR 8-bit does not support user mode"
@@ -238,9 +239,9 @@ static inline void cpu_set_sreg(CPUAVRState *env, uint8_t sreg)
     env->sregI = (sreg >> 7) & 0x01;
 }
 
-bool avr_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                      MMUAccessType access_type, int mmu_idx,
-                      bool probe, uintptr_t retaddr);
+bool avr_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out, vaddr addr,
+                            MMUAccessType access_type, int mmu_idx,
+                            MemOp memop, int size, bool probe, uintptr_t ra);
 
 #include "exec/cpu-all.h"
 
diff --git a/target/avr/cpu.c b/target/avr/cpu.c
index 3132842d56..a7fe869396 100644
--- a/target/avr/cpu.c
+++ b/target/avr/cpu.c
@@ -211,7 +211,7 @@ static const TCGCPUOps avr_tcg_ops = {
     .restore_state_to_opc = avr_restore_state_to_opc,
     .cpu_exec_interrupt = avr_cpu_exec_interrupt,
     .cpu_exec_halt = avr_cpu_has_work,
-    .tlb_fill = avr_cpu_tlb_fill,
+    .tlb_fill_align = avr_cpu_tlb_fill_align,
     .do_interrupt = avr_cpu_do_interrupt,
 };
 
diff --git a/target/avr/helper.c b/target/avr/helper.c
index 345708a1b3..a18f11aa9f 100644
--- a/target/avr/helper.c
+++ b/target/avr/helper.c
@@ -104,11 +104,11 @@ hwaddr avr_cpu_get_phys_page_debug(CPUState *cs, vaddr addr)
     return addr; /* I assume 1:1 address correspondence */
 }
 
-bool avr_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                      MMUAccessType access_type, int mmu_idx,
-                      bool probe, uintptr_t retaddr)
+bool avr_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out, vaddr address,
+                            MMUAccessType access_type, int mmu_idx,
+                            MemOp memop, int size, bool probe, uintptr_t ra)
 {
-    int prot, page_size = TARGET_PAGE_SIZE;
+    int prot, lg_page_size = TARGET_PAGE_BITS;
     uint32_t paddr;
 
     address &= TARGET_PAGE_MASK;
@@ -141,15 +141,20 @@ bool avr_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
              * to force tlb_fill to be called for the next access.
              */
             if (probe) {
-                page_size = 1;
+                lg_page_size = 0;
             } else {
                 cpu_env(cs)->fullacc = 1;
-                cpu_loop_exit_restore(cs, retaddr);
+                cpu_loop_exit_restore(cs, ra);
             }
         }
     }
 
-    tlb_set_page(cs, address, paddr, prot, mmu_idx, page_size);
+    memset(out, 0, sizeof(*out));
+    out->phys_addr = paddr;
+    out->prot = prot;
+    out->attrs = MEMTXATTRS_UNSPECIFIED;
+    out->lg_page_size = lg_page_size;
+
     return true;
 }
 
-- 
2.43.0

[PATCH v2 36/54] target/i386: Convert to TCGCPUOps.tlb_fill_align

[Richard Henderson]

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 target/i386/tcg/helper-tcg.h         |  6 +++---
 target/i386/tcg/sysemu/excp_helper.c | 28 ++++++++++++++++------------
 target/i386/tcg/tcg-cpu.c            |  2 +-
 3 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/target/i386/tcg/helper-tcg.h b/target/i386/tcg/helper-tcg.h
index 696d6ef016..b2164f41e6 100644
--- a/target/i386/tcg/helper-tcg.h
+++ b/target/i386/tcg/helper-tcg.h
@@ -79,9 +79,9 @@ void x86_cpu_record_sigsegv(CPUState *cs, vaddr addr,
 void x86_cpu_record_sigbus(CPUState *cs, vaddr addr,
                            MMUAccessType access_type, uintptr_t ra);
 #else
-bool x86_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                      MMUAccessType access_type, int mmu_idx,
-                      bool probe, uintptr_t retaddr);
+bool x86_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out, vaddr addr,
+                            MMUAccessType access_type, int mmu_idx,
+                            MemOp memop, int size, bool probe, uintptr_t ra);
 G_NORETURN void x86_cpu_do_unaligned_access(CPUState *cs, vaddr vaddr,
                                             MMUAccessType access_type,
                                             int mmu_idx, uintptr_t retaddr);
diff --git a/target/i386/tcg/sysemu/excp_helper.c b/target/i386/tcg/sysemu/excp_helper.c
index 168ff8e5f3..d23d28fef5 100644
--- a/target/i386/tcg/sysemu/excp_helper.c
+++ b/target/i386/tcg/sysemu/excp_helper.c
@@ -601,25 +601,29 @@ static bool get_physical_address(CPUX86State *env, vaddr addr,
     return true;
 }
 
-bool x86_cpu_tlb_fill(CPUState *cs, vaddr addr, int size,
-                      MMUAccessType access_type, int mmu_idx,
-                      bool probe, uintptr_t retaddr)
+bool x86_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *full, vaddr addr,
+                            MMUAccessType access_type, int mmu_idx,
+                            MemOp memop, int size, bool probe,
+                            uintptr_t retaddr)
 {
     CPUX86State *env = cpu_env(cs);
     TranslateResult out;
     TranslateFault err;
 
+    if (addr & ((1 << memop_alignment_bits(memop)) - 1)) {
+        if (probe) {
+            return false;
+        }
+        x86_cpu_do_unaligned_access(cs, addr, access_type, mmu_idx, retaddr);
+    }
+
     if (get_physical_address(env, addr, access_type, mmu_idx, &out, &err,
                              retaddr)) {
-        /*
-         * Even if 4MB pages, we map only one 4KB page in the cache to
-         * avoid filling it too fast.
-         */
-        assert(out.prot & (1 << access_type));
-        tlb_set_page_with_attrs(cs, addr & TARGET_PAGE_MASK,
-                                out.paddr & TARGET_PAGE_MASK,
-                                cpu_get_mem_attrs(env),
-                                out.prot, mmu_idx, out.page_size);
+        memset(full, 0, sizeof(*full));
+        full->phys_addr = out.paddr;
+        full->prot = out.prot;
+        full->lg_page_size = ctz32(out.page_size);
+        full->attrs = cpu_get_mem_attrs(env);
         return true;
     }
 
diff --git a/target/i386/tcg/tcg-cpu.c b/target/i386/tcg/tcg-cpu.c
index cca19cd40e..6fce6227c7 100644
--- a/target/i386/tcg/tcg-cpu.c
+++ b/target/i386/tcg/tcg-cpu.c
@@ -117,7 +117,7 @@ static const TCGCPUOps x86_tcg_ops = {
     .record_sigsegv = x86_cpu_record_sigsegv,
     .record_sigbus = x86_cpu_record_sigbus,
 #else
-    .tlb_fill = x86_cpu_tlb_fill,
+    .tlb_fill_align = x86_cpu_tlb_fill_align,
     .do_interrupt = x86_cpu_do_interrupt,
     .cpu_exec_halt = x86_cpu_exec_halt,
     .cpu_exec_interrupt = x86_cpu_exec_interrupt,
-- 
2.43.0

[PATCH v2 37/54] target/loongarch: Convert to TCGCPUOps.tlb_fill_align

[Richard Henderson]

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 target/loongarch/internals.h      |  7 ++++---
 target/loongarch/cpu.c            |  2 +-
 target/loongarch/tcg/tlb_helper.c | 17 +++++++++++------
 3 files changed, 16 insertions(+), 10 deletions(-)

diff --git a/target/loongarch/internals.h b/target/loongarch/internals.h
index 1a02427627..a9f73f27b2 100644
--- a/target/loongarch/internals.h
+++ b/target/loongarch/internals.h
@@ -60,9 +60,10 @@ int get_physical_address(CPULoongArchState *env, hwaddr *physical,
 hwaddr loongarch_cpu_get_phys_page_debug(CPUState *cpu, vaddr addr);
 
 #ifdef CONFIG_TCG
-bool loongarch_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                            MMUAccessType access_type, int mmu_idx,
-                            bool probe, uintptr_t retaddr);
+bool loongarch_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
+                                  vaddr addr, MMUAccessType access_type,
+                                  int mmu_idx, MemOp memop, int size,
+                                  bool probe, uintptr_t ra);
 #endif
 #endif /* !CONFIG_USER_ONLY */
 
diff --git a/target/loongarch/cpu.c b/target/loongarch/cpu.c
index 57cc4f314b..47d69f1788 100644
--- a/target/loongarch/cpu.c
+++ b/target/loongarch/cpu.c
@@ -798,7 +798,7 @@ static const TCGCPUOps loongarch_tcg_ops = {
     .restore_state_to_opc = loongarch_restore_state_to_opc,
 
 #ifndef CONFIG_USER_ONLY
-    .tlb_fill = loongarch_cpu_tlb_fill,
+    .tlb_fill_align = loongarch_cpu_tlb_fill_align,
     .cpu_exec_interrupt = loongarch_cpu_exec_interrupt,
     .cpu_exec_halt = loongarch_cpu_has_work,
     .do_interrupt = loongarch_cpu_do_interrupt,
diff --git a/target/loongarch/tcg/tlb_helper.c b/target/loongarch/tcg/tlb_helper.c
index 97f38fc391..94d5df08a4 100644
--- a/target/loongarch/tcg/tlb_helper.c
+++ b/target/loongarch/tcg/tlb_helper.c
@@ -474,9 +474,10 @@ void helper_invtlb_page_asid_or_g(CPULoongArchState *env,
     tlb_flush(env_cpu(env));
 }
 
-bool loongarch_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                            MMUAccessType access_type, int mmu_idx,
-                            bool probe, uintptr_t retaddr)
+bool loongarch_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
+                                  vaddr address, MMUAccessType access_type,
+                                  int mmu_idx, MemOp memop, int size,
+                                  bool probe, uintptr_t retaddr)
 {
     CPULoongArchState *env = cpu_env(cs);
     hwaddr physical;
@@ -488,12 +489,16 @@ bool loongarch_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
                                access_type, mmu_idx);
 
     if (ret == TLBRET_MATCH) {
-        tlb_set_page(cs, address & TARGET_PAGE_MASK,
-                     physical & TARGET_PAGE_MASK, prot,
-                     mmu_idx, TARGET_PAGE_SIZE);
         qemu_log_mask(CPU_LOG_MMU,
                       "%s address=%" VADDR_PRIx " physical " HWADDR_FMT_plx
                       " prot %d\n", __func__, address, physical, prot);
+
+        memset(out, 0, sizeof(*out));
+        out->phys_addr = physical;
+        out->prot = prot;
+        out->attrs = MEMTXATTRS_UNSPECIFIED;
+        out->lg_page_size = TARGET_PAGE_BITS;
+
         return true;
     } else {
         qemu_log_mask(CPU_LOG_MMU,
-- 
2.43.0

[PATCH v2 38/54] target/m68k: Convert to TCGCPUOps.tlb_fill_align

[Richard Henderson]

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 target/m68k/cpu.h    |  7 ++++---
 target/m68k/cpu.c    |  2 +-
 target/m68k/helper.c | 22 +++++++++++++---------
 3 files changed, 18 insertions(+), 13 deletions(-)

diff --git a/target/m68k/cpu.h b/target/m68k/cpu.h
index b5bbeedb7a..4401426a0b 100644
--- a/target/m68k/cpu.h
+++ b/target/m68k/cpu.h
@@ -22,6 +22,7 @@
 #define M68K_CPU_H
 
 #include "exec/cpu-defs.h"
+#include "exec/memop.h"
 #include "qemu/cpu-float.h"
 #include "cpu-qom.h"
 
@@ -582,10 +583,10 @@ enum {
 #define MMU_KERNEL_IDX 0
 #define MMU_USER_IDX 1
 
-bool m68k_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                       MMUAccessType access_type, int mmu_idx,
-                       bool probe, uintptr_t retaddr);
 #ifndef CONFIG_USER_ONLY
+bool m68k_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out, vaddr addr,
+                             MMUAccessType access_type, int mmu_idx,
+                             MemOp memop, int size, bool probe, uintptr_t ra);
 void m68k_cpu_transaction_failed(CPUState *cs, hwaddr physaddr, vaddr addr,
                                  unsigned size, MMUAccessType access_type,
                                  int mmu_idx, MemTxAttrs attrs,
diff --git a/target/m68k/cpu.c b/target/m68k/cpu.c
index 5fe335558a..5316cf8922 100644
--- a/target/m68k/cpu.c
+++ b/target/m68k/cpu.c
@@ -550,7 +550,7 @@ static const TCGCPUOps m68k_tcg_ops = {
     .restore_state_to_opc = m68k_restore_state_to_opc,
 
 #ifndef CONFIG_USER_ONLY
-    .tlb_fill = m68k_cpu_tlb_fill,
+    .tlb_fill_align = m68k_cpu_tlb_fill_align,
     .cpu_exec_interrupt = m68k_cpu_exec_interrupt,
     .cpu_exec_halt = m68k_cpu_has_work,
     .do_interrupt = m68k_cpu_do_interrupt,
diff --git a/target/m68k/helper.c b/target/m68k/helper.c
index 9bfc6ae97c..1decb6f39c 100644
--- a/target/m68k/helper.c
+++ b/target/m68k/helper.c
@@ -950,9 +950,10 @@ void m68k_set_irq_level(M68kCPU *cpu, int level, uint8_t vector)
     }
 }
 
-bool m68k_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                       MMUAccessType qemu_access_type, int mmu_idx,
-                       bool probe, uintptr_t retaddr)
+bool m68k_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
+                             vaddr address, MMUAccessType qemu_access_type,
+                             int mmu_idx, MemOp memop, int size,
+                             bool probe, uintptr_t retaddr)
 {
     CPUM68KState *env = cpu_env(cs);
     hwaddr physical;
@@ -961,12 +962,14 @@ bool m68k_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
     int ret;
     target_ulong page_size;
 
+    memset(out, 0, sizeof(*out));
+    out->attrs = MEMTXATTRS_UNSPECIFIED;
+
     if ((env->mmu.tcr & M68K_TCR_ENABLED) == 0) {
         /* MMU disabled */
-        tlb_set_page(cs, address & TARGET_PAGE_MASK,
-                     address & TARGET_PAGE_MASK,
-                     PAGE_READ | PAGE_WRITE | PAGE_EXEC,
-                     mmu_idx, TARGET_PAGE_SIZE);
+        out->phys_addr = address;
+        out->prot = PAGE_READ | PAGE_WRITE | PAGE_EXEC;
+        out->lg_page_size = TARGET_PAGE_BITS;
         return true;
     }
 
@@ -985,8 +988,9 @@ bool m68k_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
     ret = get_physical_address(env, &physical, &prot,
                                address, access_type, &page_size);
     if (likely(ret == 0)) {
-        tlb_set_page(cs, address & TARGET_PAGE_MASK,
-                     physical & TARGET_PAGE_MASK, prot, mmu_idx, page_size);
+        out->phys_addr = physical;
+        out->prot = prot;
+        out->lg_page_size = ctz32(page_size);
         return true;
     }
 
-- 
2.43.0

[PATCH v2 39/54] target/m68k: Do not call tlb_set_page in helper_ptest

[Richard Henderson]

The entire operation of ptest is performed within
get_physical_address as part of ACCESS_PTEST.
There is no need to install the page into softmmu.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 target/m68k/helper.c | 10 +---------
 1 file changed, 1 insertion(+), 9 deletions(-)

diff --git a/target/m68k/helper.c b/target/m68k/helper.c
index 1decb6f39c..0a54eca9bb 100644
--- a/target/m68k/helper.c
+++ b/target/m68k/helper.c
@@ -1460,7 +1460,6 @@ void HELPER(ptest)(CPUM68KState *env, uint32_t addr, uint32_t is_read)
     hwaddr physical;
     int access_type;
     int prot;
-    int ret;
     target_ulong page_size;
 
     access_type = ACCESS_PTEST;
@@ -1476,14 +1475,7 @@ void HELPER(ptest)(CPUM68KState *env, uint32_t addr, uint32_t is_read)
 
     env->mmu.mmusr = 0;
     env->mmu.ssw = 0;
-    ret = get_physical_address(env, &physical, &prot, addr,
-                               access_type, &page_size);
-    if (ret == 0) {
-        tlb_set_page(env_cpu(env), addr & TARGET_PAGE_MASK,
-                     physical & TARGET_PAGE_MASK,
-                     prot, access_type & ACCESS_SUPER ?
-                     MMU_KERNEL_IDX : MMU_USER_IDX, page_size);
-    }
+    get_physical_address(env, &physical, &prot, addr, access_type, &page_size);
 }
 
 void HELPER(pflush)(CPUM68KState *env, uint32_t addr, uint32_t opmode)
-- 
2.43.0

[PATCH v2 40/54] target/microblaze: Convert to TCGCPUOps.tlb_fill_align

[Richard Henderson]

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 target/microblaze/cpu.h    |  7 +++----
 target/microblaze/cpu.c    |  2 +-
 target/microblaze/helper.c | 33 ++++++++++++++++++++-------------
 3 files changed, 24 insertions(+), 18 deletions(-)

diff --git a/target/microblaze/cpu.h b/target/microblaze/cpu.h
index 3e5a3e5c60..b0eadfd9b1 100644
--- a/target/microblaze/cpu.h
+++ b/target/microblaze/cpu.h
@@ -421,10 +421,9 @@ static inline void cpu_get_tb_cpu_state(CPUMBState *env, vaddr *pc,
 }
 
 #if !defined(CONFIG_USER_ONLY)
-bool mb_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                     MMUAccessType access_type, int mmu_idx,
-                     bool probe, uintptr_t retaddr);
-
+bool mb_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out, vaddr address,
+                            MMUAccessType access_type, int mmu_idx,
+                           MemOp memop, int size, bool probe, uintptr_t ra);
 void mb_cpu_transaction_failed(CPUState *cs, hwaddr physaddr, vaddr addr,
                                unsigned size, MMUAccessType access_type,
                                int mmu_idx, MemTxAttrs attrs,
diff --git a/target/microblaze/cpu.c b/target/microblaze/cpu.c
index 710eb1146c..212cad2143 100644
--- a/target/microblaze/cpu.c
+++ b/target/microblaze/cpu.c
@@ -425,7 +425,7 @@ static const TCGCPUOps mb_tcg_ops = {
     .restore_state_to_opc = mb_restore_state_to_opc,
 
 #ifndef CONFIG_USER_ONLY
-    .tlb_fill = mb_cpu_tlb_fill,
+    .tlb_fill_align = mb_cpu_tlb_fill_align,
     .cpu_exec_interrupt = mb_cpu_exec_interrupt,
     .cpu_exec_halt = mb_cpu_has_work,
     .do_interrupt = mb_cpu_do_interrupt,
diff --git a/target/microblaze/helper.c b/target/microblaze/helper.c
index 5d3259ce31..b6375564b4 100644
--- a/target/microblaze/helper.c
+++ b/target/microblaze/helper.c
@@ -36,37 +36,44 @@ static bool mb_cpu_access_is_secure(MicroBlazeCPU *cpu,
     }
 }
 
-bool mb_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                     MMUAccessType access_type, int mmu_idx,
-                     bool probe, uintptr_t retaddr)
+bool mb_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out, vaddr address,
+                           MMUAccessType access_type, int mmu_idx,
+                           MemOp memop, int size,
+                           bool probe, uintptr_t retaddr)
 {
     MicroBlazeCPU *cpu = MICROBLAZE_CPU(cs);
     CPUMBState *env = &cpu->env;
     MicroBlazeMMULookup lu;
     unsigned int hit;
-    int prot;
-    MemTxAttrs attrs = {};
 
-    attrs.secure = mb_cpu_access_is_secure(cpu, access_type);
+    if (address & ((1 << memop_alignment_bits(memop)) - 1)) {
+        if (probe) {
+            return false;
+        }
+        mb_cpu_do_unaligned_access(cs, address, access_type, mmu_idx, retaddr);
+    }
+
+    memset(out, 0, sizeof(*out));
+    out->attrs.secure = mb_cpu_access_is_secure(cpu, access_type);
+    out->lg_page_size = TARGET_PAGE_BITS;
 
     if (mmu_idx == MMU_NOMMU_IDX) {
         /* MMU disabled or not available.  */
-        address &= TARGET_PAGE_MASK;
-        prot = PAGE_RWX;
-        tlb_set_page_with_attrs(cs, address, address, attrs, prot, mmu_idx,
-                                TARGET_PAGE_SIZE);
+        out->phys_addr = address;
+        out->prot = PAGE_RWX;
         return true;
     }
 
     hit = mmu_translate(cpu, &lu, address, access_type, mmu_idx);
     if (likely(hit)) {
-        uint32_t vaddr = address & TARGET_PAGE_MASK;
+        uint32_t vaddr = address;
         uint32_t paddr = lu.paddr + vaddr - lu.vaddr;
 
         qemu_log_mask(CPU_LOG_MMU, "MMU map mmu=%d v=%x p=%x prot=%x\n",
                       mmu_idx, vaddr, paddr, lu.prot);
-        tlb_set_page_with_attrs(cs, vaddr, paddr, attrs, lu.prot, mmu_idx,
-                                TARGET_PAGE_SIZE);
+
+        out->phys_addr = paddr;
+        out->prot = lu.prot;
         return true;
     }
 
-- 
2.43.0

[PATCH v2 41/54] target/mips: Convert to TCGCPUOps.tlb_fill_align

[Richard Henderson]

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 target/mips/tcg/tcg-internal.h      |  6 +++---
 target/mips/cpu.c                   |  2 +-
 target/mips/tcg/sysemu/tlb_helper.c | 29 ++++++++++++++++++++---------
 3 files changed, 24 insertions(+), 13 deletions(-)

diff --git a/target/mips/tcg/tcg-internal.h b/target/mips/tcg/tcg-internal.h
index aef032c48d..f4b00354af 100644
--- a/target/mips/tcg/tcg-internal.h
+++ b/target/mips/tcg/tcg-internal.h
@@ -61,9 +61,9 @@ void mips_cpu_do_transaction_failed(CPUState *cs, hwaddr physaddr,
                                     MemTxResult response, uintptr_t retaddr);
 void cpu_mips_tlb_flush(CPUMIPSState *env);
 
-bool mips_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                       MMUAccessType access_type, int mmu_idx,
-                       bool probe, uintptr_t retaddr);
+bool mips_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out, vaddr address,
+                             MMUAccessType access_type, int mmu_idx,
+                             MemOp memop, int size, bool probe, uintptr_t ra);
 
 void mips_semihosting(CPUMIPSState *env);
 
diff --git a/target/mips/cpu.c b/target/mips/cpu.c
index d0a43b6d5c..3a453c9285 100644
--- a/target/mips/cpu.c
+++ b/target/mips/cpu.c
@@ -556,7 +556,7 @@ static const TCGCPUOps mips_tcg_ops = {
     .restore_state_to_opc = mips_restore_state_to_opc,
 
 #if !defined(CONFIG_USER_ONLY)
-    .tlb_fill = mips_cpu_tlb_fill,
+    .tlb_fill_align = mips_cpu_tlb_fill_align,
     .cpu_exec_interrupt = mips_cpu_exec_interrupt,
     .cpu_exec_halt = mips_cpu_has_work,
     .do_interrupt = mips_cpu_do_interrupt,
diff --git a/target/mips/tcg/sysemu/tlb_helper.c b/target/mips/tcg/sysemu/tlb_helper.c
index e98bb95951..ac76396525 100644
--- a/target/mips/tcg/sysemu/tlb_helper.c
+++ b/target/mips/tcg/sysemu/tlb_helper.c
@@ -904,15 +904,28 @@ refill:
 }
 #endif
 
-bool mips_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                       MMUAccessType access_type, int mmu_idx,
-                       bool probe, uintptr_t retaddr)
+bool mips_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out, vaddr address,
+                             MMUAccessType access_type, int mmu_idx,
+                             MemOp memop, int size,
+                             bool probe, uintptr_t retaddr)
 {
     CPUMIPSState *env = cpu_env(cs);
     hwaddr physical;
     int prot;
     int ret = TLBRET_BADADDR;
 
+    if (address & ((1 << memop_alignment_bits(memop)) - 1)) {
+        if (probe) {
+            return false;
+        }
+        mips_cpu_do_unaligned_access(cs, address, access_type,
+                                     mmu_idx, retaddr);
+    }
+
+    memset(out, 0, sizeof(*out));
+    out->attrs = MEMTXATTRS_UNSPECIFIED;
+    out->lg_page_size = TARGET_PAGE_BITS;
+
     /* data access */
     /* XXX: put correct access by using cpu_restore_state() correctly */
     ret = get_physical_address(env, &physical, &prot, address,
@@ -930,9 +943,8 @@ bool mips_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
         break;
     }
     if (ret == TLBRET_MATCH) {
-        tlb_set_page(cs, address & TARGET_PAGE_MASK,
-                     physical & TARGET_PAGE_MASK, prot,
-                     mmu_idx, TARGET_PAGE_SIZE);
+        out->phys_addr = physical;
+        out->prot = prot;
         return true;
     }
 #if !defined(TARGET_MIPS64)
@@ -948,9 +960,8 @@ bool mips_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
             ret = get_physical_address(env, &physical, &prot, address,
                                        access_type, mmu_idx);
             if (ret == TLBRET_MATCH) {
-                tlb_set_page(cs, address & TARGET_PAGE_MASK,
-                             physical & TARGET_PAGE_MASK, prot,
-                             mmu_idx, TARGET_PAGE_SIZE);
+                out->phys_addr = physical;
+                out->prot = prot;
                 return true;
             }
         }
-- 
2.43.0

[PATCH v2 42/54] target/openrisc: Convert to TCGCPUOps.tlb_fill_align

[Richard Henderson]

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 target/openrisc/cpu.h |  8 +++++---
 target/openrisc/cpu.c |  2 +-
 target/openrisc/mmu.c | 39 +++++++++++++++++++++------------------
 3 files changed, 27 insertions(+), 22 deletions(-)

diff --git a/target/openrisc/cpu.h b/target/openrisc/cpu.h
index c9fe9ae12d..e177ad8b84 100644
--- a/target/openrisc/cpu.h
+++ b/target/openrisc/cpu.h
@@ -22,6 +22,7 @@
 
 #include "cpu-qom.h"
 #include "exec/cpu-defs.h"
+#include "exec/memop.h"
 #include "fpu/softfloat-types.h"
 
 /**
@@ -306,9 +307,10 @@ int print_insn_or1k(bfd_vma addr, disassemble_info *info);
 #ifndef CONFIG_USER_ONLY
 hwaddr openrisc_cpu_get_phys_page_debug(CPUState *cpu, vaddr addr);
 
-bool openrisc_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                           MMUAccessType access_type, int mmu_idx,
-                           bool probe, uintptr_t retaddr);
+bool openrisc_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
+                                 vaddr addr, MMUAccessType access_type,
+                                 int mmu_idx, MemOp memop, int size,
+                                 bool probe, uintptr_t ra);
 
 extern const VMStateDescription vmstate_openrisc_cpu;
 
diff --git a/target/openrisc/cpu.c b/target/openrisc/cpu.c
index b96561d1f2..6aa04ff7d3 100644
--- a/target/openrisc/cpu.c
+++ b/target/openrisc/cpu.c
@@ -237,7 +237,7 @@ static const TCGCPUOps openrisc_tcg_ops = {
     .restore_state_to_opc = openrisc_restore_state_to_opc,
 
 #ifndef CONFIG_USER_ONLY
-    .tlb_fill = openrisc_cpu_tlb_fill,
+    .tlb_fill_align = openrisc_cpu_tlb_fill_align,
     .cpu_exec_interrupt = openrisc_cpu_exec_interrupt,
     .cpu_exec_halt = openrisc_cpu_has_work,
     .do_interrupt = openrisc_cpu_do_interrupt,
diff --git a/target/openrisc/mmu.c b/target/openrisc/mmu.c
index c632d5230b..eafab356a6 100644
--- a/target/openrisc/mmu.c
+++ b/target/openrisc/mmu.c
@@ -104,39 +104,42 @@ static void raise_mmu_exception(OpenRISCCPU *cpu, target_ulong address,
     cpu->env.lock_addr = -1;
 }
 
-bool openrisc_cpu_tlb_fill(CPUState *cs, vaddr addr, int size,
-                           MMUAccessType access_type, int mmu_idx,
-                           bool probe, uintptr_t retaddr)
+bool openrisc_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
+                                 vaddr addr, MMUAccessType access_type,
+                                 int mmu_idx, MemOp memop, int size,
+                                 bool probe, uintptr_t retaddr)
 {
     OpenRISCCPU *cpu = OPENRISC_CPU(cs);
-    int excp = EXCP_DPF;
     int prot;
     hwaddr phys_addr;
 
+    /* TODO: alignment faults not currently handled. */
+
     if (mmu_idx == MMU_NOMMU_IDX) {
         /* The mmu is disabled; lookups never fail.  */
         get_phys_nommu(&phys_addr, &prot, addr);
-        excp = 0;
     } else {
         bool super = mmu_idx == MMU_SUPERVISOR_IDX;
         int need = (access_type == MMU_INST_FETCH ? PAGE_EXEC
                     : access_type == MMU_DATA_STORE ? PAGE_WRITE
                     : PAGE_READ);
-        excp = get_phys_mmu(cpu, &phys_addr, &prot, addr, need, super);
+        int excp = get_phys_mmu(cpu, &phys_addr, &prot, addr, need, super);
+
+        if (unlikely(excp)) {
+            if (probe) {
+                return false;
+            }
+            raise_mmu_exception(cpu, addr, excp);
+            cpu_loop_exit_restore(cs, retaddr);
+        }
     }
 
-    if (likely(excp == 0)) {
-        tlb_set_page(cs, addr & TARGET_PAGE_MASK,
-                     phys_addr & TARGET_PAGE_MASK, prot,
-                     mmu_idx, TARGET_PAGE_SIZE);
-        return true;
-    }
-    if (probe) {
-        return false;
-    }
-
-    raise_mmu_exception(cpu, addr, excp);
-    cpu_loop_exit_restore(cs, retaddr);
+    memset(out, 0, sizeof(*out));
+    out->phys_addr = phys_addr;
+    out->prot = prot;
+    out->lg_page_size = TARGET_PAGE_BITS;
+    out->attrs = MEMTXATTRS_UNSPECIFIED;
+    return true;
 }
 
 hwaddr openrisc_cpu_get_phys_page_debug(CPUState *cs, vaddr addr)
-- 
2.43.0

[PATCH v2 43/54] target/ppc: Convert to TCGCPUOps.tlb_fill_align

[Richard Henderson]

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 target/ppc/internal.h   |  7 ++++---
 target/ppc/cpu_init.c   |  2 +-
 target/ppc/mmu_helper.c | 21 ++++++++++++++++-----
 3 files changed, 21 insertions(+), 9 deletions(-)

diff --git a/target/ppc/internal.h b/target/ppc/internal.h
index 20fb2ec593..9d132d35a1 100644
--- a/target/ppc/internal.h
+++ b/target/ppc/internal.h
@@ -273,9 +273,10 @@ void ppc_cpu_record_sigsegv(CPUState *cs, vaddr addr,
                             MMUAccessType access_type,
                             bool maperr, uintptr_t ra);
 #else
-bool ppc_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                      MMUAccessType access_type, int mmu_idx,
-                      bool probe, uintptr_t retaddr);
+bool ppc_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
+                            vaddr addr, MMUAccessType access_type,
+                            int mmu_idx, MemOp memop, int size,
+                            bool probe, uintptr_t ra);
 G_NORETURN void ppc_cpu_do_unaligned_access(CPUState *cs, vaddr addr,
                                             MMUAccessType access_type, int mmu_idx,
                                             uintptr_t retaddr);
diff --git a/target/ppc/cpu_init.c b/target/ppc/cpu_init.c
index efcb80d1c2..387c7ff2da 100644
--- a/target/ppc/cpu_init.c
+++ b/target/ppc/cpu_init.c
@@ -7422,7 +7422,7 @@ static const TCGCPUOps ppc_tcg_ops = {
 #ifdef CONFIG_USER_ONLY
   .record_sigsegv = ppc_cpu_record_sigsegv,
 #else
-  .tlb_fill = ppc_cpu_tlb_fill,
+  .tlb_fill_align = ppc_cpu_tlb_fill_align,
   .cpu_exec_interrupt = ppc_cpu_exec_interrupt,
   .cpu_exec_halt = ppc_cpu_has_work,
   .do_interrupt = ppc_cpu_do_interrupt,
diff --git a/target/ppc/mmu_helper.c b/target/ppc/mmu_helper.c
index b167b37e0a..bf98e0efb0 100644
--- a/target/ppc/mmu_helper.c
+++ b/target/ppc/mmu_helper.c
@@ -1357,18 +1357,29 @@ void helper_check_tlb_flush_global(CPUPPCState *env)
 }
 
 
-bool ppc_cpu_tlb_fill(CPUState *cs, vaddr eaddr, int size,
-                      MMUAccessType access_type, int mmu_idx,
-                      bool probe, uintptr_t retaddr)
+bool ppc_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
+                            vaddr eaddr, MMUAccessType access_type,
+                            int mmu_idx, MemOp memop, int size,
+                            bool probe, uintptr_t retaddr)
 {
     PowerPCCPU *cpu = POWERPC_CPU(cs);
     hwaddr raddr;
     int page_size, prot;
 
+    if (eaddr & ((1 << memop_alignment_bits(memop)) - 1)) {
+        if (probe) {
+            return false;
+        }
+        ppc_cpu_do_unaligned_access(cs, eaddr, access_type, mmu_idx, retaddr);
+    }
+
     if (ppc_xlate(cpu, eaddr, access_type, &raddr,
                   &page_size, &prot, mmu_idx, !probe)) {
-        tlb_set_page(cs, eaddr & TARGET_PAGE_MASK, raddr & TARGET_PAGE_MASK,
-                     prot, mmu_idx, 1UL << page_size);
+        memset(out, 0, sizeof(*out));
+        out->phys_addr = raddr;
+        out->prot = prot;
+        out->lg_page_size = page_size;
+        out->attrs = MEMTXATTRS_UNSPECIFIED;
         return true;
     }
     if (probe) {
-- 
2.43.0

[PATCH v2 44/54] target/riscv: Convert to TCGCPUOps.tlb_fill_align

[Richard Henderson]

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 target/riscv/cpu.h         |  8 +++++---
 target/riscv/cpu_helper.c  | 22 +++++++++++++++++-----
 target/riscv/tcg/tcg-cpu.c |  2 +-
 3 files changed, 23 insertions(+), 9 deletions(-)

diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h
index 284b112821..f97c4f3410 100644
--- a/target/riscv/cpu.h
+++ b/target/riscv/cpu.h
@@ -25,6 +25,7 @@
 #include "hw/qdev-properties.h"
 #include "exec/cpu-defs.h"
 #include "exec/gdbstub.h"
+#include "exec/memop.h"
 #include "qemu/cpu-float.h"
 #include "qom/object.h"
 #include "qemu/int128.h"
@@ -563,9 +564,10 @@ bool cpu_get_bcfien(CPURISCVState *env);
 G_NORETURN void  riscv_cpu_do_unaligned_access(CPUState *cs, vaddr addr,
                                                MMUAccessType access_type,
                                                int mmu_idx, uintptr_t retaddr);
-bool riscv_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                        MMUAccessType access_type, int mmu_idx,
-                        bool probe, uintptr_t retaddr);
+bool riscv_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
+                              vaddr addr, MMUAccessType access_type,
+                              int mmu_idx, MemOp memop, int size,
+                              bool probe, uintptr_t ra);
 char *riscv_isa_string(RISCVCPU *cpu);
 int riscv_cpu_max_xlen(RISCVCPUClass *mcc);
 bool riscv_cpu_option_set(const char *optname);
diff --git a/target/riscv/cpu_helper.c b/target/riscv/cpu_helper.c
index 0a3ead69ea..edb2edfc55 100644
--- a/target/riscv/cpu_helper.c
+++ b/target/riscv/cpu_helper.c
@@ -1429,9 +1429,10 @@ static void pmu_tlb_fill_incr_ctr(RISCVCPU *cpu, MMUAccessType access_type)
     riscv_pmu_incr_ctr(cpu, pmu_event_type);
 }
 
-bool riscv_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                        MMUAccessType access_type, int mmu_idx,
-                        bool probe, uintptr_t retaddr)
+bool riscv_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
+                              vaddr address, MMUAccessType access_type,
+                              int mmu_idx, MemOp memop, int size,
+                              bool probe, uintptr_t retaddr)
 {
     RISCVCPU *cpu = RISCV_CPU(cs);
     CPURISCVState *env = &cpu->env;
@@ -1452,6 +1453,14 @@ bool riscv_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
     qemu_log_mask(CPU_LOG_MMU, "%s ad %" VADDR_PRIx " rw %d mmu_idx %d\n",
                   __func__, address, access_type, mmu_idx);
 
+    if (address & ((1 << memop_alignment_bits(memop)) - 1)) {
+        if (probe) {
+            return false;
+        }
+        riscv_cpu_do_unaligned_access(cs, address, access_type,
+                                      mmu_idx, retaddr);
+    }
+
     pmu_tlb_fill_incr_ctr(cpu, access_type);
     if (two_stage_lookup) {
         /* Two stage lookup */
@@ -1544,8 +1553,11 @@ bool riscv_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
     }
 
     if (ret == TRANSLATE_SUCCESS) {
-        tlb_set_page(cs, address & ~(tlb_size - 1), pa & ~(tlb_size - 1),
-                     prot, mmu_idx, tlb_size);
+        memset(out, 0, sizeof(*out));
+        out->phys_addr = pa;
+        out->prot = prot;
+        out->lg_page_size = ctz64(tlb_size);
+        out->attrs = MEMTXATTRS_UNSPECIFIED;
         return true;
     } else if (probe) {
         return false;
diff --git a/target/riscv/tcg/tcg-cpu.c b/target/riscv/tcg/tcg-cpu.c
index c62c221696..f3b436bb86 100644
--- a/target/riscv/tcg/tcg-cpu.c
+++ b/target/riscv/tcg/tcg-cpu.c
@@ -138,7 +138,7 @@ static const TCGCPUOps riscv_tcg_ops = {
     .restore_state_to_opc = riscv_restore_state_to_opc,
 
 #ifndef CONFIG_USER_ONLY
-    .tlb_fill = riscv_cpu_tlb_fill,
+    .tlb_fill_align = riscv_cpu_tlb_fill_align,
     .cpu_exec_interrupt = riscv_cpu_exec_interrupt,
     .cpu_exec_halt = riscv_cpu_has_work,
     .do_interrupt = riscv_cpu_do_interrupt,
-- 
2.43.0

[PATCH v2 45/54] target/rx: Convert to TCGCPUOps.tlb_fill_align

[Richard Henderson]

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 target/rx/cpu.c | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/target/rx/cpu.c b/target/rx/cpu.c
index 65a74ce720..c83a582141 100644
--- a/target/rx/cpu.c
+++ b/target/rx/cpu.c
@@ -161,16 +161,19 @@ static void rx_cpu_disas_set_info(CPUState *cpu, disassemble_info *info)
     info->print_insn = print_insn_rx;
 }
 
-static bool rx_cpu_tlb_fill(CPUState *cs, vaddr addr, int size,
-                            MMUAccessType access_type, int mmu_idx,
-                            bool probe, uintptr_t retaddr)
+static bool rx_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
+                                  vaddr addr, MMUAccessType access_type,
+                                  int mmu_idx, MemOp memop, int size,
+                                  bool probe, uintptr_t retaddr)
 {
-    uint32_t address, physical, prot;
+    /* TODO: alignment faults not currently handled. */
 
     /* Linear mapping */
-    address = physical = addr & TARGET_PAGE_MASK;
-    prot = PAGE_READ | PAGE_WRITE | PAGE_EXEC;
-    tlb_set_page(cs, address, physical, prot, mmu_idx, TARGET_PAGE_SIZE);
+    memset(out, 0, sizeof(*out));
+    out->phys_addr = addr;
+    out->prot = PAGE_READ | PAGE_WRITE | PAGE_EXEC;
+    out->lg_page_size = TARGET_PAGE_BITS;
+    out->attrs = MEMTXATTRS_UNSPECIFIED;
     return true;
 }
 
@@ -195,7 +198,7 @@ static const TCGCPUOps rx_tcg_ops = {
     .initialize = rx_translate_init,
     .synchronize_from_tb = rx_cpu_synchronize_from_tb,
     .restore_state_to_opc = rx_restore_state_to_opc,
-    .tlb_fill = rx_cpu_tlb_fill,
+    .tlb_fill_align = rx_cpu_tlb_fill_align,
 
 #ifndef CONFIG_USER_ONLY
     .cpu_exec_interrupt = rx_cpu_exec_interrupt,
-- 
2.43.0

[PATCH v2 46/54] target/s390x: Convert to TCGCPUOps.tlb_fill_align

[Richard Henderson]

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 target/s390x/s390x-internal.h  |  7 ++++---
 target/s390x/cpu.c             |  4 ++--
 target/s390x/tcg/excp_helper.c | 23 ++++++++++++++++++-----
 3 files changed, 24 insertions(+), 10 deletions(-)

diff --git a/target/s390x/s390x-internal.h b/target/s390x/s390x-internal.h
index 825252d728..eb6fe24c9a 100644
--- a/target/s390x/s390x-internal.h
+++ b/target/s390x/s390x-internal.h
@@ -278,9 +278,10 @@ void s390_cpu_record_sigsegv(CPUState *cs, vaddr address,
 void s390_cpu_record_sigbus(CPUState *cs, vaddr address,
                             MMUAccessType access_type, uintptr_t retaddr);
 #else
-bool s390_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                       MMUAccessType access_type, int mmu_idx,
-                       bool probe, uintptr_t retaddr);
+bool s390x_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
+                              vaddr addr, MMUAccessType access_type,
+                              int mmu_idx, MemOp memop, int size,
+                              bool probe, uintptr_t retaddr);
 G_NORETURN void s390x_cpu_do_unaligned_access(CPUState *cs, vaddr addr,
                                               MMUAccessType access_type, int mmu_idx,
                                               uintptr_t retaddr);
diff --git a/target/s390x/cpu.c b/target/s390x/cpu.c
index 514c70f301..4d0eb129e3 100644
--- a/target/s390x/cpu.c
+++ b/target/s390x/cpu.c
@@ -330,7 +330,7 @@ void cpu_get_tb_cpu_state(CPUS390XState *env, vaddr *pc,
          * Instructions must be at even addresses.
          * This needs to be checked before address translation.
          */
-        env->int_pgm_ilen = 2; /* see s390_cpu_tlb_fill() */
+        env->int_pgm_ilen = 2; /* see s390x_cpu_tlb_fill_align() */
         tcg_s390_program_interrupt(env, PGM_SPECIFICATION, 0);
     }
 
@@ -364,7 +364,7 @@ static const TCGCPUOps s390_tcg_ops = {
     .record_sigsegv = s390_cpu_record_sigsegv,
     .record_sigbus = s390_cpu_record_sigbus,
 #else
-    .tlb_fill = s390_cpu_tlb_fill,
+    .tlb_fill_align = s390x_cpu_tlb_fill_align,
     .cpu_exec_interrupt = s390_cpu_exec_interrupt,
     .cpu_exec_halt = s390_cpu_has_work,
     .do_interrupt = s390_cpu_do_interrupt,
diff --git a/target/s390x/tcg/excp_helper.c b/target/s390x/tcg/excp_helper.c
index 4c0b692c9e..6d61032a4a 100644
--- a/target/s390x/tcg/excp_helper.c
+++ b/target/s390x/tcg/excp_helper.c
@@ -139,9 +139,10 @@ static inline uint64_t cpu_mmu_idx_to_asc(int mmu_idx)
     }
 }
 
-bool s390_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                       MMUAccessType access_type, int mmu_idx,
-                       bool probe, uintptr_t retaddr)
+bool s390x_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
+                              vaddr address, MMUAccessType access_type,
+                              int mmu_idx, MemOp memop, int size,
+                              bool probe, uintptr_t retaddr)
 {
     CPUS390XState *env = cpu_env(cs);
     target_ulong vaddr, raddr;
@@ -151,6 +152,14 @@ bool s390_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
     qemu_log_mask(CPU_LOG_MMU, "%s: addr 0x%" VADDR_PRIx " rw %d mmu_idx %d\n",
                   __func__, address, access_type, mmu_idx);
 
+    if (address & ((1 << memop_alignment_bits(memop)) - 1)) {
+        if (probe) {
+            return false;
+        }
+        s390x_cpu_do_unaligned_access(cs, address, access_type,
+                                      mmu_idx, retaddr);
+    }
+
     vaddr = address;
 
     if (mmu_idx < MMU_REAL_IDX) {
@@ -177,8 +186,12 @@ bool s390_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
         qemu_log_mask(CPU_LOG_MMU,
                       "%s: set tlb %" PRIx64 " -> %" PRIx64 " (%x)\n",
                       __func__, (uint64_t)vaddr, (uint64_t)raddr, prot);
-        tlb_set_page(cs, address & TARGET_PAGE_MASK, raddr, prot,
-                     mmu_idx, TARGET_PAGE_SIZE);
+
+        memset(out, 0, sizeof(*out));
+        out->phys_addr = raddr;
+        out->prot = prot;
+        out->lg_page_size = TARGET_PAGE_BITS;
+        out->attrs = MEMTXATTRS_UNSPECIFIED;
         return true;
     }
     if (probe) {
-- 
2.43.0

[PATCH v2 47/54] target/sh4: Convert to TCGCPUOps.tlb_fill_align

[Richard Henderson]

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 target/sh4/cpu.h    |  8 +++++---
 target/sh4/cpu.c    |  2 +-
 target/sh4/helper.c | 24 +++++++++++++++++-------
 3 files changed, 23 insertions(+), 11 deletions(-)

diff --git a/target/sh4/cpu.h b/target/sh4/cpu.h
index d928bcf006..161efdefcf 100644
--- a/target/sh4/cpu.h
+++ b/target/sh4/cpu.h
@@ -22,6 +22,7 @@
 
 #include "cpu-qom.h"
 #include "exec/cpu-defs.h"
+#include "exec/memop.h"
 #include "qemu/cpu-float.h"
 
 /* CPU Subtypes */
@@ -251,9 +252,10 @@ void sh4_translate_init(void);
 
 #if !defined(CONFIG_USER_ONLY)
 hwaddr superh_cpu_get_phys_page_debug(CPUState *cpu, vaddr addr);
-bool superh_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                         MMUAccessType access_type, int mmu_idx,
-                         bool probe, uintptr_t retaddr);
+bool superh_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
+                               vaddr addr, MMUAccessType access_type,
+                               int mmu_idx, MemOp memop, int size,
+                               bool probe, uintptr_t retaddr);
 void superh_cpu_do_interrupt(CPUState *cpu);
 bool superh_cpu_exec_interrupt(CPUState *cpu, int int_req);
 void cpu_sh4_invalidate_tlb(CPUSH4State *s);
diff --git a/target/sh4/cpu.c b/target/sh4/cpu.c
index 8f07261dcf..8ca8b90e3c 100644
--- a/target/sh4/cpu.c
+++ b/target/sh4/cpu.c
@@ -252,7 +252,7 @@ static const TCGCPUOps superh_tcg_ops = {
     .restore_state_to_opc = superh_restore_state_to_opc,
 
 #ifndef CONFIG_USER_ONLY
-    .tlb_fill = superh_cpu_tlb_fill,
+    .tlb_fill_align = superh_cpu_tlb_fill_align,
     .cpu_exec_interrupt = superh_cpu_exec_interrupt,
     .cpu_exec_halt = superh_cpu_has_work,
     .do_interrupt = superh_cpu_do_interrupt,
diff --git a/target/sh4/helper.c b/target/sh4/helper.c
index 9659c69550..543ac1b843 100644
--- a/target/sh4/helper.c
+++ b/target/sh4/helper.c
@@ -792,22 +792,32 @@ bool superh_cpu_exec_interrupt(CPUState *cs, int interrupt_request)
     return false;
 }
 
-bool superh_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                         MMUAccessType access_type, int mmu_idx,
-                         bool probe, uintptr_t retaddr)
+bool superh_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
+                               vaddr address, MMUAccessType access_type,
+                               int mmu_idx, MemOp memop, int size,
+                               bool probe, uintptr_t retaddr)
 {
     CPUSH4State *env = cpu_env(cs);
     int ret;
-
     target_ulong physical;
     int prot;
 
+    if (address & ((1 << memop_alignment_bits(memop)) - 1)) {
+        if (probe) {
+            return false;
+        }
+        superh_cpu_do_unaligned_access(cs, address, access_type,
+                                       mmu_idx, retaddr);
+    }
+
     ret = get_physical_address(env, &physical, &prot, address, access_type);
 
     if (ret == MMU_OK) {
-        address &= TARGET_PAGE_MASK;
-        physical &= TARGET_PAGE_MASK;
-        tlb_set_page(cs, address, physical, prot, mmu_idx, TARGET_PAGE_SIZE);
+        memset(out, 0, sizeof(*out));
+        out->phys_addr = physical;
+        out->prot = prot;
+        out->lg_page_size = TARGET_PAGE_BITS;
+        out->attrs = MEMTXATTRS_UNSPECIFIED;
         return true;
     }
     if (probe) {
-- 
2.43.0

[PATCH v2 48/54] target/sparc: Convert to TCGCPUOps.tlb_fill_align

[Richard Henderson]

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 target/sparc/cpu.h        |  8 ++++---
 target/sparc/cpu.c        |  2 +-
 target/sparc/mmu_helper.c | 44 +++++++++++++++++++++++++--------------
 3 files changed, 34 insertions(+), 20 deletions(-)

diff --git a/target/sparc/cpu.h b/target/sparc/cpu.h
index f517e5a383..4c8927e9fa 100644
--- a/target/sparc/cpu.h
+++ b/target/sparc/cpu.h
@@ -4,6 +4,7 @@
 #include "qemu/bswap.h"
 #include "cpu-qom.h"
 #include "exec/cpu-defs.h"
+#include "exec/memop.h"
 #include "qemu/cpu-float.h"
 
 #if !defined(TARGET_SPARC64)
@@ -596,9 +597,10 @@ G_NORETURN void cpu_raise_exception_ra(CPUSPARCState *, int, uintptr_t);
 void cpu_sparc_set_id(CPUSPARCState *env, unsigned int cpu);
 void sparc_cpu_list(void);
 /* mmu_helper.c */
-bool sparc_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                        MMUAccessType access_type, int mmu_idx,
-                        bool probe, uintptr_t retaddr);
+bool sparc_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
+                              vaddr addr, MMUAccessType access_type,
+                              int mmu_idx, MemOp memop, int size,
+                              bool probe, uintptr_t retaddr);
 target_ulong mmu_probe(CPUSPARCState *env, target_ulong address, int mmulev);
 void dump_mmu(CPUSPARCState *env);
 
diff --git a/target/sparc/cpu.c b/target/sparc/cpu.c
index dd7af86de7..57ae53bd71 100644
--- a/target/sparc/cpu.c
+++ b/target/sparc/cpu.c
@@ -932,7 +932,7 @@ static const TCGCPUOps sparc_tcg_ops = {
     .restore_state_to_opc = sparc_restore_state_to_opc,
 
 #ifndef CONFIG_USER_ONLY
-    .tlb_fill = sparc_cpu_tlb_fill,
+    .tlb_fill_align = sparc_cpu_tlb_fill_align,
     .cpu_exec_interrupt = sparc_cpu_exec_interrupt,
     .cpu_exec_halt = sparc_cpu_has_work,
     .do_interrupt = sparc_cpu_do_interrupt,
diff --git a/target/sparc/mmu_helper.c b/target/sparc/mmu_helper.c
index 9ff06026b8..32766a37d6 100644
--- a/target/sparc/mmu_helper.c
+++ b/target/sparc/mmu_helper.c
@@ -203,12 +203,12 @@ static int get_physical_address(CPUSPARCState *env, CPUTLBEntryFull *full,
 }
 
 /* Perform address translation */
-bool sparc_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                        MMUAccessType access_type, int mmu_idx,
-                        bool probe, uintptr_t retaddr)
+bool sparc_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
+                              vaddr address, MMUAccessType access_type,
+                              int mmu_idx, MemOp memop, int size,
+                              bool probe, uintptr_t retaddr)
 {
     CPUSPARCState *env = cpu_env(cs);
-    CPUTLBEntryFull full = {};
     target_ulong vaddr;
     int error_code = 0, access_index;
 
@@ -220,16 +220,21 @@ bool sparc_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
      */
     assert(!probe);
 
+    if (address & ((1 << memop_alignment_bits(memop)) - 1)) {
+        sparc_cpu_do_unaligned_access(cs, address, access_type,
+                                      mmu_idx, retaddr);
+    }
+
+    memset(out, 0, sizeof(*out));
     address &= TARGET_PAGE_MASK;
-    error_code = get_physical_address(env, &full, &access_index,
+    error_code = get_physical_address(env, out, &access_index,
                                       address, access_type, mmu_idx);
     vaddr = address;
     if (likely(error_code == 0)) {
         qemu_log_mask(CPU_LOG_MMU,
                       "Translate at %" VADDR_PRIx " -> "
                       HWADDR_FMT_plx ", vaddr " TARGET_FMT_lx "\n",
-                      address, full.phys_addr, vaddr);
-        tlb_set_page_full(cs, mmu_idx, vaddr, &full);
+                      address, out->phys_addr, vaddr);
         return true;
     }
 
@@ -244,8 +249,7 @@ bool sparc_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
            permissions. If no mapping is available, redirect accesses to
            neverland. Fake/overridden mappings will be flushed when
            switching to normal mode. */
-        full.prot = PAGE_READ | PAGE_WRITE | PAGE_EXEC;
-        tlb_set_page_full(cs, mmu_idx, vaddr, &full);
+        out->prot = PAGE_READ | PAGE_WRITE | PAGE_EXEC;
         return true;
     } else {
         if (access_type == MMU_INST_FETCH) {
@@ -754,22 +758,30 @@ static int get_physical_address(CPUSPARCState *env, CPUTLBEntryFull *full,
 }
 
 /* Perform address translation */
-bool sparc_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                        MMUAccessType access_type, int mmu_idx,
-                        bool probe, uintptr_t retaddr)
+bool sparc_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
+                              vaddr address, MMUAccessType access_type,
+                              int mmu_idx, MemOp memop, int size,
+                              bool probe, uintptr_t retaddr)
 {
     CPUSPARCState *env = cpu_env(cs);
-    CPUTLBEntryFull full = {};
     int error_code = 0, access_index;
 
+    if (address & ((1 << memop_alignment_bits(memop)) - 1)) {
+        if (probe) {
+            return false;
+        }
+        sparc_cpu_do_unaligned_access(cs, address, access_type,
+                                      mmu_idx, retaddr);
+    }
+
+    memset(out, 0, sizeof(*out));
     address &= TARGET_PAGE_MASK;
-    error_code = get_physical_address(env, &full, &access_index,
+    error_code = get_physical_address(env, out, &access_index,
                                       address, access_type, mmu_idx);
     if (likely(error_code == 0)) {
-        trace_mmu_helper_mmu_fault(address, full.phys_addr, mmu_idx, env->tl,
+        trace_mmu_helper_mmu_fault(address, out->phys_addr, mmu_idx, env->tl,
                                    env->dmmu.mmu_primary_context,
                                    env->dmmu.mmu_secondary_context);
-        tlb_set_page_full(cs, mmu_idx, address, &full);
         return true;
     }
     if (probe) {
-- 
2.43.0

[PATCH v2 49/54] target/tricore: Convert to TCGCPUOps.tlb_fill_align

[Richard Henderson]

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 target/tricore/cpu.h    |  7 ++++---
 target/tricore/cpu.c    |  2 +-
 target/tricore/helper.c | 19 ++++++++++++-------
 3 files changed, 17 insertions(+), 11 deletions(-)

diff --git a/target/tricore/cpu.h b/target/tricore/cpu.h
index 220af69fc2..5f141ce8f3 100644
--- a/target/tricore/cpu.h
+++ b/target/tricore/cpu.h
@@ -268,8 +268,9 @@ static inline void cpu_get_tb_cpu_state(CPUTriCoreState *env, vaddr *pc,
 #define CPU_RESOLVING_TYPE TYPE_TRICORE_CPU
 
 /* helpers.c */
-bool tricore_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                          MMUAccessType access_type, int mmu_idx,
-                          bool probe, uintptr_t retaddr);
+bool tricore_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
+                                vaddr addr, MMUAccessType access_type,
+                                int mmu_idx, MemOp memop, int size,
+                                bool probe, uintptr_t retaddr);
 
 #endif /* TRICORE_CPU_H */
diff --git a/target/tricore/cpu.c b/target/tricore/cpu.c
index 1a26171590..29e0b5d129 100644
--- a/target/tricore/cpu.c
+++ b/target/tricore/cpu.c
@@ -173,7 +173,7 @@ static const TCGCPUOps tricore_tcg_ops = {
     .initialize = tricore_tcg_init,
     .synchronize_from_tb = tricore_cpu_synchronize_from_tb,
     .restore_state_to_opc = tricore_restore_state_to_opc,
-    .tlb_fill = tricore_cpu_tlb_fill,
+    .tlb_fill_align = tricore_cpu_tlb_fill_align,
     .cpu_exec_interrupt = tricore_cpu_exec_interrupt,
     .cpu_exec_halt = tricore_cpu_has_work,
 };
diff --git a/target/tricore/helper.c b/target/tricore/helper.c
index 7014255f77..8c6bf63298 100644
--- a/target/tricore/helper.c
+++ b/target/tricore/helper.c
@@ -64,16 +64,19 @@ static void raise_mmu_exception(CPUTriCoreState *env, target_ulong address,
 {
 }
 
-bool tricore_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                          MMUAccessType rw, int mmu_idx,
-                          bool probe, uintptr_t retaddr)
+bool tricore_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
+                                vaddr address, MMUAccessType access_type,
+                                int mmu_idx, MemOp memop, int size,
+                                bool probe, uintptr_t retaddr)
 {
     CPUTriCoreState *env = cpu_env(cs);
     hwaddr physical;
     int prot;
     int ret = 0;
+    int rw = access_type & 1;
+
+    /* TODO: alignment faults not currently handled. */
 
-    rw &= 1;
     ret = get_physical_address(env, &physical, &prot,
                                address, rw, mmu_idx);
 
@@ -82,9 +85,11 @@ bool tricore_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
                   __func__, address, ret, physical, prot);
 
     if (ret == TLBRET_MATCH) {
-        tlb_set_page(cs, address & TARGET_PAGE_MASK,
-                     physical & TARGET_PAGE_MASK, prot | PAGE_EXEC,
-                     mmu_idx, TARGET_PAGE_SIZE);
+        memset(out, 0, sizeof(*out));
+        out->phys_addr = physical;
+        out->prot = prot | PAGE_EXEC;
+        out->lg_page_size = TARGET_PAGE_BITS;
+        out->attrs = MEMTXATTRS_UNSPECIFIED;
         return true;
     } else {
         assert(ret < 0);
-- 
2.43.0

[PATCH v2 50/54] target/xtensa: Convert to TCGCPUOps.tlb_fill_align

[Richard Henderson]

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 target/xtensa/cpu.h    |  8 +++++---
 target/xtensa/cpu.c    |  2 +-
 target/xtensa/helper.c | 28 ++++++++++++++++++++--------
 3 files changed, 26 insertions(+), 12 deletions(-)

diff --git a/target/xtensa/cpu.h b/target/xtensa/cpu.h
index 77e48eef19..68c3d90d41 100644
--- a/target/xtensa/cpu.h
+++ b/target/xtensa/cpu.h
@@ -31,6 +31,7 @@
 #include "cpu-qom.h"
 #include "qemu/cpu-float.h"
 #include "exec/cpu-defs.h"
+#include "exec/memop.h"
 #include "hw/clock.h"
 #include "xtensa-isa.h"
 
@@ -580,9 +581,10 @@ struct XtensaCPUClass {
 };
 
 #ifndef CONFIG_USER_ONLY
-bool xtensa_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                         MMUAccessType access_type, int mmu_idx,
-                         bool probe, uintptr_t retaddr);
+bool xtensa_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
+                               vaddr addr, MMUAccessType access_type,
+                               int mmu_idx, MemOp memop, int size,
+                               bool probe, uintptr_t retaddr);
 void xtensa_cpu_do_interrupt(CPUState *cpu);
 bool xtensa_cpu_exec_interrupt(CPUState *cpu, int interrupt_request);
 void xtensa_cpu_do_transaction_failed(CPUState *cs, hwaddr physaddr, vaddr addr,
diff --git a/target/xtensa/cpu.c b/target/xtensa/cpu.c
index 6f9039abae..3e4ec97e0e 100644
--- a/target/xtensa/cpu.c
+++ b/target/xtensa/cpu.c
@@ -232,7 +232,7 @@ static const TCGCPUOps xtensa_tcg_ops = {
     .restore_state_to_opc = xtensa_restore_state_to_opc,
 
 #ifndef CONFIG_USER_ONLY
-    .tlb_fill = xtensa_cpu_tlb_fill,
+    .tlb_fill_align = xtensa_cpu_tlb_fill_align,
     .cpu_exec_interrupt = xtensa_cpu_exec_interrupt,
     .cpu_exec_halt = xtensa_cpu_has_work,
     .do_interrupt = xtensa_cpu_do_interrupt,
diff --git a/target/xtensa/helper.c b/target/xtensa/helper.c
index ca214b948a..69b0e661c8 100644
--- a/target/xtensa/helper.c
+++ b/target/xtensa/helper.c
@@ -261,15 +261,26 @@ void xtensa_cpu_do_unaligned_access(CPUState *cs,
                                   addr);
 }
 
-bool xtensa_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
-                         MMUAccessType access_type, int mmu_idx,
-                         bool probe, uintptr_t retaddr)
+bool xtensa_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
+                               vaddr address, MMUAccessType access_type,
+                               int mmu_idx, MemOp memop, int size,
+                               bool probe, uintptr_t retaddr)
 {
     CPUXtensaState *env = cpu_env(cs);
     uint32_t paddr;
     uint32_t page_size;
     unsigned access;
-    int ret = xtensa_get_physical_addr(env, true, address, access_type,
+    int ret;
+
+    if (address & ((1 << memop_alignment_bits(memop)) - 1)) {
+        if (probe) {
+            return false;
+        }
+        xtensa_cpu_do_unaligned_access(cs, address, access_type,
+                                       mmu_idx, retaddr);
+    }
+
+    ret = xtensa_get_physical_addr(env, true, address, access_type,
                                        mmu_idx, &paddr, &page_size, &access);
 
     qemu_log_mask(CPU_LOG_MMU, "%s(%08" VADDR_PRIx
@@ -277,10 +288,11 @@ bool xtensa_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
                   __func__, address, access_type, mmu_idx, paddr, ret);
 
     if (ret == 0) {
-        tlb_set_page(cs,
-                     address & TARGET_PAGE_MASK,
-                     paddr & TARGET_PAGE_MASK,
-                     access, mmu_idx, page_size);
+        memset(out, 0, sizeof(*out));
+        out->phys_addr = paddr;
+        out->prot = access;
+        out->lg_page_size = ctz32(page_size);
+        out->attrs = MEMTXATTRS_UNSPECIFIED;
         return true;
     } else if (probe) {
         return false;
-- 
2.43.0

[PATCH v2 51/54] accel/tcg: Drop TCGCPUOps.tlb_fill

[Richard Henderson]

Now that all targets have been converted to tlb_fill_align,
remove the tlb_fill hook.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 include/hw/core/tcg-cpu-ops.h | 10 ----------
 accel/tcg/cputlb.c            | 19 ++++---------------
 2 files changed, 4 insertions(+), 25 deletions(-)

diff --git a/include/hw/core/tcg-cpu-ops.h b/include/hw/core/tcg-cpu-ops.h
index 663efb9133..70cafcc6cd 100644
--- a/include/hw/core/tcg-cpu-ops.h
+++ b/include/hw/core/tcg-cpu-ops.h
@@ -157,16 +157,6 @@ struct TCGCPUOps {
     bool (*tlb_fill_align)(CPUState *cpu, CPUTLBEntryFull *out, vaddr addr,
                            MMUAccessType access_type, int mmu_idx,
                            MemOp memop, int size, bool probe, uintptr_t ra);
-    /**
-     * @tlb_fill: Handle a softmmu tlb miss
-     *
-     * If the access is valid, call tlb_set_page and return true;
-     * if the access is invalid and probe is true, return false;
-     * otherwise raise an exception and do not return.
-     */
-    bool (*tlb_fill)(CPUState *cpu, vaddr address, int size,
-                     MMUAccessType access_type, int mmu_idx,
-                     bool probe, uintptr_t retaddr);
     /**
      * @do_transaction_failed: Callback for handling failed memory transactions
      * (ie bus faults or external aborts; not MMU faults)
diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 7f63dc3fd8..ec597ed6f5 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -1222,23 +1222,12 @@ static bool tlb_fill_align(CPUState *cpu, vaddr addr, MMUAccessType type,
                            int mmu_idx, MemOp memop, int size,
                            bool probe, uintptr_t ra)
 {
-    const TCGCPUOps *ops = cpu->cc->tcg_ops;
     CPUTLBEntryFull full;
 
-    if (ops->tlb_fill_align) {
-        if (ops->tlb_fill_align(cpu, &full, addr, type, mmu_idx,
-                                memop, size, probe, ra)) {
-            tlb_set_page_full(cpu, mmu_idx, addr, &full);
-            return true;
-        }
-    } else {
-        /* Legacy behaviour is alignment before paging. */
-        if (addr & ((1u << memop_alignment_bits(memop)) - 1)) {
-            ops->do_unaligned_access(cpu, addr, type, mmu_idx, ra);
-        }
-        if (ops->tlb_fill(cpu, addr, size, type, mmu_idx, probe, ra)) {
-            return true;
-        }
+    if (cpu->cc->tcg_ops->tlb_fill_align(cpu, &full, addr, type, mmu_idx,
+                                         memop, size, probe, ra)) {
+        tlb_set_page_full(cpu, mmu_idx, addr, &full);
+        return true;
     }
     assert(probe);
     return false;
-- 
2.43.0

[PATCH v2 52/54] accel/tcg: Unexport tlb_set_page*

[Richard Henderson]

The new tlb_fill_align hook returns page data via structure
rather than by function call, so we can make tlb_set_page_full
be local to cputlb.c.  There are no users of tlb_set_page
or tlb_set_page_with_attrs, so those can be eliminated.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 include/exec/exec-all.h | 57 -----------------------------------------
 accel/tcg/cputlb.c      | 27 ++-----------------
 2 files changed, 2 insertions(+), 82 deletions(-)

diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
index 69bdb77584..b65fc547bd 100644
--- a/include/exec/exec-all.h
+++ b/include/exec/exec-all.h
@@ -184,63 +184,6 @@ void tlb_flush_range_by_mmuidx_all_cpus_synced(CPUState *cpu,
                                                vaddr len,
                                                uint16_t idxmap,
                                                unsigned bits);
-
-/**
- * tlb_set_page_full:
- * @cpu: CPU context
- * @mmu_idx: mmu index of the tlb to modify
- * @addr: virtual address of the entry to add
- * @full: the details of the tlb entry
- *
- * Add an entry to @cpu tlb index @mmu_idx.  All of the fields of
- * @full must be filled, except for xlat_section, and constitute
- * the complete description of the translated page.
- *
- * This is generally called by the target tlb_fill function after
- * having performed a successful page table walk to find the physical
- * address and attributes for the translation.
- *
- * At most one entry for a given virtual address is permitted. Only a
- * single TARGET_PAGE_SIZE region is mapped; @full->lg_page_size is only
- * used by tlb_flush_page.
- */
-void tlb_set_page_full(CPUState *cpu, int mmu_idx, vaddr addr,
-                       CPUTLBEntryFull *full);
-
-/**
- * tlb_set_page_with_attrs:
- * @cpu: CPU to add this TLB entry for
- * @addr: virtual address of page to add entry for
- * @paddr: physical address of the page
- * @attrs: memory transaction attributes
- * @prot: access permissions (PAGE_READ/PAGE_WRITE/PAGE_EXEC bits)
- * @mmu_idx: MMU index to insert TLB entry for
- * @size: size of the page in bytes
- *
- * Add an entry to this CPU's TLB (a mapping from virtual address
- * @addr to physical address @paddr) with the specified memory
- * transaction attributes. This is generally called by the target CPU
- * specific code after it has been called through the tlb_fill()
- * entry point and performed a successful page table walk to find
- * the physical address and attributes for the virtual address
- * which provoked the TLB miss.
- *
- * At most one entry for a given virtual address is permitted. Only a
- * single TARGET_PAGE_SIZE region is mapped; the supplied @size is only
- * used by tlb_flush_page.
- */
-void tlb_set_page_with_attrs(CPUState *cpu, vaddr addr,
-                             hwaddr paddr, MemTxAttrs attrs,
-                             int prot, int mmu_idx, vaddr size);
-/* tlb_set_page:
- *
- * This function is equivalent to calling tlb_set_page_with_attrs()
- * with an @attrs argument of MEMTXATTRS_UNSPECIFIED. It's provided
- * as a convenience for CPUs which don't use memory transaction attributes.
- */
-void tlb_set_page(CPUState *cpu, vaddr addr,
-                  hwaddr paddr, int prot,
-                  int mmu_idx, vaddr size);
 #else
 static inline void tlb_init(CPUState *cpu)
 {
diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index ec597ed6f5..3d731b8f3d 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -1037,8 +1037,8 @@ static inline void tlb_set_compare(CPUTLBEntryFull *full, CPUTLBEntry *ent,
  * Called from TCG-generated code, which is under an RCU read-side
  * critical section.
  */
-void tlb_set_page_full(CPUState *cpu, int mmu_idx,
-                       vaddr addr, CPUTLBEntryFull *full)
+static void tlb_set_page_full(CPUState *cpu, int mmu_idx,
+                              vaddr addr, CPUTLBEntryFull *full)
 {
     CPUTLB *tlb = &cpu->neg.tlb;
     CPUTLBDesc *desc = &tlb->d[mmu_idx];
@@ -1189,29 +1189,6 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
     qemu_spin_unlock(&tlb->c.lock);
 }
 
-void tlb_set_page_with_attrs(CPUState *cpu, vaddr addr,
-                             hwaddr paddr, MemTxAttrs attrs, int prot,
-                             int mmu_idx, uint64_t size)
-{
-    CPUTLBEntryFull full = {
-        .phys_addr = paddr,
-        .attrs = attrs,
-        .prot = prot,
-        .lg_page_size = ctz64(size)
-    };
-
-    assert(is_power_of_2(size));
-    tlb_set_page_full(cpu, mmu_idx, addr, &full);
-}
-
-void tlb_set_page(CPUState *cpu, vaddr addr,
-                  hwaddr paddr, int prot,
-                  int mmu_idx, uint64_t size)
-{
-    tlb_set_page_with_attrs(cpu, addr, paddr, MEMTXATTRS_UNSPECIFIED,
-                            prot, mmu_idx, size);
-}
-
 /*
  * Note: tlb_fill_align() can trigger a resize of the TLB.
  * This means that all of the caller's prior references to the TLB table
-- 
2.43.0

[PATCH v2 53/54] accel/tcg: Merge tlb_fill_align into callers

[Richard Henderson]

In tlb_lookup, we still call tlb_set_page_full.
In atomic_mmu_lookup, we're expecting noreturn.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 31 ++++++-------------------------
 1 file changed, 6 insertions(+), 25 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 3d731b8f3d..20af48c6c5 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -1189,27 +1189,6 @@ static void tlb_set_page_full(CPUState *cpu, int mmu_idx,
     qemu_spin_unlock(&tlb->c.lock);
 }
 
-/*
- * Note: tlb_fill_align() can trigger a resize of the TLB.
- * This means that all of the caller's prior references to the TLB table
- * (e.g. CPUTLBEntry pointers) must be discarded and looked up again
- * (e.g. via tlb_entry()).
- */
-static bool tlb_fill_align(CPUState *cpu, vaddr addr, MMUAccessType type,
-                           int mmu_idx, MemOp memop, int size,
-                           bool probe, uintptr_t ra)
-{
-    CPUTLBEntryFull full;
-
-    if (cpu->cc->tcg_ops->tlb_fill_align(cpu, &full, addr, type, mmu_idx,
-                                         memop, size, probe, ra)) {
-        tlb_set_page_full(cpu, mmu_idx, addr, &full);
-        return true;
-    }
-    assert(probe);
-    return false;
-}
-
 static inline void cpu_unaligned_access(CPUState *cpu, vaddr addr,
                                         MMUAccessType access_type,
                                         int mmu_idx, uintptr_t retaddr)
@@ -1281,11 +1260,13 @@ static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
     }
 
     /* Finally, query the target hook. */
-    if (!tlb_fill_align(cpu, addr, access_type, i->mmu_idx,
-                        memop, i->size, probe, i->ra)) {
+    if (!cpu->cc->tcg_ops->tlb_fill_align(cpu, &o->full, addr, access_type,
+                                          i->mmu_idx, memop, i->size,
+                                          probe, i->ra)) {
         tcg_debug_assert(probe);
         return false;
     }
+    tlb_set_page_full(cpu, i->mmu_idx, addr, &o->full);
     o->did_tlb_fill = true;
 
     if (access_type == MMU_INST_FETCH) {
@@ -1794,8 +1775,8 @@ static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
      * We have just verified that the page is writable.
      */
     if (unlikely(!(o.full.prot & PAGE_READ))) {
-        tlb_fill_align(cpu, addr, MMU_DATA_LOAD, i.mmu_idx,
-                       0, i.size, false, i.ra);
+        cpu->cc->tcg_ops->tlb_fill_align(cpu, &o.full, addr, MMU_DATA_LOAD,
+                                         i.mmu_idx, 0, i.size, false, i.ra);
         /*
          * Since we don't support reads and writes to different
          * addresses, and we do have the proper page loaded for
-- 
2.43.0

[PATCH v2 54/54] accel/tcg: Return CPUTLBEntryTree from tlb_set_page_full

[Richard Henderson]

Avoid a lookup to find the node that we have just inserted.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/cputlb.c | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 20af48c6c5..6d316e8767 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -1037,8 +1037,8 @@ static inline void tlb_set_compare(CPUTLBEntryFull *full, CPUTLBEntry *ent,
  * Called from TCG-generated code, which is under an RCU read-side
  * critical section.
  */
-static void tlb_set_page_full(CPUState *cpu, int mmu_idx,
-                              vaddr addr, CPUTLBEntryFull *full)
+static CPUTLBEntryTree *tlb_set_page_full(CPUState *cpu, int mmu_idx,
+                                          vaddr addr, CPUTLBEntryFull *full)
 {
     CPUTLB *tlb = &cpu->neg.tlb;
     CPUTLBDesc *desc = &tlb->d[mmu_idx];
@@ -1187,6 +1187,8 @@ static void tlb_set_page_full(CPUState *cpu, int mmu_idx,
     copy_tlb_helper_locked(te, &node->copy);
     desc->n_used_entries++;
     qemu_spin_unlock(&tlb->c.lock);
+
+    return node;
 }
 
 static inline void cpu_unaligned_access(CPUState *cpu, vaddr addr,
@@ -1266,18 +1268,14 @@ static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
         tcg_debug_assert(probe);
         return false;
     }
-    tlb_set_page_full(cpu, i->mmu_idx, addr, &o->full);
+    node = tlb_set_page_full(cpu, i->mmu_idx, addr, &o->full);
     o->did_tlb_fill = true;
 
     if (access_type == MMU_INST_FETCH) {
-        node = tlbtree_lookup_addr(desc, addr);
-        tcg_debug_assert(node);
         goto found_code;
     }
 
-    entry = tlbfast_entry(fast, addr);
-    cmp = tlb_read_idx(entry, access_type);
-    node = entry->tree;
+    cmp = tlb_read_idx(&node->copy, access_type);
     /*
      * With PAGE_WRITE_INV, we set TLB_INVALID_MASK immediately,
      * to force the next access through tlb_fill_align.  We've just
-- 
2.43.0

Re: [PATCH v2 01/54] util/interval-tree: Introduce interval_tree_free_nodes

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> Provide a general-purpose release-all-nodes operation, that allows
> for the IntervalTreeNode to be embeded within a larger structure.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   include/qemu/interval-tree.h | 11 +++++++++++
>   util/interval-tree.c         | 20 ++++++++++++++++++++
>   util/selfmap.c               | 13 +------------
>   3 files changed, 32 insertions(+), 12 deletions(-)
> 
> diff --git a/include/qemu/interval-tree.h b/include/qemu/interval-tree.h
> index 25006debe8..d90ea6d17f 100644
> --- a/include/qemu/interval-tree.h
> +++ b/include/qemu/interval-tree.h
> @@ -96,4 +96,15 @@ IntervalTreeNode *interval_tree_iter_first(IntervalTreeRoot *root,
>   IntervalTreeNode *interval_tree_iter_next(IntervalTreeNode *node,
>                                             uint64_t start, uint64_t last);
>   
> +/**
> + * interval_tree_free_nodes:
> + * @root: root of the tree
> + * @it_offset: offset from outermost type to IntervalTreeNode
> + *
> + * Free, via g_free, all nodes under @root.  IntervalTreeNode may
> + * not be the true type of the nodes allocated; @it_offset gives
> + * the offset from the outermost type to the IntervalTreeNode member.
> + */
> +void interval_tree_free_nodes(IntervalTreeRoot *root, size_t it_offset);
> +
>   #endif /* QEMU_INTERVAL_TREE_H */
> diff --git a/util/interval-tree.c b/util/interval-tree.c
> index 53465182e6..663d3ec222 100644
> --- a/util/interval-tree.c
> +++ b/util/interval-tree.c
> @@ -639,6 +639,16 @@ static void rb_erase_augmented_cached(RBNode *node, RBRootLeftCached *root,
>       rb_erase_augmented(node, &root->rb_root, augment);
>   }
>   
> +static void rb_node_free(RBNode *rb, size_t rb_offset)
> +{
> +    if (rb->rb_left) {
> +        rb_node_free(rb->rb_left, rb_offset);
> +    }
> +    if (rb->rb_right) {
> +        rb_node_free(rb->rb_right, rb_offset);
> +    }
> +    g_free((void *)rb - rb_offset);
> +}
>   
>   /*
>    * Interval trees.
> @@ -870,6 +880,16 @@ IntervalTreeNode *interval_tree_iter_next(IntervalTreeNode *node,
>       }
>   }
>   
> +void interval_tree_free_nodes(IntervalTreeRoot *root, size_t it_offset)
> +{
> +    if (root && root->rb_root.rb_node) {
> +        rb_node_free(root->rb_root.rb_node,
> +                     it_offset + offsetof(IntervalTreeNode, rb));
> +        root->rb_root.rb_node = NULL;
> +        root->rb_leftmost = NULL;
> +    }
> +}
> +
>   /* Occasionally useful for calling from within the debugger. */
>   #if 0
>   static void debug_interval_tree_int(IntervalTreeNode *node,
> diff --git a/util/selfmap.c b/util/selfmap.c
> index 483cb617e2..d2b86da301 100644
> --- a/util/selfmap.c
> +++ b/util/selfmap.c
> @@ -87,23 +87,12 @@ IntervalTreeRoot *read_self_maps(void)
>    * @root: an interval tree
>    *
>    * Free a tree of MapInfo structures.
> - * Since we allocated each MapInfo in one chunk, we need not consider the
> - * contents and can simply free each RBNode.
>    */
>   
> -static void free_rbnode(RBNode *n)
> -{
> -    if (n) {
> -        free_rbnode(n->rb_left);
> -        free_rbnode(n->rb_right);
> -        g_free(n);
> -    }
> -}
> -
>   void free_self_maps(IntervalTreeRoot *root)
>   {
>       if (root) {
> -        free_rbnode(root->rb_root.rb_node);
> +        interval_tree_free_nodes(root, offsetof(MapInfo, itree));
>           g_free(root);
>       }
>   }

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 02/54] accel/tcg: Split out tlbfast_flush_locked

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> We will have a need to flush only the "fast" portion
> of the tlb, allowing re-fill from the "full" portion.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 9 +++++++--
>   1 file changed, 7 insertions(+), 2 deletions(-)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index b76a4eac4e..c1838412e8 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -284,13 +284,18 @@ static void tlb_mmu_resize_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast,
>       }
>   }
>   
> -static void tlb_mmu_flush_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast)
> +static void tlbfast_flush_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast)
>   {
>       desc->n_used_entries = 0;
> +    memset(fast->table, -1, sizeof_tlb(fast));
> +}
> +
> +static void tlb_mmu_flush_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast)
> +{
> +    tlbfast_flush_locked(desc, fast);
>       desc->large_page_addr = -1;
>       desc->large_page_mask = -1;
>       desc->vindex = 0;
> -    memset(fast->table, -1, sizeof_tlb(fast));
>       memset(desc->vtable, -1, sizeof(desc->vtable));
>   }
>   

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 03/54] accel/tcg: Split out tlbfast_{index,entry}

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> Often we already have the CPUTLBDescFast structure pointer.
> Allows future code simplification.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 16 ++++++++++++----
>   1 file changed, 12 insertions(+), 4 deletions(-)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index c1838412e8..e37af24525 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -131,20 +131,28 @@ static inline uint64_t tlb_addr_write(const CPUTLBEntry *entry)
>       return tlb_read_idx(entry, MMU_DATA_STORE);
>   }
>   
> +static inline uintptr_t tlbfast_index(CPUTLBDescFast *fast, vaddr addr)
> +{
> +    return (addr >> TARGET_PAGE_BITS) & (fast->mask >> CPU_TLB_ENTRY_BITS);
> +}
> +
> +static inline CPUTLBEntry *tlbfast_entry(CPUTLBDescFast *fast, vaddr addr)
> +{
> +    return fast->table + tlbfast_index(fast, addr);
> +}
> +
>   /* Find the TLB index corresponding to the mmu_idx + address pair.  */
>   static inline uintptr_t tlb_index(CPUState *cpu, uintptr_t mmu_idx,
>                                     vaddr addr)
>   {
> -    uintptr_t size_mask = cpu->neg.tlb.f[mmu_idx].mask >> CPU_TLB_ENTRY_BITS;
> -
> -    return (addr >> TARGET_PAGE_BITS) & size_mask;
> +    return tlbfast_index(&cpu->neg.tlb.f[mmu_idx], addr);
>   }
>   
>   /* Find the TLB entry corresponding to the mmu_idx + address pair.  */
>   static inline CPUTLBEntry *tlb_entry(CPUState *cpu, uintptr_t mmu_idx,
>                                        vaddr addr)
>   {
> -    return &cpu->neg.tlb.f[mmu_idx].table[tlb_index(cpu, mmu_idx, addr)];
> +    return tlbfast_entry(&cpu->neg.tlb.f[mmu_idx], addr);
>   }
>   
>   static void tlb_window_reset(CPUTLBDesc *desc, int64_t ns,

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 05/54] accel/tcg: Fix flags usage in mmu_lookup1, atomic_mmu_lookup

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> The INVALID bit should only be auto-cleared when we have
> just called tlb_fill, not along the victim_tlb_hit path.
> 
> In atomic_mmu_lookup, rename tlb_addr to flags, as that
> is what we're actually carrying around.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 33 ++++++++++++++++++++++-----------
>   1 file changed, 22 insertions(+), 11 deletions(-)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index 46fa0ae802..77b972fd93 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -1652,7 +1652,7 @@ static bool mmu_lookup1(CPUState *cpu, MMULookupPageData *data, MemOp memop,
>       uint64_t tlb_addr = tlb_read_idx(entry, access_type);
>       bool maybe_resized = false;
>       CPUTLBEntryFull *full;
> -    int flags;
> +    int flags = TLB_FLAGS_MASK & ~TLB_FORCE_SLOW;
>   
>       /* If the TLB entry is for a different page, reload and try again.  */
>       if (!tlb_hit(tlb_addr, addr)) {
> @@ -1663,8 +1663,14 @@ static bool mmu_lookup1(CPUState *cpu, MMULookupPageData *data, MemOp memop,
>               maybe_resized = true;
>               index = tlb_index(cpu, mmu_idx, addr);
>               entry = tlb_entry(cpu, mmu_idx, addr);
> +            /*
> +             * With PAGE_WRITE_INV, we set TLB_INVALID_MASK immediately,
> +             * to force the next access through tlb_fill.  We've just
> +             * called tlb_fill, so we know that this entry *is* valid.
> +             */
> +            flags &= ~TLB_INVALID_MASK;
>           }
> -        tlb_addr = tlb_read_idx(entry, access_type) & ~TLB_INVALID_MASK;
> +        tlb_addr = tlb_read_idx(entry, access_type);
>       }
>   
>       full = &cpu->neg.tlb.d[mmu_idx].fulltlb[index];
> @@ -1814,10 +1820,10 @@ static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
>       MemOp mop = get_memop(oi);
>       uintptr_t index;
>       CPUTLBEntry *tlbe;
> -    vaddr tlb_addr;
>       void *hostaddr;
>       CPUTLBEntryFull *full;
>       bool did_tlb_fill = false;
> +    int flags;
>   
>       tcg_debug_assert(mmu_idx < NB_MMU_MODES);
>   
> @@ -1828,8 +1834,8 @@ static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
>       tlbe = tlb_entry(cpu, mmu_idx, addr);
>   
>       /* Check TLB entry and enforce page permissions.  */
> -    tlb_addr = tlb_addr_write(tlbe);
> -    if (!tlb_hit(tlb_addr, addr)) {
> +    flags = TLB_FLAGS_MASK;
> +    if (!tlb_hit(tlb_addr_write(tlbe), addr)) {
>           if (!victim_tlb_hit(cpu, mmu_idx, index, MMU_DATA_STORE,
>                               addr & TARGET_PAGE_MASK)) {
>               tlb_fill_align(cpu, addr, MMU_DATA_STORE, mmu_idx,
> @@ -1837,8 +1843,13 @@ static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
>               did_tlb_fill = true;
>               index = tlb_index(cpu, mmu_idx, addr);
>               tlbe = tlb_entry(cpu, mmu_idx, addr);
> +            /*
> +             * With PAGE_WRITE_INV, we set TLB_INVALID_MASK immediately,
> +             * to force the next access through tlb_fill.  We've just
> +             * called tlb_fill, so we know that this entry *is* valid.
> +             */
> +            flags &= ~TLB_INVALID_MASK;
>           }
> -        tlb_addr = tlb_addr_write(tlbe) & ~TLB_INVALID_MASK;
>       }
>   
>       /*
> @@ -1874,11 +1885,11 @@ static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
>           goto stop_the_world;
>       }
>   
> -    /* Collect tlb flags for read. */
> -    tlb_addr |= tlbe->addr_read;
> +    /* Collect tlb flags for read and write. */
> +    flags &= tlbe->addr_read | tlb_addr_write(tlbe);
>   
>       /* Notice an IO access or a needs-MMU-lookup access */
> -    if (unlikely(tlb_addr & (TLB_MMIO | TLB_DISCARD_WRITE))) {
> +    if (unlikely(flags & (TLB_MMIO | TLB_DISCARD_WRITE))) {
>           /* There's really nothing that can be done to
>              support this apart from stop-the-world.  */
>           goto stop_the_world;
> @@ -1887,11 +1898,11 @@ static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
>       hostaddr = (void *)((uintptr_t)addr + tlbe->addend);
>       full = &cpu->neg.tlb.d[mmu_idx].fulltlb[index];
>   
> -    if (unlikely(tlb_addr & TLB_NOTDIRTY)) {
> +    if (unlikely(flags & TLB_NOTDIRTY)) {
>           notdirty_write(cpu, addr, size, full, retaddr);
>       }
>   
> -    if (unlikely(tlb_addr & TLB_FORCE_SLOW)) {
> +    if (unlikely(flags & TLB_FORCE_SLOW)) {
>           int wp_flags = 0;
>   
>           if (full->slow_flags[MMU_DATA_STORE] & TLB_WATCHPOINT) {

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 06/54] accel/tcg: Assert non-zero length in tlb_flush_range_by_mmuidx*

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> Next patches will assume non-zero length.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index 77b972fd93..1346a26d90 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -791,6 +791,7 @@ void tlb_flush_range_by_mmuidx(CPUState *cpu, vaddr addr,
>       TLBFlushRangeData d;
>   
>       assert_cpu_is_self(cpu);
> +    assert(len != 0);
>   
>       /*
>        * If all bits are significant, and len is small,
> @@ -830,6 +831,8 @@ void tlb_flush_range_by_mmuidx_all_cpus_synced(CPUState *src_cpu,
>       TLBFlushRangeData d, *p;
>       CPUState *dst_cpu;
>   
> +    assert(len != 0);
> +
>       /*
>        * If all bits are significant, and len is small,
>        * this devolves to tlb_flush_page.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 07/54] accel/tcg: Assert bits in range in tlb_flush_range_by_mmuidx*

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> The only target that does not use TARGET_LONG_BITS is Arm, which
> only reduces bits based on TBI.  There is no point in handling
> odd combinations of parameters.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 16 ++++------------
>   1 file changed, 4 insertions(+), 12 deletions(-)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index 1346a26d90..5510f40333 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -792,20 +792,16 @@ void tlb_flush_range_by_mmuidx(CPUState *cpu, vaddr addr,
>   
>       assert_cpu_is_self(cpu);
>       assert(len != 0);
> +    assert(bits > TARGET_PAGE_BITS && bits <= TARGET_LONG_BITS);
>   
>       /*
>        * If all bits are significant, and len is small,
>        * this devolves to tlb_flush_page.
>        */
> -    if (bits >= TARGET_LONG_BITS && len <= TARGET_PAGE_SIZE) {
> +    if (bits == TARGET_LONG_BITS && len <= TARGET_PAGE_SIZE) {
>           tlb_flush_page_by_mmuidx(cpu, addr, idxmap);
>           return;
>       }
> -    /* If no page bits are significant, this devolves to tlb_flush. */
> -    if (bits < TARGET_PAGE_BITS) {
> -        tlb_flush_by_mmuidx(cpu, idxmap);
> -        return;
> -    }
>   
>       /* This should already be page aligned */
>       d.addr = addr & TARGET_PAGE_MASK;
> @@ -832,20 +828,16 @@ void tlb_flush_range_by_mmuidx_all_cpus_synced(CPUState *src_cpu,
>       CPUState *dst_cpu;
>   
>       assert(len != 0);
> +    assert(bits > TARGET_PAGE_BITS && bits <= TARGET_LONG_BITS);
>   
>       /*
>        * If all bits are significant, and len is small,
>        * this devolves to tlb_flush_page.
>        */
> -    if (bits >= TARGET_LONG_BITS && len <= TARGET_PAGE_SIZE) {
> +    if (bits == TARGET_LONG_BITS && len <= TARGET_PAGE_SIZE) {
>           tlb_flush_page_by_mmuidx_all_cpus_synced(src_cpu, addr, idxmap);
>           return;
>       }
> -    /* If no page bits are significant, this devolves to tlb_flush. */
> -    if (bits < TARGET_PAGE_BITS) {
> -        tlb_flush_by_mmuidx_all_cpus_synced(src_cpu, idxmap);
> -        return;
> -    }
>   
>       /* This should already be page aligned */
>       d.addr = addr & TARGET_PAGE_MASK;

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 08/54] accel/tcg: Flush entire tlb when a masked range wraps

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> We expect masked address spaces to be quite large, e.g. 56 bits
> for AArch64 top-byte-ignore mode.  We do not expect addr+len to
> wrap around, but it is possible with AArch64 guest flush range
> instructions.
> 
> Convert this unlikely case to a full tlb flush.  This can simplify
> the subroutines actually performing the range flush.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 10 ++++++++++
>   1 file changed, 10 insertions(+)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index 5510f40333..31c45a6213 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -802,6 +802,11 @@ void tlb_flush_range_by_mmuidx(CPUState *cpu, vaddr addr,
>           tlb_flush_page_by_mmuidx(cpu, addr, idxmap);
>           return;
>       }
> +    /* If addr+len wraps in len bits, fall back to full flush. */
> +    if (bits < TARGET_LONG_BITS && ((addr ^ (addr + len - 1)) >> bits)) {
> +        tlb_flush_by_mmuidx(cpu, idxmap);
> +        return;
> +    }
>   
>       /* This should already be page aligned */
>       d.addr = addr & TARGET_PAGE_MASK;
> @@ -838,6 +843,11 @@ void tlb_flush_range_by_mmuidx_all_cpus_synced(CPUState *src_cpu,
>           tlb_flush_page_by_mmuidx_all_cpus_synced(src_cpu, addr, idxmap);
>           return;
>       }
> +    /* If addr+len wraps in len bits, fall back to full flush. */
> +    if (bits < TARGET_LONG_BITS && ((addr ^ (addr + len - 1)) >> bits)) {
> +        tlb_flush_by_mmuidx_all_cpus_synced(src_cpu, idxmap);
> +        return;
> +    }
>   
>       /* This should already be page aligned */
>       d.addr = addr & TARGET_PAGE_MASK;

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 09/54] accel/tcg: Add IntervalTreeRoot to CPUTLBDesc

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> Add the data structures for tracking softmmu pages via
> a balanced interval tree.  So far, only initialize and
> destroy the data structure.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   include/hw/core/cpu.h |  3 +++
>   accel/tcg/cputlb.c    | 11 +++++++++++
>   2 files changed, 14 insertions(+)
> 
> diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
> index db8a6fbc6e..1ebc999a73 100644
> --- a/include/hw/core/cpu.h
> +++ b/include/hw/core/cpu.h
> @@ -35,6 +35,7 @@
>   #include "qemu/queue.h"
>   #include "qemu/lockcnt.h"
>   #include "qemu/thread.h"
> +#include "qemu/interval-tree.h"
>   #include "qom/object.h"
>   
>   typedef int (*WriteCoreDumpFunction)(const void *buf, size_t size,
> @@ -290,6 +291,8 @@ typedef struct CPUTLBDesc {
>       CPUTLBEntry vtable[CPU_VTLB_SIZE];
>       CPUTLBEntryFull vfulltlb[CPU_VTLB_SIZE];
>       CPUTLBEntryFull *fulltlb;
> +    /* All active tlb entries for this address space. */
> +    IntervalTreeRoot iroot;
>   } CPUTLBDesc;
>   
>   /*
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index 31c45a6213..aa51fc1d26 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -89,6 +89,13 @@ QEMU_BUILD_BUG_ON(sizeof(vaddr) > sizeof(run_on_cpu_data));
>   QEMU_BUILD_BUG_ON(NB_MMU_MODES > 16);
>   #define ALL_MMUIDX_BITS ((1 << NB_MMU_MODES) - 1)
>   
> +/* Extra data required to manage CPUTLBEntryFull within an interval tree. */
> +typedef struct CPUTLBEntryTree {
> +    IntervalTreeNode itree;
> +    CPUTLBEntry copy;
> +    CPUTLBEntryFull full;
> +} CPUTLBEntryTree;
> +
>   static inline size_t tlb_n_entries(CPUTLBDescFast *fast)
>   {
>       return (fast->mask >> CPU_TLB_ENTRY_BITS) + 1;
> @@ -305,6 +312,7 @@ static void tlb_mmu_flush_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast)
>       desc->large_page_mask = -1;
>       desc->vindex = 0;
>       memset(desc->vtable, -1, sizeof(desc->vtable));
> +    interval_tree_free_nodes(&desc->iroot, offsetof(CPUTLBEntryTree, itree));
>   }
>   
>   static void tlb_flush_one_mmuidx_locked(CPUState *cpu, int mmu_idx,
> @@ -326,6 +334,7 @@ static void tlb_mmu_init(CPUTLBDesc *desc, CPUTLBDescFast *fast, int64_t now)
>       fast->mask = (n_entries - 1) << CPU_TLB_ENTRY_BITS;
>       fast->table = g_new(CPUTLBEntry, n_entries);
>       desc->fulltlb = g_new(CPUTLBEntryFull, n_entries);
> +    memset(&desc->iroot, 0, sizeof(desc->iroot));
>       tlb_mmu_flush_locked(desc, fast);
>   }
>   
> @@ -365,6 +374,8 @@ void tlb_destroy(CPUState *cpu)
>   
>           g_free(fast->table);
>           g_free(desc->fulltlb);
> +        interval_tree_free_nodes(&cpu->neg.tlb.d[i].iroot,
> +                                 offsetof(CPUTLBEntryTree, itree));
>       }
>   }
>   

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 10/54] accel/tcg: Populate IntervalTree in tlb_set_page_full

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> Add or replace an entry in the IntervalTree for each
> page installed into softmmu.  We do not yet use the
> tree for anything else.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 34 ++++++++++++++++++++++++++++------
>   1 file changed, 28 insertions(+), 6 deletions(-)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index aa51fc1d26..ea6a5177de 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -305,6 +305,17 @@ static void tlbfast_flush_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast)
>       memset(fast->table, -1, sizeof_tlb(fast));
>   }
>   
> +static CPUTLBEntryTree *tlbtree_lookup_range(CPUTLBDesc *desc, vaddr s, vaddr l)
> +{
> +    IntervalTreeNode *i = interval_tree_iter_first(&desc->iroot, s, l);
> +    return i ? container_of(i, CPUTLBEntryTree, itree) : NULL;
> +}
> +
> +static CPUTLBEntryTree *tlbtree_lookup_addr(CPUTLBDesc *desc, vaddr addr)
> +{
> +    return tlbtree_lookup_range(desc, addr, addr);
> +}
> +
>   static void tlb_mmu_flush_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast)
>   {
>       tlbfast_flush_locked(desc, fast);
> @@ -1072,7 +1083,8 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
>       MemoryRegionSection *section;
>       unsigned int index, read_flags, write_flags;
>       uintptr_t addend;
> -    CPUTLBEntry *te, tn;
> +    CPUTLBEntry *te;
> +    CPUTLBEntryTree *node;
>       hwaddr iotlb, xlat, sz, paddr_page;
>       vaddr addr_page;
>       int asidx, wp_flags, prot;
> @@ -1180,6 +1192,15 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
>           tlb_n_used_entries_dec(cpu, mmu_idx);
>       }
>   
> +    /* Replace an old IntervalTree entry, or create a new one. */
> +    node = tlbtree_lookup_addr(desc, addr_page);
> +    if (!node) {
> +        node = g_new(CPUTLBEntryTree, 1);
> +        node->itree.start = addr_page;
> +        node->itree.last = addr_page + TARGET_PAGE_SIZE - 1;
> +        interval_tree_insert(&node->itree, &desc->iroot);
> +    }
> +
>       /* refill the tlb */
>       /*
>        * When memory region is ram, iotlb contains a TARGET_PAGE_BITS
> @@ -1201,15 +1222,15 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
>       full->phys_addr = paddr_page;
>   
>       /* Now calculate the new entry */
> -    tn.addend = addend - addr_page;
> +    node->copy.addend = addend - addr_page;
>   
> -    tlb_set_compare(full, &tn, addr_page, read_flags,
> +    tlb_set_compare(full, &node->copy, addr_page, read_flags,
>                       MMU_INST_FETCH, prot & PAGE_EXEC);
>   
>       if (wp_flags & BP_MEM_READ) {
>           read_flags |= TLB_WATCHPOINT;
>       }
> -    tlb_set_compare(full, &tn, addr_page, read_flags,
> +    tlb_set_compare(full, &node->copy, addr_page, read_flags,
>                       MMU_DATA_LOAD, prot & PAGE_READ);
>   
>       if (prot & PAGE_WRITE_INV) {
> @@ -1218,10 +1239,11 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
>       if (wp_flags & BP_MEM_WRITE) {
>           write_flags |= TLB_WATCHPOINT;
>       }
> -    tlb_set_compare(full, &tn, addr_page, write_flags,
> +    tlb_set_compare(full, &node->copy, addr_page, write_flags,
>                       MMU_DATA_STORE, prot & PAGE_WRITE);
>   
> -    copy_tlb_helper_locked(te, &tn);
> +    node->full = *full;
> +    copy_tlb_helper_locked(te, &node->copy);
>       tlb_n_used_entries_inc(cpu, mmu_idx);
>       qemu_spin_unlock(&tlb->c.lock);
>   }

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 11/54] accel/tcg: Remove IntervalTree entry in tlb_flush_page_locked

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> Flush a page from the IntervalTree cache.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 16 ++++++++++++----
>   1 file changed, 12 insertions(+), 4 deletions(-)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index ea6a5177de..d532d69083 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -568,6 +568,7 @@ static void tlb_flush_page_locked(CPUState *cpu, int midx, vaddr page)
>       CPUTLBDesc *desc = &cpu->neg.tlb.d[midx];
>       vaddr lp_addr = desc->large_page_addr;
>       vaddr lp_mask = desc->large_page_mask;
> +    CPUTLBEntryTree *node;
>   
>       /* Check if we need to flush due to large pages.  */
>       if ((page & lp_mask) == lp_addr) {
> @@ -575,10 +576,17 @@ static void tlb_flush_page_locked(CPUState *cpu, int midx, vaddr page)
>                     VADDR_PRIx "/%016" VADDR_PRIx ")\n",
>                     midx, lp_addr, lp_mask);
>           tlb_flush_one_mmuidx_locked(cpu, midx, get_clock_realtime());
> -    } else {
> -        tlbfast_flush_range_locked(desc, &cpu->neg.tlb.f[midx],
> -                                   page, TARGET_PAGE_SIZE, -1);
> -        tlb_flush_vtlb_page_locked(cpu, midx, page);
> +        return;
> +    }
> +
> +    tlbfast_flush_range_locked(desc, &cpu->neg.tlb.f[midx],
> +                               page, TARGET_PAGE_SIZE, -1);
> +    tlb_flush_vtlb_page_locked(cpu, midx, page);
> +
> +    node = tlbtree_lookup_addr(desc, page);
> +    if (node) {
> +        interval_tree_remove(&node->itree, &desc->iroot);
> +        g_free(node);
>       }
>   }
>   

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 12/54] accel/tcg: Remove IntervalTree entries in tlb_flush_range_locked

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> Flush a masked range of pages from the IntervalTree cache.
> When the mask is not used there is a redundant comparison,
> but that is better than duplicating code at this point.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 25 +++++++++++++++++++++++++
>   1 file changed, 25 insertions(+)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index d532d69083..e2c855f147 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -311,6 +311,13 @@ static CPUTLBEntryTree *tlbtree_lookup_range(CPUTLBDesc *desc, vaddr s, vaddr l)
>       return i ? container_of(i, CPUTLBEntryTree, itree) : NULL;
>   }
>   
> +static CPUTLBEntryTree *tlbtree_lookup_range_next(CPUTLBEntryTree *prev,
> +                                                  vaddr s, vaddr l)
> +{
> +    IntervalTreeNode *i = interval_tree_iter_next(&prev->itree, s, l);
> +    return i ? container_of(i, CPUTLBEntryTree, itree) : NULL;
> +}
> +
>   static CPUTLBEntryTree *tlbtree_lookup_addr(CPUTLBDesc *desc, vaddr addr)
>   {
>       return tlbtree_lookup_range(desc, addr, addr);
> @@ -739,6 +746,8 @@ static void tlb_flush_range_locked(CPUState *cpu, int midx,
>       CPUTLBDesc *d = &cpu->neg.tlb.d[midx];
>       CPUTLBDescFast *f = &cpu->neg.tlb.f[midx];
>       vaddr mask = MAKE_64BIT_MASK(0, bits);
> +    CPUTLBEntryTree *node;
> +    vaddr addr_mask, last_mask, last_imask;
>   
>       /*
>        * Check if we need to flush due to large pages.
> @@ -759,6 +768,22 @@ static void tlb_flush_range_locked(CPUState *cpu, int midx,
>           vaddr page = addr + i;
>           tlb_flush_vtlb_page_mask_locked(cpu, midx, page, mask);
>       }
> +
> +    addr_mask = addr & mask;
> +    last_mask = addr_mask + len - 1;
> +    last_imask = last_mask | ~mask;
> +    node = tlbtree_lookup_range(d, addr_mask, last_imask);
> +    while (node) {
> +        CPUTLBEntryTree *next =
> +            tlbtree_lookup_range_next(node, addr_mask, last_imask);
> +        vaddr page_mask = node->itree.start & mask;
> +
> +        if (page_mask >= addr_mask && page_mask < last_mask) {
> +            interval_tree_remove(&node->itree, &d->iroot);
> +            g_free(node);
> +        }
> +        node = next;
> +    }
>   }
>   
>   typedef struct {

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 13/54] accel/tcg: Process IntervalTree entries in tlb_reset_dirty

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> Update the addr_write copy within each interval tree node.
> Tidy the iteration within the other two loops as well.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 19 +++++++++++--------
>   1 file changed, 11 insertions(+), 8 deletions(-)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index e2c855f147..0c9f834cbe 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -1010,17 +1010,20 @@ void tlb_reset_dirty(CPUState *cpu, ram_addr_t start1, ram_addr_t length)
>   
>       qemu_spin_lock(&cpu->neg.tlb.c.lock);
>       for (mmu_idx = 0; mmu_idx < NB_MMU_MODES; mmu_idx++) {
> -        unsigned int i;
> -        unsigned int n = tlb_n_entries(&cpu->neg.tlb.f[mmu_idx]);
> +        CPUTLBDesc *desc = &cpu->neg.tlb.d[mmu_idx];
> +        CPUTLBDescFast *fast = &cpu->neg.tlb.f[mmu_idx];
>   
> -        for (i = 0; i < n; i++) {
> -            tlb_reset_dirty_range_locked(&cpu->neg.tlb.f[mmu_idx].table[i],
> -                                         start1, length);
> +        for (size_t i = 0, n = tlb_n_entries(fast); i < n; i++) {
> +            tlb_reset_dirty_range_locked(&fast->table[i], start1, length);
>           }
>   
> -        for (i = 0; i < CPU_VTLB_SIZE; i++) {
> -            tlb_reset_dirty_range_locked(&cpu->neg.tlb.d[mmu_idx].vtable[i],
> -                                         start1, length);
> +        for (size_t i = 0; i < CPU_VTLB_SIZE; i++) {
> +            tlb_reset_dirty_range_locked(&desc->vtable[i], start1, length);
> +        }
> +
> +        for (CPUTLBEntryTree *t = tlbtree_lookup_range(desc, 0, -1); t;
> +             t = tlbtree_lookup_range_next(t, 0, -1)) {
> +            tlb_reset_dirty_range_locked(&t->copy, start1, length);
>           }
>       }
>       qemu_spin_unlock(&cpu->neg.tlb.c.lock);

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 14/54] accel/tcg: Process IntervalTree entries in tlb_set_dirty

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> Update the addr_write copy within an interval tree node.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 17 +++++++++++------
>   1 file changed, 11 insertions(+), 6 deletions(-)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index 0c9f834cbe..eb85e96ee2 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -1049,13 +1049,18 @@ static void tlb_set_dirty(CPUState *cpu, vaddr addr)
>       addr &= TARGET_PAGE_MASK;
>       qemu_spin_lock(&cpu->neg.tlb.c.lock);
>       for (mmu_idx = 0; mmu_idx < NB_MMU_MODES; mmu_idx++) {
> -        tlb_set_dirty1_locked(tlb_entry(cpu, mmu_idx, addr), addr);
> -    }
> +        CPUTLBDesc *desc = &cpu->neg.tlb.d[mmu_idx];
> +        CPUTLBEntryTree *node;
>   
> -    for (mmu_idx = 0; mmu_idx < NB_MMU_MODES; mmu_idx++) {
> -        int k;
> -        for (k = 0; k < CPU_VTLB_SIZE; k++) {
> -            tlb_set_dirty1_locked(&cpu->neg.tlb.d[mmu_idx].vtable[k], addr);
> +        tlb_set_dirty1_locked(tlb_entry(cpu, mmu_idx, addr), addr);
> +
> +        for (int k = 0; k < CPU_VTLB_SIZE; k++) {
> +            tlb_set_dirty1_locked(&desc->vtable[k], addr);
> +        }
> +
> +        node = tlbtree_lookup_addr(desc, addr);
> +        if (node) {
> +            tlb_set_dirty1_locked(&node->copy, addr);
>           }
>       }
>       qemu_spin_unlock(&cpu->neg.tlb.c.lock);

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 15/54] accel/tcg: Use tlb_hit_page in victim_tlb_hit

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> This is clearer than directly comparing the
> page address and the comparator.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 3 +--
>   1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index eb85e96ee2..7ecd327297 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -1394,9 +1394,8 @@ static bool victim_tlb_hit(CPUState *cpu, size_t mmu_idx, size_t index,
>       assert_cpu_is_self(cpu);
>       for (vidx = 0; vidx < CPU_VTLB_SIZE; ++vidx) {
>           CPUTLBEntry *vtlb = &cpu->neg.tlb.d[mmu_idx].vtable[vidx];
> -        uint64_t cmp = tlb_read_idx(vtlb, access_type);
>   
> -        if (cmp == page) {
> +        if (tlb_hit_page(tlb_read_idx(vtlb, access_type), page)) {
>               /* Found entry in victim tlb, swap tlb and iotlb.  */
>               CPUTLBEntry tmptlb, *tlb = &cpu->neg.tlb.f[mmu_idx].table[index];
>   

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 16/54] accel/tcg: Pass full addr to victim_tlb_hit

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> Do not mask the address to the page in these calls.
> It is easy enough to use a different helper instead.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 15 ++++++---------
>   1 file changed, 6 insertions(+), 9 deletions(-)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index 7ecd327297..3aab72ea82 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -1387,7 +1387,7 @@ static void io_failed(CPUState *cpu, CPUTLBEntryFull *full, vaddr addr,
>   /* Return true if ADDR is present in the victim tlb, and has been copied
>      back to the main tlb.  */
>   static bool victim_tlb_hit(CPUState *cpu, size_t mmu_idx, size_t index,
> -                           MMUAccessType access_type, vaddr page)
> +                           MMUAccessType access_type, vaddr addr)
>   {
>       size_t vidx;
>   
> @@ -1395,7 +1395,7 @@ static bool victim_tlb_hit(CPUState *cpu, size_t mmu_idx, size_t index,
>       for (vidx = 0; vidx < CPU_VTLB_SIZE; ++vidx) {
>           CPUTLBEntry *vtlb = &cpu->neg.tlb.d[mmu_idx].vtable[vidx];
>   
> -        if (tlb_hit_page(tlb_read_idx(vtlb, access_type), page)) {
> +        if (tlb_hit(tlb_read_idx(vtlb, access_type), addr)) {
>               /* Found entry in victim tlb, swap tlb and iotlb.  */
>               CPUTLBEntry tmptlb, *tlb = &cpu->neg.tlb.f[mmu_idx].table[index];
>   
> @@ -1448,13 +1448,12 @@ static int probe_access_internal(CPUState *cpu, vaddr addr,
>       uintptr_t index = tlb_index(cpu, mmu_idx, addr);
>       CPUTLBEntry *entry = tlb_entry(cpu, mmu_idx, addr);
>       uint64_t tlb_addr = tlb_read_idx(entry, access_type);
> -    vaddr page_addr = addr & TARGET_PAGE_MASK;
>       int flags = TLB_FLAGS_MASK & ~TLB_FORCE_SLOW;
>       bool force_mmio = check_mem_cbs && cpu_plugin_mem_cbs_enabled(cpu);
>       CPUTLBEntryFull *full;
>   
> -    if (!tlb_hit_page(tlb_addr, page_addr)) {
> -        if (!victim_tlb_hit(cpu, mmu_idx, index, access_type, page_addr)) {
> +    if (!tlb_hit(tlb_addr, addr)) {
> +        if (!victim_tlb_hit(cpu, mmu_idx, index, access_type, addr)) {
>               if (!tlb_fill_align(cpu, addr, access_type, mmu_idx,
>                                   0, fault_size, nonfault, retaddr)) {
>                   /* Non-faulting page table read failed.  */
> @@ -1734,8 +1733,7 @@ static bool mmu_lookup1(CPUState *cpu, MMULookupPageData *data, MemOp memop,
>   
>       /* If the TLB entry is for a different page, reload and try again.  */
>       if (!tlb_hit(tlb_addr, addr)) {
> -        if (!victim_tlb_hit(cpu, mmu_idx, index, access_type,
> -                            addr & TARGET_PAGE_MASK)) {
> +        if (!victim_tlb_hit(cpu, mmu_idx, index, access_type, addr)) {
>               tlb_fill_align(cpu, addr, access_type, mmu_idx,
>                              memop, data->size, false, ra);
>               maybe_resized = true;
> @@ -1914,8 +1912,7 @@ static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
>       /* Check TLB entry and enforce page permissions.  */
>       flags = TLB_FLAGS_MASK;
>       if (!tlb_hit(tlb_addr_write(tlbe), addr)) {
> -        if (!victim_tlb_hit(cpu, mmu_idx, index, MMU_DATA_STORE,
> -                            addr & TARGET_PAGE_MASK)) {
> +        if (!victim_tlb_hit(cpu, mmu_idx, index, MMU_DATA_STORE, addr)) {
>               tlb_fill_align(cpu, addr, MMU_DATA_STORE, mmu_idx,
>                              mop, size, false, retaddr);
>               did_tlb_fill = true;

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 17/54] accel/tcg: Replace victim_tlb_hit with tlbtree_hit

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> Change from a linear search on the victim tlb
> to a balanced binary tree search on the interval tree.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 59 ++++++++++++++++++++++++----------------------
>   1 file changed, 31 insertions(+), 28 deletions(-)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index 3aab72ea82..ea4b78866b 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -1384,35 +1384,38 @@ static void io_failed(CPUState *cpu, CPUTLBEntryFull *full, vaddr addr,
>       }
>   }
>   
> -/* Return true if ADDR is present in the victim tlb, and has been copied
> -   back to the main tlb.  */
> -static bool victim_tlb_hit(CPUState *cpu, size_t mmu_idx, size_t index,
> -                           MMUAccessType access_type, vaddr addr)
> +/*
> + * Return true if ADDR is present in the interval tree,
> + * and has been copied back to the main tlb.
> + */
> +static bool tlbtree_hit(CPUState *cpu, int mmu_idx,
> +                        MMUAccessType access_type, vaddr addr)
>   {
> -    size_t vidx;
> +    CPUTLBDesc *desc = &cpu->neg.tlb.d[mmu_idx];
> +    CPUTLBDescFast *fast = &cpu->neg.tlb.f[mmu_idx];
> +    CPUTLBEntryTree *node;
> +    size_t index;
>   
>       assert_cpu_is_self(cpu);
> -    for (vidx = 0; vidx < CPU_VTLB_SIZE; ++vidx) {
> -        CPUTLBEntry *vtlb = &cpu->neg.tlb.d[mmu_idx].vtable[vidx];
> -
> -        if (tlb_hit(tlb_read_idx(vtlb, access_type), addr)) {
> -            /* Found entry in victim tlb, swap tlb and iotlb.  */
> -            CPUTLBEntry tmptlb, *tlb = &cpu->neg.tlb.f[mmu_idx].table[index];
> -
> -            qemu_spin_lock(&cpu->neg.tlb.c.lock);
> -            copy_tlb_helper_locked(&tmptlb, tlb);
> -            copy_tlb_helper_locked(tlb, vtlb);
> -            copy_tlb_helper_locked(vtlb, &tmptlb);
> -            qemu_spin_unlock(&cpu->neg.tlb.c.lock);
> -
> -            CPUTLBEntryFull *f1 = &cpu->neg.tlb.d[mmu_idx].fulltlb[index];
> -            CPUTLBEntryFull *f2 = &cpu->neg.tlb.d[mmu_idx].vfulltlb[vidx];
> -            CPUTLBEntryFull tmpf;
> -            tmpf = *f1; *f1 = *f2; *f2 = tmpf;
> -            return true;
> -        }
> +    node = tlbtree_lookup_addr(desc, addr);
> +    if (!node) {
> +        /* There is no cached mapping for this page. */
> +        return false;
>       }
> -    return false;
> +
> +    if (!tlb_hit(tlb_read_idx(&node->copy, access_type), addr)) {
> +        /* This access is not permitted. */
> +        return false;
> +    }
> +
> +    /* Install the cached entry. */
> +    index = tlbfast_index(fast, addr);
> +    qemu_spin_lock(&cpu->neg.tlb.c.lock);
> +    copy_tlb_helper_locked(&fast->table[index], &node->copy);
> +    qemu_spin_unlock(&cpu->neg.tlb.c.lock);
> +
> +    desc->fulltlb[index] = node->full;
> +    return true;
>   }
>   
>   static void notdirty_write(CPUState *cpu, vaddr mem_vaddr, unsigned size,
> @@ -1453,7 +1456,7 @@ static int probe_access_internal(CPUState *cpu, vaddr addr,
>       CPUTLBEntryFull *full;
>   
>       if (!tlb_hit(tlb_addr, addr)) {
> -        if (!victim_tlb_hit(cpu, mmu_idx, index, access_type, addr)) {
> +        if (!tlbtree_hit(cpu, mmu_idx, access_type, addr)) {
>               if (!tlb_fill_align(cpu, addr, access_type, mmu_idx,
>                                   0, fault_size, nonfault, retaddr)) {
>                   /* Non-faulting page table read failed.  */
> @@ -1733,7 +1736,7 @@ static bool mmu_lookup1(CPUState *cpu, MMULookupPageData *data, MemOp memop,
>   
>       /* If the TLB entry is for a different page, reload and try again.  */
>       if (!tlb_hit(tlb_addr, addr)) {
> -        if (!victim_tlb_hit(cpu, mmu_idx, index, access_type, addr)) {
> +        if (!tlbtree_hit(cpu, mmu_idx, access_type, addr)) {
>               tlb_fill_align(cpu, addr, access_type, mmu_idx,
>                              memop, data->size, false, ra);
>               maybe_resized = true;
> @@ -1912,7 +1915,7 @@ static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
>       /* Check TLB entry and enforce page permissions.  */
>       flags = TLB_FLAGS_MASK;
>       if (!tlb_hit(tlb_addr_write(tlbe), addr)) {
> -        if (!victim_tlb_hit(cpu, mmu_idx, index, MMU_DATA_STORE, addr)) {
> +        if (!tlbtree_hit(cpu, mmu_idx, MMU_DATA_STORE, addr)) {
>               tlb_fill_align(cpu, addr, MMU_DATA_STORE, mmu_idx,
>                              mop, size, false, retaddr);
>               did_tlb_fill = true;

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 18/54] accel/tcg: Remove the victim tlb

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> This has been functionally replaced by the IntervalTree.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   include/hw/core/cpu.h |  8 -----
>   accel/tcg/cputlb.c    | 74 -------------------------------------------
>   2 files changed, 82 deletions(-)
> 
> diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
> index 1ebc999a73..8eda0574b2 100644
> --- a/include/hw/core/cpu.h
> +++ b/include/hw/core/cpu.h
> @@ -201,9 +201,6 @@ struct CPUClass {
>    */
>   #define NB_MMU_MODES 16
>   
> -/* Use a fully associative victim tlb of 8 entries. */
> -#define CPU_VTLB_SIZE 8
> -
>   /*
>    * The full TLB entry, which is not accessed by generated TCG code,
>    * so the layout is not as critical as that of CPUTLBEntry. This is
> @@ -285,11 +282,6 @@ typedef struct CPUTLBDesc {
>       /* maximum number of entries observed in the window */
>       size_t window_max_entries;
>       size_t n_used_entries;
> -    /* The next index to use in the tlb victim table.  */
> -    size_t vindex;
> -    /* The tlb victim table, in two parts.  */
> -    CPUTLBEntry vtable[CPU_VTLB_SIZE];
> -    CPUTLBEntryFull vfulltlb[CPU_VTLB_SIZE];
>       CPUTLBEntryFull *fulltlb;
>       /* All active tlb entries for this address space. */
>       IntervalTreeRoot iroot;
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index ea4b78866b..8caa8c0f1d 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -328,8 +328,6 @@ static void tlb_mmu_flush_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast)
>       tlbfast_flush_locked(desc, fast);
>       desc->large_page_addr = -1;
>       desc->large_page_mask = -1;
> -    desc->vindex = 0;
> -    memset(desc->vtable, -1, sizeof(desc->vtable));
>       interval_tree_free_nodes(&desc->iroot, offsetof(CPUTLBEntryTree, itree));
>   }
>   
> @@ -361,11 +359,6 @@ static inline void tlb_n_used_entries_inc(CPUState *cpu, uintptr_t mmu_idx)
>       cpu->neg.tlb.d[mmu_idx].n_used_entries++;
>   }
>   
> -static inline void tlb_n_used_entries_dec(CPUState *cpu, uintptr_t mmu_idx)
> -{
> -    cpu->neg.tlb.d[mmu_idx].n_used_entries--;
> -}
> -
>   void tlb_init(CPUState *cpu)
>   {
>       int64_t now = get_clock_realtime();
> @@ -496,20 +489,6 @@ static bool tlb_hit_page_mask_anyprot(CPUTLBEntry *tlb_entry,
>               page == (tlb_entry->addr_code & mask));
>   }
>   
> -static inline bool tlb_hit_page_anyprot(CPUTLBEntry *tlb_entry, vaddr page)
> -{
> -    return tlb_hit_page_mask_anyprot(tlb_entry, page, -1);
> -}
> -
> -/**
> - * tlb_entry_is_empty - return true if the entry is not in use
> - * @te: pointer to CPUTLBEntry
> - */
> -static inline bool tlb_entry_is_empty(const CPUTLBEntry *te)
> -{
> -    return te->addr_read == -1 && te->addr_write == -1 && te->addr_code == -1;
> -}
> -
>   /* Called with tlb_c.lock held */
>   static bool tlb_flush_entry_mask_locked(CPUTLBEntry *tlb_entry,
>                                           vaddr page,
> @@ -522,28 +501,6 @@ static bool tlb_flush_entry_mask_locked(CPUTLBEntry *tlb_entry,
>       return false;
>   }
>   
> -/* Called with tlb_c.lock held */
> -static void tlb_flush_vtlb_page_mask_locked(CPUState *cpu, int mmu_idx,
> -                                            vaddr page,
> -                                            vaddr mask)
> -{
> -    CPUTLBDesc *d = &cpu->neg.tlb.d[mmu_idx];
> -    int k;
> -
> -    assert_cpu_is_self(cpu);
> -    for (k = 0; k < CPU_VTLB_SIZE; k++) {
> -        if (tlb_flush_entry_mask_locked(&d->vtable[k], page, mask)) {
> -            tlb_n_used_entries_dec(cpu, mmu_idx);
> -        }
> -    }
> -}
> -
> -static inline void tlb_flush_vtlb_page_locked(CPUState *cpu, int mmu_idx,
> -                                              vaddr page)
> -{
> -    tlb_flush_vtlb_page_mask_locked(cpu, mmu_idx, page, -1);
> -}
> -
>   static void tlbfast_flush_range_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast,
>                                          vaddr addr, vaddr len, vaddr mask)
>   {
> @@ -588,7 +545,6 @@ static void tlb_flush_page_locked(CPUState *cpu, int midx, vaddr page)
>   
>       tlbfast_flush_range_locked(desc, &cpu->neg.tlb.f[midx],
>                                  page, TARGET_PAGE_SIZE, -1);
> -    tlb_flush_vtlb_page_locked(cpu, midx, page);
>   
>       node = tlbtree_lookup_addr(desc, page);
>       if (node) {
> @@ -764,11 +720,6 @@ static void tlb_flush_range_locked(CPUState *cpu, int midx,
>   
>       tlbfast_flush_range_locked(d, f, addr, len, mask);
>   
> -    for (vaddr i = 0; i < len; i += TARGET_PAGE_SIZE) {
> -        vaddr page = addr + i;
> -        tlb_flush_vtlb_page_mask_locked(cpu, midx, page, mask);
> -    }
> -
>       addr_mask = addr & mask;
>       last_mask = addr_mask + len - 1;
>       last_imask = last_mask | ~mask;
> @@ -1017,10 +968,6 @@ void tlb_reset_dirty(CPUState *cpu, ram_addr_t start1, ram_addr_t length)
>               tlb_reset_dirty_range_locked(&fast->table[i], start1, length);
>           }
>   
> -        for (size_t i = 0; i < CPU_VTLB_SIZE; i++) {
> -            tlb_reset_dirty_range_locked(&desc->vtable[i], start1, length);
> -        }
> -
>           for (CPUTLBEntryTree *t = tlbtree_lookup_range(desc, 0, -1); t;
>                t = tlbtree_lookup_range_next(t, 0, -1)) {
>               tlb_reset_dirty_range_locked(&t->copy, start1, length);
> @@ -1054,10 +1001,6 @@ static void tlb_set_dirty(CPUState *cpu, vaddr addr)
>   
>           tlb_set_dirty1_locked(tlb_entry(cpu, mmu_idx, addr), addr);
>   
> -        for (int k = 0; k < CPU_VTLB_SIZE; k++) {
> -            tlb_set_dirty1_locked(&desc->vtable[k], addr);
> -        }
> -
>           node = tlbtree_lookup_addr(desc, addr);
>           if (node) {
>               tlb_set_dirty1_locked(&node->copy, addr);
> @@ -1216,23 +1159,6 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
>       /* Note that the tlb is no longer clean.  */
>       tlb->c.dirty |= 1 << mmu_idx;
>   
> -    /* Make sure there's no cached translation for the new page.  */
> -    tlb_flush_vtlb_page_locked(cpu, mmu_idx, addr_page);
> -
> -    /*
> -     * Only evict the old entry to the victim tlb if it's for a
> -     * different page; otherwise just overwrite the stale data.
> -     */
> -    if (!tlb_hit_page_anyprot(te, addr_page) && !tlb_entry_is_empty(te)) {
> -        unsigned vidx = desc->vindex++ % CPU_VTLB_SIZE;
> -        CPUTLBEntry *tv = &desc->vtable[vidx];
> -
> -        /* Evict the old entry into the victim tlb.  */
> -        copy_tlb_helper_locked(tv, te);
> -        desc->vfulltlb[vidx] = desc->fulltlb[index];
> -        tlb_n_used_entries_dec(cpu, mmu_idx);
> -    }
> -
>       /* Replace an old IntervalTree entry, or create a new one. */
>       node = tlbtree_lookup_addr(desc, addr_page);
>       if (!node) {

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 19/54] accel/tcg: Remove tlb_n_used_entries_inc

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> Expand the function into its only caller, using the
> existing CPUTLBDesc local pointer.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 7 +------
>   1 file changed, 1 insertion(+), 6 deletions(-)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index 8caa8c0f1d..3e24529f4f 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -354,11 +354,6 @@ static void tlb_mmu_init(CPUTLBDesc *desc, CPUTLBDescFast *fast, int64_t now)
>       tlb_mmu_flush_locked(desc, fast);
>   }
>   
> -static inline void tlb_n_used_entries_inc(CPUState *cpu, uintptr_t mmu_idx)
> -{
> -    cpu->neg.tlb.d[mmu_idx].n_used_entries++;
> -}
> -
>   void tlb_init(CPUState *cpu)
>   {
>       int64_t now = get_clock_realtime();
> @@ -1211,7 +1206,7 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
>   
>       node->full = *full;
>       copy_tlb_helper_locked(te, &node->copy);
> -    tlb_n_used_entries_inc(cpu, mmu_idx);
> +    desc->n_used_entries++;
>       qemu_spin_unlock(&tlb->c.lock);
>   }
>   

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 20/54] include/exec/tlb-common: Move CPUTLBEntryFull from hw/core/cpu.h

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> CPUTLBEntryFull structures are no longer directly included within
> the CPUState structure.  Move the structure definition out of cpu.h
> to reduce visibility.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   include/exec/tlb-common.h | 63 +++++++++++++++++++++++++++++++++++++++
>   include/hw/core/cpu.h     | 63 ---------------------------------------
>   2 files changed, 63 insertions(+), 63 deletions(-)
> 
> diff --git a/include/exec/tlb-common.h b/include/exec/tlb-common.h
> index dc5a5faa0b..300f9fae67 100644
> --- a/include/exec/tlb-common.h
> +++ b/include/exec/tlb-common.h
> @@ -53,4 +53,67 @@ typedef struct CPUTLBDescFast {
>       CPUTLBEntry *table;
>   } CPUTLBDescFast QEMU_ALIGNED(2 * sizeof(void *));
>   
> +/*
> + * The full TLB entry, which is not accessed by generated TCG code,
> + * so the layout is not as critical as that of CPUTLBEntry. This is
> + * also why we don't want to combine the two structs.
> + */
> +struct CPUTLBEntryFull {
> +    /*
> +     * @xlat_section contains:
> +     *  - in the lower TARGET_PAGE_BITS, a physical section number
> +     *  - with the lower TARGET_PAGE_BITS masked off, an offset which
> +     *    must be added to the virtual address to obtain:
> +     *     + the ram_addr_t of the target RAM (if the physical section
> +     *       number is PHYS_SECTION_NOTDIRTY or PHYS_SECTION_ROM)
> +     *     + the offset within the target MemoryRegion (otherwise)
> +     */
> +    hwaddr xlat_section;
> +
> +    /*
> +     * @phys_addr contains the physical address in the address space
> +     * given by cpu_asidx_from_attrs(cpu, @attrs).
> +     */
> +    hwaddr phys_addr;
> +
> +    /* @attrs contains the memory transaction attributes for the page. */
> +    MemTxAttrs attrs;
> +
> +    /* @prot contains the complete protections for the page. */
> +    uint8_t prot;
> +
> +    /* @lg_page_size contains the log2 of the page size. */
> +    uint8_t lg_page_size;
> +
> +    /* Additional tlb flags requested by tlb_fill. */
> +    uint8_t tlb_fill_flags;
> +
> +    /*
> +     * Additional tlb flags for use by the slow path. If non-zero,
> +     * the corresponding CPUTLBEntry comparator must have TLB_FORCE_SLOW.
> +     */
> +    uint8_t slow_flags[MMU_ACCESS_COUNT];
> +
> +    /*
> +     * Allow target-specific additions to this structure.
> +     * This may be used to cache items from the guest cpu
> +     * page tables for later use by the implementation.
> +     */
> +    union {
> +        /*
> +         * Cache the attrs and shareability fields from the page table entry.
> +         *
> +         * For ARMMMUIdx_Stage2*, pte_attrs is the S2 descriptor bits [5:2].
> +         * Otherwise, pte_attrs is the same as the MAIR_EL1 8-bit format.
> +         * For shareability and guarded, as in the SH and GP fields respectively
> +         * of the VMSAv8-64 PTEs.
> +         */
> +        struct {
> +            uint8_t pte_attrs;
> +            uint8_t shareability;
> +            bool guarded;
> +        } arm;
> +    } extra;
> +};
> +
>   #endif /* EXEC_TLB_COMMON_H */
> diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
> index 8eda0574b2..4364ddb1db 100644
> --- a/include/hw/core/cpu.h
> +++ b/include/hw/core/cpu.h
> @@ -201,69 +201,6 @@ struct CPUClass {
>    */
>   #define NB_MMU_MODES 16
>   
> -/*
> - * The full TLB entry, which is not accessed by generated TCG code,
> - * so the layout is not as critical as that of CPUTLBEntry. This is
> - * also why we don't want to combine the two structs.
> - */
> -struct CPUTLBEntryFull {
> -    /*
> -     * @xlat_section contains:
> -     *  - in the lower TARGET_PAGE_BITS, a physical section number
> -     *  - with the lower TARGET_PAGE_BITS masked off, an offset which
> -     *    must be added to the virtual address to obtain:
> -     *     + the ram_addr_t of the target RAM (if the physical section
> -     *       number is PHYS_SECTION_NOTDIRTY or PHYS_SECTION_ROM)
> -     *     + the offset within the target MemoryRegion (otherwise)
> -     */
> -    hwaddr xlat_section;
> -
> -    /*
> -     * @phys_addr contains the physical address in the address space
> -     * given by cpu_asidx_from_attrs(cpu, @attrs).
> -     */
> -    hwaddr phys_addr;
> -
> -    /* @attrs contains the memory transaction attributes for the page. */
> -    MemTxAttrs attrs;
> -
> -    /* @prot contains the complete protections for the page. */
> -    uint8_t prot;
> -
> -    /* @lg_page_size contains the log2 of the page size. */
> -    uint8_t lg_page_size;
> -
> -    /* Additional tlb flags requested by tlb_fill. */
> -    uint8_t tlb_fill_flags;
> -
> -    /*
> -     * Additional tlb flags for use by the slow path. If non-zero,
> -     * the corresponding CPUTLBEntry comparator must have TLB_FORCE_SLOW.
> -     */
> -    uint8_t slow_flags[MMU_ACCESS_COUNT];
> -
> -    /*
> -     * Allow target-specific additions to this structure.
> -     * This may be used to cache items from the guest cpu
> -     * page tables for later use by the implementation.
> -     */
> -    union {
> -        /*
> -         * Cache the attrs and shareability fields from the page table entry.
> -         *
> -         * For ARMMMUIdx_Stage2*, pte_attrs is the S2 descriptor bits [5:2].
> -         * Otherwise, pte_attrs is the same as the MAIR_EL1 8-bit format.
> -         * For shareability and guarded, as in the SH and GP fields respectively
> -         * of the VMSAv8-64 PTEs.
> -         */
> -        struct {
> -            uint8_t pte_attrs;
> -            uint8_t shareability;
> -            bool guarded;
> -        } arm;
> -    } extra;
> -};
> -
>   /*
>    * Data elements that are per MMU mode, minus the bits accessed by
>    * the TCG fast path.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 21/54] accel/tcg: Delay plugin adjustment in probe_access_internal

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> Remove force_mmio and place the expression into the IF
> expression, behind the short-circuit logic expressions
> that might eliminate its computation.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 12 ++++++++----
>   1 file changed, 8 insertions(+), 4 deletions(-)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index 3e24529f4f..a4c69bcbf1 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -1373,7 +1373,6 @@ static int probe_access_internal(CPUState *cpu, vaddr addr,
>       CPUTLBEntry *entry = tlb_entry(cpu, mmu_idx, addr);
>       uint64_t tlb_addr = tlb_read_idx(entry, access_type);
>       int flags = TLB_FLAGS_MASK & ~TLB_FORCE_SLOW;
> -    bool force_mmio = check_mem_cbs && cpu_plugin_mem_cbs_enabled(cpu);
>       CPUTLBEntryFull *full;
>   
>       if (!tlb_hit(tlb_addr, addr)) {
> @@ -1404,9 +1403,14 @@ static int probe_access_internal(CPUState *cpu, vaddr addr,
>       *pfull = full = &cpu->neg.tlb.d[mmu_idx].fulltlb[index];
>       flags |= full->slow_flags[access_type];
>   
> -    /* Fold all "mmio-like" bits into TLB_MMIO.  This is not RAM.  */
> -    if (unlikely(flags & ~(TLB_WATCHPOINT | TLB_NOTDIRTY | TLB_CHECK_ALIGNED))
> -        || (access_type != MMU_INST_FETCH && force_mmio)) {
> +    /*
> +     * Fold all "mmio-like" bits, and required plugin callbacks, to TLB_MMIO.
> +     * These cannot be treated as RAM.
> +     */
> +    if ((flags & ~(TLB_WATCHPOINT | TLB_NOTDIRTY | TLB_CHECK_ALIGNED))
> +        || (access_type != MMU_INST_FETCH
> +            && check_mem_cbs
> +            && cpu_plugin_mem_cbs_enabled(cpu))) {
>           *phost = NULL;
>           return TLB_MMIO;
>       }

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 22/54] accel/tcg: Call cpu_ld*_code_mmu from cpu_ld*_code

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> Ensure a common entry point for all code lookups.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 8 ++++----
>   1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index a4c69bcbf1..c975dd2322 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -2924,28 +2924,28 @@ uint32_t cpu_ldub_code(CPUArchState *env, abi_ptr addr)
>   {
>       CPUState *cs = env_cpu(env);
>       MemOpIdx oi = make_memop_idx(MO_UB, cpu_mmu_index(cs, true));
> -    return do_ld1_mmu(cs, addr, oi, 0, MMU_INST_FETCH);
> +    return cpu_ldb_code_mmu(env, addr, oi, 0);
>   }
>   
>   uint32_t cpu_lduw_code(CPUArchState *env, abi_ptr addr)
>   {
>       CPUState *cs = env_cpu(env);
>       MemOpIdx oi = make_memop_idx(MO_TEUW, cpu_mmu_index(cs, true));
> -    return do_ld2_mmu(cs, addr, oi, 0, MMU_INST_FETCH);
> +    return cpu_ldw_code_mmu(env, addr, oi, 0);
>   }
>   
>   uint32_t cpu_ldl_code(CPUArchState *env, abi_ptr addr)
>   {
>       CPUState *cs = env_cpu(env);
>       MemOpIdx oi = make_memop_idx(MO_TEUL, cpu_mmu_index(cs, true));
> -    return do_ld4_mmu(cs, addr, oi, 0, MMU_INST_FETCH);
> +    return cpu_ldl_code_mmu(env, addr, oi, 0);
>   }
>   
>   uint64_t cpu_ldq_code(CPUArchState *env, abi_ptr addr)
>   {
>       CPUState *cs = env_cpu(env);
>       MemOpIdx oi = make_memop_idx(MO_TEUQ, cpu_mmu_index(cs, true));
> -    return do_ld8_mmu(cs, addr, oi, 0, MMU_INST_FETCH);
> +    return cpu_ldq_code_mmu(env, addr, oi, 0);
>   }
>   
>   uint8_t cpu_ldb_code_mmu(CPUArchState *env, abi_ptr addr,

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 23/54] accel/tcg: Check original prot bits for read in atomic_mmu_lookup

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> In the mist before CPUTLBEntryFull existed, we had to be
> clever to detect write-only pages.  Now we can directly
> test the saved prot bits, which is clearer.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 6 ++----
>   1 file changed, 2 insertions(+), 4 deletions(-)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index c975dd2322..ae3a99eb47 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -1854,14 +1854,13 @@ static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
>               flags &= ~TLB_INVALID_MASK;
>           }
>       }
> +    full = &cpu->neg.tlb.d[mmu_idx].fulltlb[index];
>   
>       /*
>        * Let the guest notice RMW on a write-only page.
>        * We have just verified that the page is writable.
> -     * Subpage lookups may have left TLB_INVALID_MASK set,
> -     * but addr_read will only be -1 if PAGE_READ was unset.
>        */
> -    if (unlikely(tlbe->addr_read == -1)) {
> +    if (unlikely(!(full->prot & PAGE_READ))) {
>           tlb_fill_align(cpu, addr, MMU_DATA_LOAD, mmu_idx,
>                          0, size, false, retaddr);
>           /*
> @@ -1899,7 +1898,6 @@ static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
>       }
>   
>       hostaddr = (void *)((uintptr_t)addr + tlbe->addend);
> -    full = &cpu->neg.tlb.d[mmu_idx].fulltlb[index];
>   
>       if (unlikely(flags & TLB_NOTDIRTY)) {
>           notdirty_write(cpu, addr, size, full, retaddr);

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 24/54] accel/tcg: Preserve tlb flags in tlb_set_compare

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Before, if !enable, we squashed the entire address comparator to -1.
> This works because TLB_INVALID_MASK is set.  It seemed natural, because
> the tlb is cleared with memset of 0xff.
> 
> With this patch, we retain all of the other TLB_* bits even when
> the page is not enabled.  This works because TLB_INVALID_MASK is set.
> This will be used in a subsequent patch; the addr_read comparator
> contains the flags for pages that are executable but not readable.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 16 +++++++---------
>   1 file changed, 7 insertions(+), 9 deletions(-)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index ae3a99eb47..585f4171cc 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -1032,15 +1032,13 @@ static inline void tlb_set_compare(CPUTLBEntryFull *full, CPUTLBEntry *ent,
>                                      vaddr address, int flags,
>                                      MMUAccessType access_type, bool enable)
>   {
> -    if (enable) {
> -        address |= flags & TLB_FLAGS_MASK;
> -        flags &= TLB_SLOW_FLAGS_MASK;
> -        if (flags) {
> -            address |= TLB_FORCE_SLOW;
> -        }
> -    } else {
> -        address = -1;
> -        flags = 0;
> +    if (!enable) {
> +        address = TLB_INVALID_MASK;
> +    }
> +    address |= flags & TLB_FLAGS_MASK;
> +    flags &= TLB_SLOW_FLAGS_MASK;
> +    if (flags) {
> +        address |= TLB_FORCE_SLOW;
>       }
>       ent->addr_idx[access_type] = address;
>       full->slow_flags[access_type] = flags;

Good that you extracted that from original patch, it's much more clear now.
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 25/54] accel/tcg: Return CPUTLBEntryFull not pointer in probe_access_full_mmu

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Return a copy of the structure, not a pointer.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   include/exec/exec-all.h              |  2 +-
>   accel/tcg/cputlb.c                   | 13 ++++++++-----
>   target/arm/ptw.c                     | 10 +++++-----
>   target/i386/tcg/sysemu/excp_helper.c |  8 ++++----
>   4 files changed, 18 insertions(+), 15 deletions(-)
> 
> diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
> index 2e4c4cc4b4..df7d0b5ad0 100644
> --- a/include/exec/exec-all.h
> +++ b/include/exec/exec-all.h
> @@ -393,7 +393,7 @@ int probe_access_full(CPUArchState *env, vaddr addr, int size,
>    */
>   int probe_access_full_mmu(CPUArchState *env, vaddr addr, int size,
>                             MMUAccessType access_type, int mmu_idx,
> -                          void **phost, CPUTLBEntryFull **pfull);
> +                          void **phost, CPUTLBEntryFull *pfull);
>   
>   #endif /* !CONFIG_USER_ONLY */
>   #endif /* CONFIG_TCG */
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index 585f4171cc..81135524eb 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -1439,25 +1439,28 @@ int probe_access_full(CPUArchState *env, vaddr addr, int size,
>   
>   int probe_access_full_mmu(CPUArchState *env, vaddr addr, int size,
>                             MMUAccessType access_type, int mmu_idx,
> -                          void **phost, CPUTLBEntryFull **pfull)
> +                          void **phost, CPUTLBEntryFull *pfull)
>   {
>       void *discard_phost;
> -    CPUTLBEntryFull *discard_tlb;
> +    CPUTLBEntryFull *full;
>   
>       /* privately handle users that don't need full results */
>       phost = phost ? phost : &discard_phost;
> -    pfull = pfull ? pfull : &discard_tlb;
>   
>       int flags = probe_access_internal(env_cpu(env), addr, size, access_type,
> -                                      mmu_idx, true, phost, pfull, 0, false);
> +                                      mmu_idx, true, phost, &full, 0, false);
>   
>       /* Handle clean RAM pages.  */
>       if (unlikely(flags & TLB_NOTDIRTY)) {
>           int dirtysize = size == 0 ? 1 : size;
> -        notdirty_write(env_cpu(env), addr, dirtysize, *pfull, 0);
> +        notdirty_write(env_cpu(env), addr, dirtysize, full, 0);
>           flags &= ~TLB_NOTDIRTY;
>       }
>   
> +    if (pfull) {
> +        *pfull = *full;
> +    }
> +
>       return flags;
>   }
>   
> diff --git a/target/arm/ptw.c b/target/arm/ptw.c
> index 9849949508..3ae5f524de 100644
> --- a/target/arm/ptw.c
> +++ b/target/arm/ptw.c
> @@ -592,7 +592,7 @@ static bool S1_ptw_translate(CPUARMState *env, S1Translate *ptw,
>           ptw->out_space = s2.f.attrs.space;
>       } else {
>   #ifdef CONFIG_TCG
> -        CPUTLBEntryFull *full;
> +        CPUTLBEntryFull full;
>           int flags;
>   
>           env->tlb_fi = fi;
> @@ -604,10 +604,10 @@ static bool S1_ptw_translate(CPUARMState *env, S1Translate *ptw,
>           if (unlikely(flags & TLB_INVALID_MASK)) {
>               goto fail;
>           }
> -        ptw->out_phys = full->phys_addr | (addr & ~TARGET_PAGE_MASK);
> -        ptw->out_rw = full->prot & PAGE_WRITE;
> -        pte_attrs = full->extra.arm.pte_attrs;
> -        ptw->out_space = full->attrs.space;
> +        ptw->out_phys = full.phys_addr | (addr & ~TARGET_PAGE_MASK);
> +        ptw->out_rw = full.prot & PAGE_WRITE;
> +        pte_attrs = full.extra.arm.pte_attrs;
> +        ptw->out_space = full.attrs.space;
>   #else
>           g_assert_not_reached();
>   #endif
> diff --git a/target/i386/tcg/sysemu/excp_helper.c b/target/i386/tcg/sysemu/excp_helper.c
> index 02d3486421..168ff8e5f3 100644
> --- a/target/i386/tcg/sysemu/excp_helper.c
> +++ b/target/i386/tcg/sysemu/excp_helper.c
> @@ -436,7 +436,7 @@ do_check_protect_pse36:
>        * addresses) using the address with the A20 bit set.
>        */
>       if (in->ptw_idx == MMU_NESTED_IDX) {
> -        CPUTLBEntryFull *full;
> +        CPUTLBEntryFull full;
>           int flags, nested_page_size;
>   
>           flags = probe_access_full_mmu(env, paddr, 0, access_type,
> @@ -451,7 +451,7 @@ do_check_protect_pse36:
>           }
>   
>           /* Merge stage1 & stage2 protection bits. */
> -        prot &= full->prot;
> +        prot &= full.prot;
>   
>           /* Re-verify resulting protection. */
>           if ((prot & (1 << access_type)) == 0) {
> @@ -459,8 +459,8 @@ do_check_protect_pse36:
>           }
>   
>           /* Merge stage1 & stage2 addresses to final physical address. */
> -        nested_page_size = 1 << full->lg_page_size;
> -        paddr = (full->phys_addr & ~(nested_page_size - 1))
> +        nested_page_size = 1 << full.lg_page_size;
> +        paddr = (full.phys_addr & ~(nested_page_size - 1))
>                 | (paddr & (nested_page_size - 1));
>   
>           /*

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 26/54] accel/tcg: Return CPUTLBEntryFull not pointer in probe_access_full

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Return a copy of the structure, not a pointer.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   include/exec/exec-all.h     |  6 +-----
>   accel/tcg/cputlb.c          |  8 +++++---
>   target/arm/tcg/helper-a64.c |  4 ++--
>   target/arm/tcg/mte_helper.c | 15 ++++++---------
>   target/arm/tcg/sve_helper.c |  6 +++---
>   5 files changed, 17 insertions(+), 22 deletions(-)
> 
> diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
> index df7d0b5ad0..69bdb77584 100644
> --- a/include/exec/exec-all.h
> +++ b/include/exec/exec-all.h
> @@ -365,10 +365,6 @@ int probe_access_flags(CPUArchState *env, vaddr addr, int size,
>    * probe_access_full:
>    * Like probe_access_flags, except also return into @pfull.
>    *
> - * The CPUTLBEntryFull structure returned via @pfull is transient
> - * and must be consumed or copied immediately, before any further
> - * access or changes to TLB @mmu_idx.
> - *
>    * This function will not fault if @nonfault is set, but will
>    * return TLB_INVALID_MASK if the page is not mapped, or is not
>    * accessible with @access_type.
> @@ -379,7 +375,7 @@ int probe_access_flags(CPUArchState *env, vaddr addr, int size,
>   int probe_access_full(CPUArchState *env, vaddr addr, int size,
>                         MMUAccessType access_type, int mmu_idx,
>                         bool nonfault, void **phost,
> -                      CPUTLBEntryFull **pfull, uintptr_t retaddr);
> +                      CPUTLBEntryFull *pfull, uintptr_t retaddr);
>   
>   /**
>    * probe_access_full_mmu:
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index 81135524eb..84e7e633e3 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -1420,20 +1420,22 @@ static int probe_access_internal(CPUState *cpu, vaddr addr,
>   
>   int probe_access_full(CPUArchState *env, vaddr addr, int size,
>                         MMUAccessType access_type, int mmu_idx,
> -                      bool nonfault, void **phost, CPUTLBEntryFull **pfull,
> +                      bool nonfault, void **phost, CPUTLBEntryFull *pfull,
>                         uintptr_t retaddr)
>   {
> +    CPUTLBEntryFull *full;
>       int flags = probe_access_internal(env_cpu(env), addr, size, access_type,
> -                                      mmu_idx, nonfault, phost, pfull, retaddr,
> +                                      mmu_idx, nonfault, phost, &full, retaddr,
>                                         true);
>   
>       /* Handle clean RAM pages.  */
>       if (unlikely(flags & TLB_NOTDIRTY)) {
>           int dirtysize = size == 0 ? 1 : size;
> -        notdirty_write(env_cpu(env), addr, dirtysize, *pfull, retaddr);
> +        notdirty_write(env_cpu(env), addr, dirtysize, full, retaddr);
>           flags &= ~TLB_NOTDIRTY;
>       }
>   
> +    *pfull = *full;
>       return flags;
>   }
>   
> diff --git a/target/arm/tcg/helper-a64.c b/target/arm/tcg/helper-a64.c
> index 8f42a28d07..783864d6db 100644
> --- a/target/arm/tcg/helper-a64.c
> +++ b/target/arm/tcg/helper-a64.c
> @@ -1883,14 +1883,14 @@ static bool is_guarded_page(CPUARMState *env, target_ulong addr, uintptr_t ra)
>   #ifdef CONFIG_USER_ONLY
>       return page_get_flags(addr) & PAGE_BTI;
>   #else
> -    CPUTLBEntryFull *full;
> +    CPUTLBEntryFull full;
>       void *host;
>       int mmu_idx = cpu_mmu_index(env_cpu(env), true);
>       int flags = probe_access_full(env, addr, 0, MMU_INST_FETCH, mmu_idx,
>                                     false, &host, &full, ra);
>   
>       assert(!(flags & TLB_INVALID_MASK));
> -    return full->extra.arm.guarded;
> +    return full.extra.arm.guarded;
>   #endif
>   }
>   
> diff --git a/target/arm/tcg/mte_helper.c b/target/arm/tcg/mte_helper.c
> index 9d2ba287ee..870b2875af 100644
> --- a/target/arm/tcg/mte_helper.c
> +++ b/target/arm/tcg/mte_helper.c
> @@ -83,8 +83,7 @@ uint8_t *allocation_tag_mem_probe(CPUARMState *env, int ptr_mmu_idx,
>                         TARGET_PAGE_BITS - LOG2_TAG_GRANULE - 1);
>       return tags + index;
>   #else
> -    CPUTLBEntryFull *full;
> -    MemTxAttrs attrs;
> +    CPUTLBEntryFull full;
>       int in_page, flags;
>       hwaddr ptr_paddr, tag_paddr, xlat;
>       MemoryRegion *mr;
> @@ -110,7 +109,7 @@ uint8_t *allocation_tag_mem_probe(CPUARMState *env, int ptr_mmu_idx,
>       assert(!(flags & TLB_INVALID_MASK));
>   
>       /* If the virtual page MemAttr != Tagged, access unchecked. */
> -    if (full->extra.arm.pte_attrs != 0xf0) {
> +    if (full.extra.arm.pte_attrs != 0xf0) {
>           return NULL;
>       }
>   
> @@ -129,9 +128,7 @@ uint8_t *allocation_tag_mem_probe(CPUARMState *env, int ptr_mmu_idx,
>        * Remember these values across the second lookup below,
>        * which may invalidate this pointer via tlb resize.
>        */
> -    ptr_paddr = full->phys_addr | (ptr & ~TARGET_PAGE_MASK);
> -    attrs = full->attrs;
> -    full = NULL;
> +    ptr_paddr = full.phys_addr | (ptr & ~TARGET_PAGE_MASK);
>   
>       /*
>        * The Normal memory access can extend to the next page.  E.g. a single
> @@ -150,17 +147,17 @@ uint8_t *allocation_tag_mem_probe(CPUARMState *env, int ptr_mmu_idx,
>       if (!probe && unlikely(flags & TLB_WATCHPOINT)) {
>           int wp = ptr_access == MMU_DATA_LOAD ? BP_MEM_READ : BP_MEM_WRITE;
>           assert(ra != 0);
> -        cpu_check_watchpoint(env_cpu(env), ptr, ptr_size, attrs, wp, ra);
> +        cpu_check_watchpoint(env_cpu(env), ptr, ptr_size, full.attrs, wp, ra);
>       }
>   
>       /* Convert to the physical address in tag space.  */
>       tag_paddr = ptr_paddr >> (LOG2_TAG_GRANULE + 1);
>   
>       /* Look up the address in tag space. */
> -    tag_asi = attrs.secure ? ARMASIdx_TagS : ARMASIdx_TagNS;
> +    tag_asi = full.attrs.secure ? ARMASIdx_TagS : ARMASIdx_TagNS;
>       tag_as = cpu_get_address_space(env_cpu(env), tag_asi);
>       mr = address_space_translate(tag_as, tag_paddr, &xlat, NULL,
> -                                 tag_access == MMU_DATA_STORE, attrs);
> +                                 tag_access == MMU_DATA_STORE, full.attrs);
>   
>       /*
>        * Note that @mr will never be NULL.  If there is nothing in the address
> diff --git a/target/arm/tcg/sve_helper.c b/target/arm/tcg/sve_helper.c
> index f1ee0e060f..dad0d5e518 100644
> --- a/target/arm/tcg/sve_helper.c
> +++ b/target/arm/tcg/sve_helper.c
> @@ -5357,7 +5357,7 @@ bool sve_probe_page(SVEHostPage *info, bool nofault, CPUARMState *env,
>       flags = probe_access_flags(env, addr, 0, access_type, mmu_idx, nofault,
>                                  &info->host, retaddr);
>   #else
> -    CPUTLBEntryFull *full;
> +    CPUTLBEntryFull full;
>       flags = probe_access_full(env, addr, 0, access_type, mmu_idx, nofault,
>                                 &info->host, &full, retaddr);
>   #endif
> @@ -5373,8 +5373,8 @@ bool sve_probe_page(SVEHostPage *info, bool nofault, CPUARMState *env,
>       /* Require both ANON and MTE; see allocation_tag_mem(). */
>       info->tagged = (flags & PAGE_ANON) && (flags & PAGE_MTE);
>   #else
> -    info->attrs = full->attrs;
> -    info->tagged = full->extra.arm.pte_attrs == 0xf0;
> +    info->attrs = full.attrs;
> +    info->tagged = full.extra.arm.pte_attrs == 0xf0;
>   #endif
>   
>       /* Ensure that info->host[] is relative to addr, not addr + mem_off. */

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 27/54] accel/tcg: Return CPUTLBEntryFull not pointer in probe_access_internal

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Return a copy of the structure, not a pointer.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 40 ++++++++++++++++++----------------------
>   1 file changed, 18 insertions(+), 22 deletions(-)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index 84e7e633e3..41b2f76cc9 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -1364,7 +1364,7 @@ static void notdirty_write(CPUState *cpu, vaddr mem_vaddr, unsigned size,
>   static int probe_access_internal(CPUState *cpu, vaddr addr,
>                                    int fault_size, MMUAccessType access_type,
>                                    int mmu_idx, bool nonfault,
> -                                 void **phost, CPUTLBEntryFull **pfull,
> +                                 void **phost, CPUTLBEntryFull *pfull,
>                                    uintptr_t retaddr, bool check_mem_cbs)
>   {
>       uintptr_t index = tlb_index(cpu, mmu_idx, addr);
> @@ -1379,7 +1379,7 @@ static int probe_access_internal(CPUState *cpu, vaddr addr,
>                                   0, fault_size, nonfault, retaddr)) {
>                   /* Non-faulting page table read failed.  */
>                   *phost = NULL;
> -                *pfull = NULL;
> +                memset(pfull, 0, sizeof(*pfull));
>                   return TLB_INVALID_MASK;
>               }
>   
> @@ -1398,8 +1398,9 @@ static int probe_access_internal(CPUState *cpu, vaddr addr,
>       }
>       flags &= tlb_addr;
>   
> -    *pfull = full = &cpu->neg.tlb.d[mmu_idx].fulltlb[index];
> +    full = &cpu->neg.tlb.d[mmu_idx].fulltlb[index];
>       flags |= full->slow_flags[access_type];
> +    *pfull = *full;
>   
>       /*
>        * Fold all "mmio-like" bits, and required plugin callbacks, to TLB_MMIO.
> @@ -1423,19 +1424,17 @@ int probe_access_full(CPUArchState *env, vaddr addr, int size,
>                         bool nonfault, void **phost, CPUTLBEntryFull *pfull,
>                         uintptr_t retaddr)
>   {
> -    CPUTLBEntryFull *full;
>       int flags = probe_access_internal(env_cpu(env), addr, size, access_type,
> -                                      mmu_idx, nonfault, phost, &full, retaddr,
> +                                      mmu_idx, nonfault, phost, pfull, retaddr,
>                                         true);
>   
>       /* Handle clean RAM pages.  */
>       if (unlikely(flags & TLB_NOTDIRTY)) {
>           int dirtysize = size == 0 ? 1 : size;
> -        notdirty_write(env_cpu(env), addr, dirtysize, full, retaddr);
> +        notdirty_write(env_cpu(env), addr, dirtysize, pfull, retaddr);
>           flags &= ~TLB_NOTDIRTY;
>       }
>   
> -    *pfull = *full;
>       return flags;
>   }
>   
> @@ -1444,25 +1443,22 @@ int probe_access_full_mmu(CPUArchState *env, vaddr addr, int size,
>                             void **phost, CPUTLBEntryFull *pfull)
>   {
>       void *discard_phost;
> -    CPUTLBEntryFull *full;
> +    CPUTLBEntryFull discard_full;
>   
>       /* privately handle users that don't need full results */
>       phost = phost ? phost : &discard_phost;
> +    pfull = pfull ? pfull : &discard_full;
>   
>       int flags = probe_access_internal(env_cpu(env), addr, size, access_type,
> -                                      mmu_idx, true, phost, &full, 0, false);
> +                                      mmu_idx, true, phost, pfull, 0, false);
>   
>       /* Handle clean RAM pages.  */
>       if (unlikely(flags & TLB_NOTDIRTY)) {
>           int dirtysize = size == 0 ? 1 : size;
> -        notdirty_write(env_cpu(env), addr, dirtysize, full, 0);
> +        notdirty_write(env_cpu(env), addr, dirtysize, pfull, 0);
>           flags &= ~TLB_NOTDIRTY;
>       }
>   
> -    if (pfull) {
> -        *pfull = *full;
> -    }
> -
>       return flags;
>   }
>   
> @@ -1470,7 +1466,7 @@ int probe_access_flags(CPUArchState *env, vaddr addr, int size,
>                          MMUAccessType access_type, int mmu_idx,
>                          bool nonfault, void **phost, uintptr_t retaddr)
>   {
> -    CPUTLBEntryFull *full;
> +    CPUTLBEntryFull full;
>       int flags;
>   
>       g_assert(-(addr | TARGET_PAGE_MASK) >= size);
> @@ -1482,7 +1478,7 @@ int probe_access_flags(CPUArchState *env, vaddr addr, int size,
>       /* Handle clean RAM pages. */
>       if (unlikely(flags & TLB_NOTDIRTY)) {
>           int dirtysize = size == 0 ? 1 : size;
> -        notdirty_write(env_cpu(env), addr, dirtysize, full, retaddr);
> +        notdirty_write(env_cpu(env), addr, dirtysize, &full, retaddr);
>           flags &= ~TLB_NOTDIRTY;
>       }
>   
> @@ -1492,7 +1488,7 @@ int probe_access_flags(CPUArchState *env, vaddr addr, int size,
>   void *probe_access(CPUArchState *env, vaddr addr, int size,
>                      MMUAccessType access_type, int mmu_idx, uintptr_t retaddr)
>   {
> -    CPUTLBEntryFull *full;
> +    CPUTLBEntryFull full;
>       void *host;
>       int flags;
>   
> @@ -1513,12 +1509,12 @@ void *probe_access(CPUArchState *env, vaddr addr, int size,
>               int wp_access = (access_type == MMU_DATA_STORE
>                                ? BP_MEM_WRITE : BP_MEM_READ);
>               cpu_check_watchpoint(env_cpu(env), addr, size,
> -                                 full->attrs, wp_access, retaddr);
> +                                 full.attrs, wp_access, retaddr);
>           }
>   
>           /* Handle clean RAM pages.  */
>           if (flags & TLB_NOTDIRTY) {
> -            notdirty_write(env_cpu(env), addr, size, full, retaddr);
> +            notdirty_write(env_cpu(env), addr, size, &full, retaddr);
>           }
>       }
>   
> @@ -1528,7 +1524,7 @@ void *probe_access(CPUArchState *env, vaddr addr, int size,
>   void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr addr,
>                           MMUAccessType access_type, int mmu_idx)
>   {
> -    CPUTLBEntryFull *full;
> +    CPUTLBEntryFull full;
>       void *host;
>       int flags;
>   
> @@ -1552,7 +1548,7 @@ void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr addr,
>   tb_page_addr_t get_page_addr_code_hostp(CPUArchState *env, vaddr addr,
>                                           void **hostp)
>   {
> -    CPUTLBEntryFull *full;
> +    CPUTLBEntryFull full;
>       void *p;
>   
>       (void)probe_access_internal(env_cpu(env), addr, 1, MMU_INST_FETCH,
> @@ -1562,7 +1558,7 @@ tb_page_addr_t get_page_addr_code_hostp(CPUArchState *env, vaddr addr,
>           return -1;
>       }
>   
> -    if (full->lg_page_size < TARGET_PAGE_BITS) {
> +    if (full.lg_page_size < TARGET_PAGE_BITS) {
>           return -1;
>       }
>   

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 28/54] accel/tcg: Introduce tlb_lookup

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Unify 3 instances of tlb lookup, through tlb_hit, tlbtree_hit,
> and tlb_full_align.  Use structures to avoid too many arguments.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---

This is a great simplification.

> +static void tlb_lookup_nofail(CPUState *cpu, TLBLookupOutput *o,
> +                              const TLBLookupInput *i)
> +{
> +    bool ok = tlb_lookup(cpu, o, i);
> +    tcg_debug_assert(ok);

 From the function name, would that be safe to use a normal assert 
instead? In case we have a weird bug coming in the future, it would 
expose it more easily.

For the rest,
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 29/54] accel/tcg: Partially unify MMULookupPageData and TLBLookupOutput

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 151 ++++++++++++++++++++++-----------------------
>   1 file changed, 74 insertions(+), 77 deletions(-)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index a33bebf55a..8f459be5a8 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -1684,10 +1684,7 @@ bool tlb_plugin_lookup(CPUState *cpu, vaddr addr, int mmu_idx,
>    */
>   
>   typedef struct MMULookupPageData {
> -    CPUTLBEntryFull *full;
> -    void *haddr;
>       vaddr addr;
> -    int flags;
>       int size;
>       TLBLookupOutput o;
>   } MMULookupPageData;
> @@ -1724,10 +1721,6 @@ static void mmu_lookup1(CPUState *cpu, MMULookupPageData *data, MemOp memop,
>       };
>   
>       tlb_lookup_nofail(cpu, &data->o, &i);
> -
> -    data->full = &data->o.full;
> -    data->flags = data->o.flags;
> -    data->haddr = data->o.haddr;
>   }
>   
>   /**
> @@ -1743,24 +1736,22 @@ static void mmu_lookup1(CPUState *cpu, MMULookupPageData *data, MemOp memop,
>   static void mmu_watch_or_dirty(CPUState *cpu, MMULookupPageData *data,
>                                  MMUAccessType access_type, uintptr_t ra)
>   {
> -    CPUTLBEntryFull *full = data->full;
> -    vaddr addr = data->addr;
> -    int flags = data->flags;
> -    int size = data->size;
> +    int flags = data->o.flags;
>   
>       /* On watchpoint hit, this will longjmp out.  */
>       if (flags & TLB_WATCHPOINT) {
>           int wp = access_type == MMU_DATA_STORE ? BP_MEM_WRITE : BP_MEM_READ;
> -        cpu_check_watchpoint(cpu, addr, size, full->attrs, wp, ra);
> +        cpu_check_watchpoint(cpu, data->addr, data->size,
> +                             data->o.full.attrs, wp, ra);
>           flags &= ~TLB_WATCHPOINT;
>       }
>   
>       /* Note that notdirty is only set for writes. */
>       if (flags & TLB_NOTDIRTY) {
> -        notdirty_write(cpu, addr, size, full, ra);
> +        notdirty_write(cpu, data->addr, data->size, &data->o.full, ra);
>           flags &= ~TLB_NOTDIRTY;
>       }
> -    data->flags = flags;
> +    data->o.flags = flags;
>   }
>   
>   /**
> @@ -1795,7 +1786,7 @@ static bool mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
>       if (likely(!crosspage)) {
>           mmu_lookup1(cpu, &l->page[0], l->memop, l->mmu_idx, type, ra);
>   
> -        flags = l->page[0].flags;
> +        flags = l->page[0].o.flags;
>           if (unlikely(flags & (TLB_WATCHPOINT | TLB_NOTDIRTY))) {
>               mmu_watch_or_dirty(cpu, &l->page[0], type, ra);
>           }
> @@ -1812,7 +1803,7 @@ static bool mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
>           mmu_lookup1(cpu, &l->page[0], l->memop, l->mmu_idx, type, ra);
>           mmu_lookup1(cpu, &l->page[1], 0, l->mmu_idx, type, ra);
>   
> -        flags = l->page[0].flags | l->page[1].flags;
> +        flags = l->page[0].o.flags | l->page[1].o.flags;
>           if (unlikely(flags & (TLB_WATCHPOINT | TLB_NOTDIRTY))) {
>               mmu_watch_or_dirty(cpu, &l->page[0], type, ra);
>               mmu_watch_or_dirty(cpu, &l->page[1], type, ra);
> @@ -2029,7 +2020,7 @@ static Int128 do_ld16_mmio_beN(CPUState *cpu, CPUTLBEntryFull *full,
>    */
>   static uint64_t do_ld_bytes_beN(MMULookupPageData *p, uint64_t ret_be)
>   {
> -    uint8_t *haddr = p->haddr;
> +    uint8_t *haddr = p->o.haddr;
>       int i, size = p->size;
>   
>       for (i = 0; i < size; i++) {
> @@ -2047,7 +2038,7 @@ static uint64_t do_ld_bytes_beN(MMULookupPageData *p, uint64_t ret_be)
>    */
>   static uint64_t do_ld_parts_beN(MMULookupPageData *p, uint64_t ret_be)
>   {
> -    void *haddr = p->haddr;
> +    void *haddr = p->o.haddr;
>       int size = p->size;
>   
>       do {
> @@ -2097,7 +2088,7 @@ static uint64_t do_ld_parts_beN(MMULookupPageData *p, uint64_t ret_be)
>   static uint64_t do_ld_whole_be4(MMULookupPageData *p, uint64_t ret_be)
>   {
>       int o = p->addr & 3;
> -    uint32_t x = load_atomic4(p->haddr - o);
> +    uint32_t x = load_atomic4(p->o.haddr - o);
>   
>       x = cpu_to_be32(x);
>       x <<= o * 8;
> @@ -2117,7 +2108,7 @@ static uint64_t do_ld_whole_be8(CPUState *cpu, uintptr_t ra,
>                                   MMULookupPageData *p, uint64_t ret_be)
>   {
>       int o = p->addr & 7;
> -    uint64_t x = load_atomic8_or_exit(cpu, ra, p->haddr - o);
> +    uint64_t x = load_atomic8_or_exit(cpu, ra, p->o.haddr - o);
>   
>       x = cpu_to_be64(x);
>       x <<= o * 8;
> @@ -2137,7 +2128,7 @@ static Int128 do_ld_whole_be16(CPUState *cpu, uintptr_t ra,
>                                  MMULookupPageData *p, uint64_t ret_be)
>   {
>       int o = p->addr & 15;
> -    Int128 x, y = load_atomic16_or_exit(cpu, ra, p->haddr - o);
> +    Int128 x, y = load_atomic16_or_exit(cpu, ra, p->o.haddr - o);
>       int size = p->size;
>   
>       if (!HOST_BIG_ENDIAN) {
> @@ -2160,8 +2151,8 @@ static uint64_t do_ld_beN(CPUState *cpu, MMULookupPageData *p,
>       MemOp atom;
>       unsigned tmp, half_size;
>   
> -    if (unlikely(p->flags & TLB_MMIO)) {
> -        return do_ld_mmio_beN(cpu, p->full, ret_be, p->addr, p->size,
> +    if (unlikely(p->o.flags & TLB_MMIO)) {
> +        return do_ld_mmio_beN(cpu, &p->o.full, ret_be, p->addr, p->size,
>                                 mmu_idx, type, ra);
>       }
>   
> @@ -2210,8 +2201,9 @@ static Int128 do_ld16_beN(CPUState *cpu, MMULookupPageData *p,
>       uint64_t b;
>       MemOp atom;
>   
> -    if (unlikely(p->flags & TLB_MMIO)) {
> -        return do_ld16_mmio_beN(cpu, p->full, a, p->addr, size, mmu_idx, ra);
> +    if (unlikely(p->o.flags & TLB_MMIO)) {
> +        return do_ld16_mmio_beN(cpu, &p->o.full, a, p->addr,
> +                                size, mmu_idx, ra);
>       }
>   
>       /*
> @@ -2223,7 +2215,7 @@ static Int128 do_ld16_beN(CPUState *cpu, MMULookupPageData *p,
>       case MO_ATOM_SUBALIGN:
>           p->size = size - 8;
>           a = do_ld_parts_beN(p, a);
> -        p->haddr += size - 8;
> +        p->o.haddr += size - 8;
>           p->size = 8;
>           b = do_ld_parts_beN(p, 0);
>           break;
> @@ -2242,7 +2234,7 @@ static Int128 do_ld16_beN(CPUState *cpu, MMULookupPageData *p,
>       case MO_ATOM_NONE:
>           p->size = size - 8;
>           a = do_ld_bytes_beN(p, a);
> -        b = ldq_be_p(p->haddr + size - 8);
> +        b = ldq_be_p(p->o.haddr + size - 8);
>           break;
>   
>       default:
> @@ -2255,10 +2247,11 @@ static Int128 do_ld16_beN(CPUState *cpu, MMULookupPageData *p,
>   static uint8_t do_ld_1(CPUState *cpu, MMULookupPageData *p, int mmu_idx,
>                          MMUAccessType type, uintptr_t ra)
>   {
> -    if (unlikely(p->flags & TLB_MMIO)) {
> -        return do_ld_mmio_beN(cpu, p->full, 0, p->addr, 1, mmu_idx, type, ra);
> +    if (unlikely(p->o.flags & TLB_MMIO)) {
> +        return do_ld_mmio_beN(cpu, &p->o.full, 0, p->addr, 1,
> +                              mmu_idx, type, ra);
>       } else {
> -        return *(uint8_t *)p->haddr;
> +        return *(uint8_t *)p->o.haddr;
>       }
>   }
>   
> @@ -2267,14 +2260,15 @@ static uint16_t do_ld_2(CPUState *cpu, MMULookupPageData *p, int mmu_idx,
>   {
>       uint16_t ret;
>   
> -    if (unlikely(p->flags & TLB_MMIO)) {
> -        ret = do_ld_mmio_beN(cpu, p->full, 0, p->addr, 2, mmu_idx, type, ra);
> +    if (unlikely(p->o.flags & TLB_MMIO)) {
> +        ret = do_ld_mmio_beN(cpu, &p->o.full, 0, p->addr, 2,
> +                             mmu_idx, type, ra);
>           if ((memop & MO_BSWAP) == MO_LE) {
>               ret = bswap16(ret);
>           }
>       } else {
>           /* Perform the load host endian, then swap if necessary. */
> -        ret = load_atom_2(cpu, ra, p->haddr, memop);
> +        ret = load_atom_2(cpu, ra, p->o.haddr, memop);
>           if (memop & MO_BSWAP) {
>               ret = bswap16(ret);
>           }
> @@ -2287,14 +2281,15 @@ static uint32_t do_ld_4(CPUState *cpu, MMULookupPageData *p, int mmu_idx,
>   {
>       uint32_t ret;
>   
> -    if (unlikely(p->flags & TLB_MMIO)) {
> -        ret = do_ld_mmio_beN(cpu, p->full, 0, p->addr, 4, mmu_idx, type, ra);
> +    if (unlikely(p->o.flags & TLB_MMIO)) {
> +        ret = do_ld_mmio_beN(cpu, &p->o.full, 0, p->addr, 4,
> +                             mmu_idx, type, ra);
>           if ((memop & MO_BSWAP) == MO_LE) {
>               ret = bswap32(ret);
>           }
>       } else {
>           /* Perform the load host endian. */
> -        ret = load_atom_4(cpu, ra, p->haddr, memop);
> +        ret = load_atom_4(cpu, ra, p->o.haddr, memop);
>           if (memop & MO_BSWAP) {
>               ret = bswap32(ret);
>           }
> @@ -2307,14 +2302,15 @@ static uint64_t do_ld_8(CPUState *cpu, MMULookupPageData *p, int mmu_idx,
>   {
>       uint64_t ret;
>   
> -    if (unlikely(p->flags & TLB_MMIO)) {
> -        ret = do_ld_mmio_beN(cpu, p->full, 0, p->addr, 8, mmu_idx, type, ra);
> +    if (unlikely(p->o.flags & TLB_MMIO)) {
> +        ret = do_ld_mmio_beN(cpu, &p->o.full, 0, p->addr, 8,
> +                             mmu_idx, type, ra);
>           if ((memop & MO_BSWAP) == MO_LE) {
>               ret = bswap64(ret);
>           }
>       } else {
>           /* Perform the load host endian. */
> -        ret = load_atom_8(cpu, ra, p->haddr, memop);
> +        ret = load_atom_8(cpu, ra, p->o.haddr, memop);
>           if (memop & MO_BSWAP) {
>               ret = bswap64(ret);
>           }
> @@ -2414,15 +2410,15 @@ static Int128 do_ld16_mmu(CPUState *cpu, vaddr addr,
>       cpu_req_mo(TCG_MO_LD_LD | TCG_MO_ST_LD);
>       crosspage = mmu_lookup(cpu, addr, oi, ra, MMU_DATA_LOAD, &l);
>       if (likely(!crosspage)) {
> -        if (unlikely(l.page[0].flags & TLB_MMIO)) {
> -            ret = do_ld16_mmio_beN(cpu, l.page[0].full, 0, addr, 16,
> +        if (unlikely(l.page[0].o.flags & TLB_MMIO)) {
> +            ret = do_ld16_mmio_beN(cpu, &l.page[0].o.full, 0, addr, 16,
>                                      l.mmu_idx, ra);
>               if ((l.memop & MO_BSWAP) == MO_LE) {
>                   ret = bswap128(ret);
>               }
>           } else {
>               /* Perform the load host endian. */
> -            ret = load_atom_16(cpu, ra, l.page[0].haddr, l.memop);
> +            ret = load_atom_16(cpu, ra, l.page[0].o.haddr, l.memop);
>               if (l.memop & MO_BSWAP) {
>                   ret = bswap128(ret);
>               }
> @@ -2568,10 +2564,10 @@ static uint64_t do_st_leN(CPUState *cpu, MMULookupPageData *p,
>       MemOp atom;
>       unsigned tmp, half_size;
>   
> -    if (unlikely(p->flags & TLB_MMIO)) {
> -        return do_st_mmio_leN(cpu, p->full, val_le, p->addr,
> +    if (unlikely(p->o.flags & TLB_MMIO)) {
> +        return do_st_mmio_leN(cpu, &p->o.full, val_le, p->addr,
>                                 p->size, mmu_idx, ra);
> -    } else if (unlikely(p->flags & TLB_DISCARD_WRITE)) {
> +    } else if (unlikely(p->o.flags & TLB_DISCARD_WRITE)) {
>           return val_le >> (p->size * 8);
>       }
>   
> @@ -2582,7 +2578,7 @@ static uint64_t do_st_leN(CPUState *cpu, MMULookupPageData *p,
>       atom = mop & MO_ATOM_MASK;
>       switch (atom) {
>       case MO_ATOM_SUBALIGN:
> -        return store_parts_leN(p->haddr, p->size, val_le);
> +        return store_parts_leN(p->o.haddr, p->size, val_le);
>   
>       case MO_ATOM_IFALIGN_PAIR:
>       case MO_ATOM_WITHIN16_PAIR:
> @@ -2593,9 +2589,9 @@ static uint64_t do_st_leN(CPUState *cpu, MMULookupPageData *p,
>               ? p->size == half_size
>               : p->size >= half_size) {
>               if (!HAVE_al8_fast && p->size <= 4) {
> -                return store_whole_le4(p->haddr, p->size, val_le);
> +                return store_whole_le4(p->o.haddr, p->size, val_le);
>               } else if (HAVE_al8) {
> -                return store_whole_le8(p->haddr, p->size, val_le);
> +                return store_whole_le8(p->o.haddr, p->size, val_le);
>               } else {
>                   cpu_loop_exit_atomic(cpu, ra);
>               }
> @@ -2605,7 +2601,7 @@ static uint64_t do_st_leN(CPUState *cpu, MMULookupPageData *p,
>       case MO_ATOM_IFALIGN:
>       case MO_ATOM_WITHIN16:
>       case MO_ATOM_NONE:
> -        return store_bytes_leN(p->haddr, p->size, val_le);
> +        return store_bytes_leN(p->o.haddr, p->size, val_le);
>   
>       default:
>           g_assert_not_reached();
> @@ -2622,10 +2618,10 @@ static uint64_t do_st16_leN(CPUState *cpu, MMULookupPageData *p,
>       int size = p->size;
>       MemOp atom;
>   
> -    if (unlikely(p->flags & TLB_MMIO)) {
> -        return do_st16_mmio_leN(cpu, p->full, val_le, p->addr,
> +    if (unlikely(p->o.flags & TLB_MMIO)) {
> +        return do_st16_mmio_leN(cpu, &p->o.full, val_le, p->addr,
>                                   size, mmu_idx, ra);
> -    } else if (unlikely(p->flags & TLB_DISCARD_WRITE)) {
> +    } else if (unlikely(p->o.flags & TLB_DISCARD_WRITE)) {
>           return int128_gethi(val_le) >> ((size - 8) * 8);
>       }
>   
> @@ -2636,8 +2632,8 @@ static uint64_t do_st16_leN(CPUState *cpu, MMULookupPageData *p,
>       atom = mop & MO_ATOM_MASK;
>       switch (atom) {
>       case MO_ATOM_SUBALIGN:
> -        store_parts_leN(p->haddr, 8, int128_getlo(val_le));
> -        return store_parts_leN(p->haddr + 8, p->size - 8,
> +        store_parts_leN(p->o.haddr, 8, int128_getlo(val_le));
> +        return store_parts_leN(p->o.haddr + 8, p->size - 8,
>                                  int128_gethi(val_le));
>   
>       case MO_ATOM_WITHIN16_PAIR:
> @@ -2645,7 +2641,7 @@ static uint64_t do_st16_leN(CPUState *cpu, MMULookupPageData *p,
>           if (!HAVE_CMPXCHG128) {
>               cpu_loop_exit_atomic(cpu, ra);
>           }
> -        return store_whole_le16(p->haddr, p->size, val_le);
> +        return store_whole_le16(p->o.haddr, p->size, val_le);
>   
>       case MO_ATOM_IFALIGN_PAIR:
>           /*
> @@ -2655,8 +2651,8 @@ static uint64_t do_st16_leN(CPUState *cpu, MMULookupPageData *p,
>       case MO_ATOM_IFALIGN:
>       case MO_ATOM_WITHIN16:
>       case MO_ATOM_NONE:
> -        stq_le_p(p->haddr, int128_getlo(val_le));
> -        return store_bytes_leN(p->haddr + 8, p->size - 8,
> +        stq_le_p(p->o.haddr, int128_getlo(val_le));
> +        return store_bytes_leN(p->o.haddr + 8, p->size - 8,
>                                  int128_gethi(val_le));
>   
>       default:
> @@ -2667,69 +2663,69 @@ static uint64_t do_st16_leN(CPUState *cpu, MMULookupPageData *p,
>   static void do_st_1(CPUState *cpu, MMULookupPageData *p, uint8_t val,
>                       int mmu_idx, uintptr_t ra)
>   {
> -    if (unlikely(p->flags & TLB_MMIO)) {
> -        do_st_mmio_leN(cpu, p->full, val, p->addr, 1, mmu_idx, ra);
> -    } else if (unlikely(p->flags & TLB_DISCARD_WRITE)) {
> +    if (unlikely(p->o.flags & TLB_MMIO)) {
> +        do_st_mmio_leN(cpu, &p->o.full, val, p->addr, 1, mmu_idx, ra);
> +    } else if (unlikely(p->o.flags & TLB_DISCARD_WRITE)) {
>           /* nothing */
>       } else {
> -        *(uint8_t *)p->haddr = val;
> +        *(uint8_t *)p->o.haddr = val;
>       }
>   }
>   
>   static void do_st_2(CPUState *cpu, MMULookupPageData *p, uint16_t val,
>                       int mmu_idx, MemOp memop, uintptr_t ra)
>   {
> -    if (unlikely(p->flags & TLB_MMIO)) {
> +    if (unlikely(p->o.flags & TLB_MMIO)) {
>           if ((memop & MO_BSWAP) != MO_LE) {
>               val = bswap16(val);
>           }
> -        do_st_mmio_leN(cpu, p->full, val, p->addr, 2, mmu_idx, ra);
> -    } else if (unlikely(p->flags & TLB_DISCARD_WRITE)) {
> +        do_st_mmio_leN(cpu, &p->o.full, val, p->addr, 2, mmu_idx, ra);
> +    } else if (unlikely(p->o.flags & TLB_DISCARD_WRITE)) {
>           /* nothing */
>       } else {
>           /* Swap to host endian if necessary, then store. */
>           if (memop & MO_BSWAP) {
>               val = bswap16(val);
>           }
> -        store_atom_2(cpu, ra, p->haddr, memop, val);
> +        store_atom_2(cpu, ra, p->o.haddr, memop, val);
>       }
>   }
>   
>   static void do_st_4(CPUState *cpu, MMULookupPageData *p, uint32_t val,
>                       int mmu_idx, MemOp memop, uintptr_t ra)
>   {
> -    if (unlikely(p->flags & TLB_MMIO)) {
> +    if (unlikely(p->o.flags & TLB_MMIO)) {
>           if ((memop & MO_BSWAP) != MO_LE) {
>               val = bswap32(val);
>           }
> -        do_st_mmio_leN(cpu, p->full, val, p->addr, 4, mmu_idx, ra);
> -    } else if (unlikely(p->flags & TLB_DISCARD_WRITE)) {
> +        do_st_mmio_leN(cpu, &p->o.full, val, p->addr, 4, mmu_idx, ra);
> +    } else if (unlikely(p->o.flags & TLB_DISCARD_WRITE)) {
>           /* nothing */
>       } else {
>           /* Swap to host endian if necessary, then store. */
>           if (memop & MO_BSWAP) {
>               val = bswap32(val);
>           }
> -        store_atom_4(cpu, ra, p->haddr, memop, val);
> +        store_atom_4(cpu, ra, p->o.haddr, memop, val);
>       }
>   }
>   
>   static void do_st_8(CPUState *cpu, MMULookupPageData *p, uint64_t val,
>                       int mmu_idx, MemOp memop, uintptr_t ra)
>   {
> -    if (unlikely(p->flags & TLB_MMIO)) {
> +    if (unlikely(p->o.flags & TLB_MMIO)) {
>           if ((memop & MO_BSWAP) != MO_LE) {
>               val = bswap64(val);
>           }
> -        do_st_mmio_leN(cpu, p->full, val, p->addr, 8, mmu_idx, ra);
> -    } else if (unlikely(p->flags & TLB_DISCARD_WRITE)) {
> +        do_st_mmio_leN(cpu, &p->o.full, val, p->addr, 8, mmu_idx, ra);
> +    } else if (unlikely(p->o.flags & TLB_DISCARD_WRITE)) {
>           /* nothing */
>       } else {
>           /* Swap to host endian if necessary, then store. */
>           if (memop & MO_BSWAP) {
>               val = bswap64(val);
>           }
> -        store_atom_8(cpu, ra, p->haddr, memop, val);
> +        store_atom_8(cpu, ra, p->o.haddr, memop, val);
>       }
>   }
>   
> @@ -2822,19 +2818,20 @@ static void do_st16_mmu(CPUState *cpu, vaddr addr, Int128 val,
>       cpu_req_mo(TCG_MO_LD_ST | TCG_MO_ST_ST);
>       crosspage = mmu_lookup(cpu, addr, oi, ra, MMU_DATA_STORE, &l);
>       if (likely(!crosspage)) {
> -        if (unlikely(l.page[0].flags & TLB_MMIO)) {
> +        if (unlikely(l.page[0].o.flags & TLB_MMIO)) {
>               if ((l.memop & MO_BSWAP) != MO_LE) {
>                   val = bswap128(val);
>               }
> -            do_st16_mmio_leN(cpu, l.page[0].full, val, addr, 16, l.mmu_idx, ra);
> -        } else if (unlikely(l.page[0].flags & TLB_DISCARD_WRITE)) {
> +            do_st16_mmio_leN(cpu, &l.page[0].o.full, val, addr,
> +                             16, l.mmu_idx, ra);
> +        } else if (unlikely(l.page[0].o.flags & TLB_DISCARD_WRITE)) {
>               /* nothing */
>           } else {
>               /* Swap to host endian if necessary, then store. */
>               if (l.memop & MO_BSWAP) {
>                   val = bswap128(val);
>               }
> -            store_atom_16(cpu, ra, l.page[0].haddr, l.memop, val);
> +            store_atom_16(cpu, ra, l.page[0].o.haddr, l.memop, val);
>           }
>           return;
>       }

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 30/54] accel/tcg: Merge mmu_lookup1 into mmu_lookup

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Reuse most of TLBLookupInput between calls to tlb_lookup.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 65 ++++++++++++++++++----------------------------
>   1 file changed, 25 insertions(+), 40 deletions(-)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index 8f459be5a8..981098a6f2 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -1695,34 +1695,6 @@ typedef struct MMULookupLocals {
>       int mmu_idx;
>   } MMULookupLocals;
>   
> -/**
> - * mmu_lookup1: translate one page
> - * @cpu: generic cpu state
> - * @data: lookup parameters
> - * @memop: memory operation for the access, or 0
> - * @mmu_idx: virtual address context
> - * @access_type: load/store/code
> - * @ra: return address into tcg generated code, or 0
> - *
> - * Resolve the translation for the one page at @data.addr, filling in
> - * the rest of @data with the results.  If the translation fails,
> - * tlb_fill_align will longjmp out.
> - */
> -static void mmu_lookup1(CPUState *cpu, MMULookupPageData *data, MemOp memop,
> -                        int mmu_idx, MMUAccessType access_type, uintptr_t ra)
> -{
> -    TLBLookupInput i = {
> -        .addr = data->addr,
> -        .ra = ra,
> -        .access_type = access_type,
> -        .memop_probe = memop,
> -        .size = data->size,
> -        .mmu_idx = mmu_idx,
> -    };
> -
> -    tlb_lookup_nofail(cpu, &data->o, &i);
> -}
> -
>   /**
>    * mmu_watch_or_dirty
>    * @cpu: generic cpu state
> @@ -1769,26 +1741,36 @@ static void mmu_watch_or_dirty(CPUState *cpu, MMULookupPageData *data,
>   static bool mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
>                          uintptr_t ra, MMUAccessType type, MMULookupLocals *l)
>   {
> +    MemOp memop = get_memop(oi);
> +    int mmu_idx = get_mmuidx(oi);
> +    TLBLookupInput i = {
> +        .addr = addr,
> +        .ra = ra,
> +        .access_type = type,
> +        .memop_probe = memop,
> +        .size = memop_size(memop),
> +        .mmu_idx = mmu_idx,
> +    };
>       bool crosspage;
>       int flags;
>   
> -    l->memop = get_memop(oi);
> -    l->mmu_idx = get_mmuidx(oi);
> +    l->memop = memop;
> +    l->mmu_idx = mmu_idx;
>   
> -    tcg_debug_assert(l->mmu_idx < NB_MMU_MODES);
> +    tcg_debug_assert(mmu_idx < NB_MMU_MODES);
>   
>       l->page[0].addr = addr;
> -    l->page[0].size = memop_size(l->memop);
> -    l->page[1].addr = (addr + l->page[0].size - 1) & TARGET_PAGE_MASK;
> +    l->page[0].size = i.size;
> +    l->page[1].addr = (addr + i.size - 1) & TARGET_PAGE_MASK;
>       l->page[1].size = 0;
>       crosspage = (addr ^ l->page[1].addr) & TARGET_PAGE_MASK;
>   
>       if (likely(!crosspage)) {
> -        mmu_lookup1(cpu, &l->page[0], l->memop, l->mmu_idx, type, ra);
> +        tlb_lookup_nofail(cpu, &l->page[0].o, &i);
>   
>           flags = l->page[0].o.flags;
>           if (unlikely(flags & (TLB_WATCHPOINT | TLB_NOTDIRTY))) {
> -            mmu_watch_or_dirty(cpu, &l->page[0], type, ra);
> +            mmu_watch_or_dirty(cpu, &l->page[0], i.access_type, i.ra);
>           }
>           if (unlikely(flags & TLB_BSWAP)) {
>               l->memop ^= MO_BSWAP;
> @@ -1796,17 +1778,20 @@ static bool mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
>       } else {
>           /* Finish compute of page crossing. */
>           int size0 = l->page[1].addr - addr;
> -        l->page[1].size = l->page[0].size - size0;
> +        l->page[1].size = i.size - size0;
>           l->page[0].size = size0;
>   
>           /* Lookup both pages, recognizing exceptions from either. */
> -        mmu_lookup1(cpu, &l->page[0], l->memop, l->mmu_idx, type, ra);
> -        mmu_lookup1(cpu, &l->page[1], 0, l->mmu_idx, type, ra);
> +        i.size = size0;
> +        tlb_lookup_nofail(cpu, &l->page[0].o, &i);
> +        i.addr = l->page[1].addr;
> +        i.size = l->page[1].size;
> +        tlb_lookup_nofail(cpu, &l->page[1].o, &i);
>   
>           flags = l->page[0].o.flags | l->page[1].o.flags;
>           if (unlikely(flags & (TLB_WATCHPOINT | TLB_NOTDIRTY))) {
> -            mmu_watch_or_dirty(cpu, &l->page[0], type, ra);
> -            mmu_watch_or_dirty(cpu, &l->page[1], type, ra);
> +            mmu_watch_or_dirty(cpu, &l->page[0], i.access_type, i.ra);
> +            mmu_watch_or_dirty(cpu, &l->page[1], i.access_type, i.ra);
>           }
>   
>           /*

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 31/54] accel/tcg: Always use IntervalTree for code lookups

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Because translation is special, we don't need the speed
> of the direct-mapped softmmu tlb.  We cache a lookups in
> DisasContextBase within the translator loop anyway.
> 
> Drop the addr_code comparator from CPUTLBEntry.
> Go directly to the IntervalTree for MMU_INST_FETCH.
> Derive exec flags from read flags.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   include/exec/cpu-all.h    |  3 ++
>   include/exec/tlb-common.h |  5 ++-
>   accel/tcg/cputlb.c        | 76 ++++++++++++++++++++++++---------------
>   3 files changed, 52 insertions(+), 32 deletions(-)
> 
> diff --git a/include/exec/cpu-all.h b/include/exec/cpu-all.h
> index 45e6676938..ad160c328a 100644
> --- a/include/exec/cpu-all.h
> +++ b/include/exec/cpu-all.h
> @@ -339,6 +339,9 @@ static inline int cpu_mmu_index(CPUState *cs, bool ifetch)
>       (TLB_INVALID_MASK | TLB_NOTDIRTY | TLB_MMIO \
>       | TLB_FORCE_SLOW | TLB_DISCARD_WRITE)
>   
> +/* Filter read flags to exec flags. */
> +#define TLB_EXEC_FLAGS_MASK  (TLB_MMIO)
> +
>   /*
>    * Flags stored in CPUTLBEntryFull.slow_flags[x].
>    * TLB_FORCE_SLOW must be set in CPUTLBEntry.addr_idx[x].
> diff --git a/include/exec/tlb-common.h b/include/exec/tlb-common.h
> index 300f9fae67..feaa471299 100644
> --- a/include/exec/tlb-common.h
> +++ b/include/exec/tlb-common.h
> @@ -26,7 +26,6 @@ typedef union CPUTLBEntry {
>       struct {
>           uint64_t addr_read;
>           uint64_t addr_write;
> -        uint64_t addr_code;
>           /*
>            * Addend to virtual address to get host address.  IO accesses
>            * use the corresponding iotlb value.
> @@ -35,7 +34,7 @@ typedef union CPUTLBEntry {
>       };
>       /*
>        * Padding to get a power of two size, as well as index
> -     * access to addr_{read,write,code}.
> +     * access to addr_{read,write}.
>        */
>       uint64_t addr_idx[(1 << CPU_TLB_ENTRY_BITS) / sizeof(uint64_t)];
>   } CPUTLBEntry;
> @@ -92,7 +91,7 @@ struct CPUTLBEntryFull {
>        * Additional tlb flags for use by the slow path. If non-zero,
>        * the corresponding CPUTLBEntry comparator must have TLB_FORCE_SLOW.
>        */
> -    uint8_t slow_flags[MMU_ACCESS_COUNT];
> +    uint8_t slow_flags[2];
>   
>       /*
>        * Allow target-specific additions to this structure.
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index 981098a6f2..be2ea1bc70 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -114,8 +114,9 @@ static inline uint64_t tlb_read_idx(const CPUTLBEntry *entry,
>                         MMU_DATA_LOAD * sizeof(uint64_t));
>       QEMU_BUILD_BUG_ON(offsetof(CPUTLBEntry, addr_write) !=
>                         MMU_DATA_STORE * sizeof(uint64_t));
> -    QEMU_BUILD_BUG_ON(offsetof(CPUTLBEntry, addr_code) !=
> -                      MMU_INST_FETCH * sizeof(uint64_t));
> +
> +    tcg_debug_assert(access_type == MMU_DATA_LOAD ||
> +                     access_type == MMU_DATA_STORE);
>   
>   #if TARGET_LONG_BITS == 32
>       /* Use qatomic_read, in case of addr_write; only care about low bits. */
> @@ -480,8 +481,7 @@ static bool tlb_hit_page_mask_anyprot(CPUTLBEntry *tlb_entry,
>       mask &= TARGET_PAGE_MASK | TLB_INVALID_MASK;
>   
>       return (page == (tlb_entry->addr_read & mask) ||
> -            page == (tlb_addr_write(tlb_entry) & mask) ||
> -            page == (tlb_entry->addr_code & mask));
> +            page == (tlb_addr_write(tlb_entry) & mask));
>   }
>   
>   /* Called with tlb_c.lock held */
> @@ -1184,9 +1184,6 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
>       /* Now calculate the new entry */
>       node->copy.addend = addend - addr_page;
>   
> -    tlb_set_compare(full, &node->copy, addr_page, read_flags,
> -                    MMU_INST_FETCH, prot & PAGE_EXEC);
> -
>       if (wp_flags & BP_MEM_READ) {
>           read_flags |= TLB_WATCHPOINT;
>       }
> @@ -1308,22 +1305,30 @@ static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
>       /* Primary lookup in the fast tlb. */
>       entry = tlbfast_entry(fast, addr);
>       full = &desc->fulltlb[tlbfast_index(fast, addr)];
> -    cmp = tlb_read_idx(entry, access_type);
> -    if (tlb_hit(cmp, addr)) {
> -        goto found;
> +    if (access_type != MMU_INST_FETCH) {
> +        cmp = tlb_read_idx(entry, access_type);
> +        if (tlb_hit(cmp, addr)) {
> +            goto found_data;
> +        }
>       }
>   
>       /* Secondary lookup in the IntervalTree. */
>       node = tlbtree_lookup_addr(desc, addr);
>       if (node) {
> -        cmp = tlb_read_idx(&node->copy, access_type);
> -        if (tlb_hit(cmp, addr)) {
> -            /* Install the cached entry. */
> -            qemu_spin_lock(&cpu->neg.tlb.c.lock);
> -            copy_tlb_helper_locked(entry, &node->copy);
> -            qemu_spin_unlock(&cpu->neg.tlb.c.lock);
> -            *full = node->full;
> -            goto found;
> +        if (access_type == MMU_INST_FETCH) {
> +            if (node->full.prot & PAGE_EXEC) {
> +                goto found_code;
> +            }
> +        } else {
> +            cmp = tlb_read_idx(&node->copy, access_type);
> +            if (tlb_hit(cmp, addr)) {
> +                /* Install the cached entry. */
> +                qemu_spin_lock(&cpu->neg.tlb.c.lock);
> +                copy_tlb_helper_locked(entry, &node->copy);
> +                qemu_spin_unlock(&cpu->neg.tlb.c.lock);
> +                *full = node->full;
> +                goto found_data;
> +            }
>           }
>       }
>   
> @@ -1333,9 +1338,14 @@ static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
>           tcg_debug_assert(probe);
>           return false;
>       }
> -
>       o->did_tlb_fill = true;
>   
> +    if (access_type == MMU_INST_FETCH) {
> +        node = tlbtree_lookup_addr(desc, addr);
> +        tcg_debug_assert(node);
> +        goto found_code;
> +    }
> +
>       entry = tlbfast_entry(fast, addr);
>       full = &desc->fulltlb[tlbfast_index(fast, addr)];
>       cmp = tlb_read_idx(entry, access_type);
> @@ -1345,14 +1355,29 @@ static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
>        * called tlb_fill_align, so we know that this entry *is* valid.
>        */
>       flags &= ~TLB_INVALID_MASK;
> +    goto found_data;
> +
> + found_data:
> +    flags &= cmp;
> +    flags |= full->slow_flags[access_type];
> +    o->flags = flags;
> +    o->full = *full;
> +    o->haddr = (void *)((uintptr_t)addr + entry->addend);
>       goto done;
>   
> - found:
> -    /* Alignment has not been checked by tlb_fill_align. */
> -    {
> + found_code:
> +    o->flags = node->copy.addr_read & TLB_EXEC_FLAGS_MASK;
> +    o->full = node->full;
> +    o->haddr = (void *)((uintptr_t)addr + node->copy.addend);
> +    goto done;
> +
> + done:
> +    if (!o->did_tlb_fill) {
>           int a_bits = memop_alignment_bits(memop);
>   
>           /*
> +         * Alignment has not been checked by tlb_fill_align.
> +         *
>            * The TLB_CHECK_ALIGNED check differs from the normal alignment
>            * check, in that this is based on the atomicity of the operation.
>            * The intended use case is the ARM memory type field of each PTE,
> @@ -1366,13 +1391,6 @@ static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
>               cpu_unaligned_access(cpu, addr, access_type, i->mmu_idx, i->ra);
>           }
>       }
> -
> - done:
> -    flags &= cmp;
> -    flags |= full->slow_flags[access_type];
> -    o->flags = flags;
> -    o->full = *full;
> -    o->haddr = (void *)((uintptr_t)addr + entry->addend);
>       return true;
>   }
>   

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 32/54] accel/tcg: Link CPUTLBEntry to CPUTLBEntryTree

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Link from the fast tlb entry to the interval tree node.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   include/exec/tlb-common.h |  2 ++
>   accel/tcg/cputlb.c        | 26 +++++++++++++-------------
>   2 files changed, 15 insertions(+), 13 deletions(-)
> 
> diff --git a/include/exec/tlb-common.h b/include/exec/tlb-common.h
> index feaa471299..3b57d61112 100644
> --- a/include/exec/tlb-common.h
> +++ b/include/exec/tlb-common.h
> @@ -31,6 +31,8 @@ typedef union CPUTLBEntry {
>            * use the corresponding iotlb value.
>            */
>           uintptr_t addend;
> +        /* The defining IntervalTree entry. */
> +        struct CPUTLBEntryTree *tree;
>       };
>       /*
>        * Padding to get a power of two size, as well as index
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index be2ea1bc70..3282436752 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -490,7 +490,10 @@ static bool tlb_flush_entry_mask_locked(CPUTLBEntry *tlb_entry,
>                                           vaddr mask)
>   {
>       if (tlb_hit_page_mask_anyprot(tlb_entry, page, mask)) {
> -        memset(tlb_entry, -1, sizeof(*tlb_entry));
> +        tlb_entry->addr_read = -1;
> +        tlb_entry->addr_write = -1;
> +        tlb_entry->addend = 0;
> +        tlb_entry->tree = NULL;
>           return true;
>       }
>       return false;
> @@ -1183,6 +1186,7 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
>   
>       /* Now calculate the new entry */
>       node->copy.addend = addend - addr_page;
> +    node->copy.tree = node;
>   
>       if (wp_flags & BP_MEM_READ) {
>           read_flags |= TLB_WATCHPOINT;
> @@ -1291,7 +1295,6 @@ static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
>       CPUTLBDescFast *fast = &cpu->neg.tlb.f[i->mmu_idx];
>       vaddr addr = i->addr;
>       MMUAccessType access_type = i->access_type;
> -    CPUTLBEntryFull *full;
>       CPUTLBEntryTree *node;
>       CPUTLBEntry *entry;
>       uint64_t cmp;
> @@ -1304,9 +1307,9 @@ static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
>   
>       /* Primary lookup in the fast tlb. */
>       entry = tlbfast_entry(fast, addr);
> -    full = &desc->fulltlb[tlbfast_index(fast, addr)];
>       if (access_type != MMU_INST_FETCH) {
>           cmp = tlb_read_idx(entry, access_type);
> +        node = entry->tree;
>           if (tlb_hit(cmp, addr)) {
>               goto found_data;
>           }
> @@ -1326,7 +1329,6 @@ static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
>                   qemu_spin_lock(&cpu->neg.tlb.c.lock);
>                   copy_tlb_helper_locked(entry, &node->copy);
>                   qemu_spin_unlock(&cpu->neg.tlb.c.lock);
> -                *full = node->full;
>                   goto found_data;
>               }
>           }
> @@ -1347,8 +1349,8 @@ static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
>       }
>   
>       entry = tlbfast_entry(fast, addr);
> -    full = &desc->fulltlb[tlbfast_index(fast, addr)];
>       cmp = tlb_read_idx(entry, access_type);
> +    node = entry->tree;
>       /*
>        * With PAGE_WRITE_INV, we set TLB_INVALID_MASK immediately,
>        * to force the next access through tlb_fill_align.  We've just
> @@ -1359,19 +1361,18 @@ static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
>   
>    found_data:
>       flags &= cmp;
> -    flags |= full->slow_flags[access_type];
> +    flags |= node->full.slow_flags[access_type];
>       o->flags = flags;
> -    o->full = *full;
> -    o->haddr = (void *)((uintptr_t)addr + entry->addend);
> -    goto done;
> +    goto found_common;
>   
>    found_code:
>       o->flags = node->copy.addr_read & TLB_EXEC_FLAGS_MASK;
> +    goto found_common;
> +
> + found_common:
>       o->full = node->full;
>       o->haddr = (void *)((uintptr_t)addr + node->copy.addend);
> -    goto done;
>   
> - done:
>       if (!o->did_tlb_fill) {
>           int a_bits = memop_alignment_bits(memop);
>   
> @@ -1669,7 +1670,6 @@ bool tlb_plugin_lookup(CPUState *cpu, vaddr addr, int mmu_idx,
>                          bool is_store, struct qemu_plugin_hwaddr *data)
>   {
>       CPUTLBEntry *tlbe = tlb_entry(cpu, mmu_idx, addr);
> -    uintptr_t index = tlb_index(cpu, mmu_idx, addr);
>       MMUAccessType access_type = is_store ? MMU_DATA_STORE : MMU_DATA_LOAD;
>       uint64_t tlb_addr = tlb_read_idx(tlbe, access_type);
>       CPUTLBEntryFull *full;
> @@ -1678,7 +1678,7 @@ bool tlb_plugin_lookup(CPUState *cpu, vaddr addr, int mmu_idx,
>           return false;
>       }
>   
> -    full = &cpu->neg.tlb.d[mmu_idx].fulltlb[index];
> +    full = &tlbe->tree->full;
>       data->phys_addr = full->phys_addr | (addr & ~TARGET_PAGE_MASK);
>   
>       /* We must have an iotlb entry for MMIO */

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 33/54] accel/tcg: Remove CPUTLBDesc.fulltlb

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> This array is now write-only, and may be removed.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   include/hw/core/cpu.h |  1 -
>   accel/tcg/cputlb.c    | 34 +++++++---------------------------
>   2 files changed, 7 insertions(+), 28 deletions(-)
> 
> diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
> index 4364ddb1db..5c069f2a00 100644
> --- a/include/hw/core/cpu.h
> +++ b/include/hw/core/cpu.h
> @@ -219,7 +219,6 @@ typedef struct CPUTLBDesc {
>       /* maximum number of entries observed in the window */
>       size_t window_max_entries;
>       size_t n_used_entries;
> -    CPUTLBEntryFull *fulltlb;
>       /* All active tlb entries for this address space. */
>       IntervalTreeRoot iroot;
>   } CPUTLBDesc;
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index 3282436752..7f63dc3fd8 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -149,13 +149,6 @@ static inline CPUTLBEntry *tlbfast_entry(CPUTLBDescFast *fast, vaddr addr)
>       return fast->table + tlbfast_index(fast, addr);
>   }
>   
> -/* Find the TLB index corresponding to the mmu_idx + address pair.  */
> -static inline uintptr_t tlb_index(CPUState *cpu, uintptr_t mmu_idx,
> -                                  vaddr addr)
> -{
> -    return tlbfast_index(&cpu->neg.tlb.f[mmu_idx], addr);
> -}
> -
>   /* Find the TLB entry corresponding to the mmu_idx + address pair.  */
>   static inline CPUTLBEntry *tlb_entry(CPUState *cpu, uintptr_t mmu_idx,
>                                        vaddr addr)
> @@ -270,22 +263,20 @@ static void tlb_mmu_resize_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast,
>       }
>   
>       g_free(fast->table);
> -    g_free(desc->fulltlb);
>   
>       tlb_window_reset(desc, now, 0);
>       /* desc->n_used_entries is cleared by the caller */
>       fast->mask = (new_size - 1) << CPU_TLB_ENTRY_BITS;
>       fast->table = g_try_new(CPUTLBEntry, new_size);
> -    desc->fulltlb = g_try_new(CPUTLBEntryFull, new_size);
>   
>       /*
> -     * If the allocations fail, try smaller sizes. We just freed some
> +     * If the allocation fails, try smaller sizes. We just freed some
>        * memory, so going back to half of new_size has a good chance of working.
>        * Increased memory pressure elsewhere in the system might cause the
>        * allocations to fail though, so we progressively reduce the allocation
>        * size, aborting if we cannot even allocate the smallest TLB we support.
>        */
> -    while (fast->table == NULL || desc->fulltlb == NULL) {
> +    while (fast->table == NULL) {
>           if (new_size == (1 << CPU_TLB_DYN_MIN_BITS)) {
>               error_report("%s: %s", __func__, strerror(errno));
>               abort();
> @@ -294,9 +285,7 @@ static void tlb_mmu_resize_locked(CPUTLBDesc *desc, CPUTLBDescFast *fast,
>           fast->mask = (new_size - 1) << CPU_TLB_ENTRY_BITS;
>   
>           g_free(fast->table);
> -        g_free(desc->fulltlb);
>           fast->table = g_try_new(CPUTLBEntry, new_size);
> -        desc->fulltlb = g_try_new(CPUTLBEntryFull, new_size);
>       }
>   }
>   
> @@ -350,7 +339,6 @@ static void tlb_mmu_init(CPUTLBDesc *desc, CPUTLBDescFast *fast, int64_t now)
>       desc->n_used_entries = 0;
>       fast->mask = (n_entries - 1) << CPU_TLB_ENTRY_BITS;
>       fast->table = g_new(CPUTLBEntry, n_entries);
> -    desc->fulltlb = g_new(CPUTLBEntryFull, n_entries);
>       memset(&desc->iroot, 0, sizeof(desc->iroot));
>       tlb_mmu_flush_locked(desc, fast);
>   }
> @@ -372,15 +360,9 @@ void tlb_init(CPUState *cpu)
>   
>   void tlb_destroy(CPUState *cpu)
>   {
> -    int i;
> -
>       qemu_spin_destroy(&cpu->neg.tlb.c.lock);
> -    for (i = 0; i < NB_MMU_MODES; i++) {
> -        CPUTLBDesc *desc = &cpu->neg.tlb.d[i];
> -        CPUTLBDescFast *fast = &cpu->neg.tlb.f[i];
> -
> -        g_free(fast->table);
> -        g_free(desc->fulltlb);
> +    for (int i = 0; i < NB_MMU_MODES; i++) {
> +        g_free(cpu->neg.tlb.f[i].table);
>           interval_tree_free_nodes(&cpu->neg.tlb.d[i].iroot,
>                                    offsetof(CPUTLBEntryTree, itree));
>       }
> @@ -1061,7 +1043,7 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
>       CPUTLB *tlb = &cpu->neg.tlb;
>       CPUTLBDesc *desc = &tlb->d[mmu_idx];
>       MemoryRegionSection *section;
> -    unsigned int index, read_flags, write_flags;
> +    unsigned int read_flags, write_flags;
>       uintptr_t addend;
>       CPUTLBEntry *te;
>       CPUTLBEntryTree *node;
> @@ -1140,7 +1122,6 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
>       wp_flags = cpu_watchpoint_address_matches(cpu, addr_page,
>                                                 TARGET_PAGE_SIZE);
>   
> -    index = tlb_index(cpu, mmu_idx, addr_page);
>       te = tlb_entry(cpu, mmu_idx, addr_page);
>   
>       /*
> @@ -1179,8 +1160,8 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
>        * subtract here is that of the page base, and not the same as the
>        * vaddr we add back in io_prepare()/get_page_addr_code().
>        */
> -    desc->fulltlb[index] = *full;
> -    full = &desc->fulltlb[index];
> +    node->full = *full;
> +    full = &node->full;
>       full->xlat_section = iotlb - addr_page;
>       full->phys_addr = paddr_page;
>   
> @@ -1203,7 +1184,6 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
>       tlb_set_compare(full, &node->copy, addr_page, write_flags,
>                       MMU_DATA_STORE, prot & PAGE_WRITE);
>   
> -    node->full = *full;
>       copy_tlb_helper_locked(te, &node->copy);
>       desc->n_used_entries++;
>       qemu_spin_unlock(&tlb->c.lock);

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 34/54] target/alpha: Convert to TCGCPUOps.tlb_fill_align

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   target/alpha/cpu.h    |  6 +++---
>   target/alpha/cpu.c    |  2 +-
>   target/alpha/helper.c | 23 +++++++++++++++++------
>   3 files changed, 21 insertions(+), 10 deletions(-)
> 
> diff --git a/target/alpha/cpu.h b/target/alpha/cpu.h
> index 3556d3227f..70331c0b83 100644
> --- a/target/alpha/cpu.h
> +++ b/target/alpha/cpu.h
> @@ -449,9 +449,9 @@ void alpha_cpu_record_sigsegv(CPUState *cs, vaddr address,
>   void alpha_cpu_record_sigbus(CPUState *cs, vaddr address,
>                                MMUAccessType access_type, uintptr_t retaddr);
>   #else
> -bool alpha_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                        MMUAccessType access_type, int mmu_idx,
> -                        bool probe, uintptr_t retaddr);
> +bool alpha_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out, vaddr addr,
> +                              MMUAccessType access_type, int mmu_idx,
> +                              MemOp memop, int size, bool probe, uintptr_t ra);
>   G_NORETURN void alpha_cpu_do_unaligned_access(CPUState *cpu, vaddr addr,
>                                                 MMUAccessType access_type, int mmu_idx,
>                                                 uintptr_t retaddr);
> diff --git a/target/alpha/cpu.c b/target/alpha/cpu.c
> index 5d75c941f7..7bcc48420d 100644
> --- a/target/alpha/cpu.c
> +++ b/target/alpha/cpu.c
> @@ -228,7 +228,7 @@ static const TCGCPUOps alpha_tcg_ops = {
>       .record_sigsegv = alpha_cpu_record_sigsegv,
>       .record_sigbus = alpha_cpu_record_sigbus,
>   #else
> -    .tlb_fill = alpha_cpu_tlb_fill,
> +    .tlb_fill_align = alpha_cpu_tlb_fill_align,
>       .cpu_exec_interrupt = alpha_cpu_exec_interrupt,
>       .cpu_exec_halt = alpha_cpu_has_work,
>       .do_interrupt = alpha_cpu_do_interrupt,
> diff --git a/target/alpha/helper.c b/target/alpha/helper.c
> index 2f1000c99f..26eadfe3ca 100644
> --- a/target/alpha/helper.c
> +++ b/target/alpha/helper.c
> @@ -294,14 +294,21 @@ hwaddr alpha_cpu_get_phys_page_debug(CPUState *cs, vaddr addr)
>       return (fail >= 0 ? -1 : phys);
>   }
>   
> -bool alpha_cpu_tlb_fill(CPUState *cs, vaddr addr, int size,
> -                        MMUAccessType access_type, int mmu_idx,
> -                        bool probe, uintptr_t retaddr)
> +bool alpha_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out, vaddr addr,
> +                              MMUAccessType access_type, int mmu_idx,
> +                              MemOp memop, int size, bool probe, uintptr_t ra)
>   {
>       CPUAlphaState *env = cpu_env(cs);
>       target_ulong phys;
>       int prot, fail;
>   
> +    if (addr & ((1 << memop_alignment_bits(memop)) - 1)) {
> +        if (probe) {
> +            return false;
> +        }
> +        alpha_cpu_do_unaligned_access(cs, addr, access_type, mmu_idx, ra);
> +    }
> +
>       fail = get_physical_address(env, addr, 1 << access_type,
>                                   mmu_idx, &phys, &prot);
>       if (unlikely(fail >= 0)) {
> @@ -314,11 +321,15 @@ bool alpha_cpu_tlb_fill(CPUState *cs, vaddr addr, int size,
>           env->trap_arg2 = (access_type == MMU_DATA_LOAD ? 0ull :
>                             access_type == MMU_DATA_STORE ? 1ull :
>                             /* access_type == MMU_INST_FETCH */ -1ull);
> -        cpu_loop_exit_restore(cs, retaddr);
> +        cpu_loop_exit_restore(cs, ra);
>       }
>   
> -    tlb_set_page(cs, addr & TARGET_PAGE_MASK, phys & TARGET_PAGE_MASK,
> -                 prot, mmu_idx, TARGET_PAGE_SIZE);
> +    memset(out, 0, sizeof(*out));
> +    out->phys_addr = phys;
> +    out->prot = prot;
> +    out->attrs = MEMTXATTRS_UNSPECIFIED;
> +    out->lg_page_size = TARGET_PAGE_BITS;
> +
>       return true;
>   }
>   

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 35/54] target/avr: Convert to TCGCPUOps.tlb_fill_align

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   target/avr/cpu.h    |  7 ++++---
>   target/avr/cpu.c    |  2 +-
>   target/avr/helper.c | 19 ++++++++++++-------
>   3 files changed, 17 insertions(+), 11 deletions(-)
> 
> diff --git a/target/avr/cpu.h b/target/avr/cpu.h
> index 4725535102..cdd3bcd418 100644
> --- a/target/avr/cpu.h
> +++ b/target/avr/cpu.h
> @@ -23,6 +23,7 @@
>   
>   #include "cpu-qom.h"
>   #include "exec/cpu-defs.h"
> +#include "exec/memop.h"
>   
>   #ifdef CONFIG_USER_ONLY
>   #error "AVR 8-bit does not support user mode"
> @@ -238,9 +239,9 @@ static inline void cpu_set_sreg(CPUAVRState *env, uint8_t sreg)
>       env->sregI = (sreg >> 7) & 0x01;
>   }
>   
> -bool avr_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                      MMUAccessType access_type, int mmu_idx,
> -                      bool probe, uintptr_t retaddr);
> +bool avr_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out, vaddr addr,
> +                            MMUAccessType access_type, int mmu_idx,
> +                            MemOp memop, int size, bool probe, uintptr_t ra);
>   
>   #include "exec/cpu-all.h"
>   
> diff --git a/target/avr/cpu.c b/target/avr/cpu.c
> index 3132842d56..a7fe869396 100644
> --- a/target/avr/cpu.c
> +++ b/target/avr/cpu.c
> @@ -211,7 +211,7 @@ static const TCGCPUOps avr_tcg_ops = {
>       .restore_state_to_opc = avr_restore_state_to_opc,
>       .cpu_exec_interrupt = avr_cpu_exec_interrupt,
>       .cpu_exec_halt = avr_cpu_has_work,
> -    .tlb_fill = avr_cpu_tlb_fill,
> +    .tlb_fill_align = avr_cpu_tlb_fill_align,
>       .do_interrupt = avr_cpu_do_interrupt,
>   };
>   
> diff --git a/target/avr/helper.c b/target/avr/helper.c
> index 345708a1b3..a18f11aa9f 100644
> --- a/target/avr/helper.c
> +++ b/target/avr/helper.c
> @@ -104,11 +104,11 @@ hwaddr avr_cpu_get_phys_page_debug(CPUState *cs, vaddr addr)
>       return addr; /* I assume 1:1 address correspondence */
>   }
>   
> -bool avr_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                      MMUAccessType access_type, int mmu_idx,
> -                      bool probe, uintptr_t retaddr)
> +bool avr_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out, vaddr address,
> +                            MMUAccessType access_type, int mmu_idx,
> +                            MemOp memop, int size, bool probe, uintptr_t ra)
>   {
> -    int prot, page_size = TARGET_PAGE_SIZE;
> +    int prot, lg_page_size = TARGET_PAGE_BITS;
>       uint32_t paddr;
>   
>       address &= TARGET_PAGE_MASK;
> @@ -141,15 +141,20 @@ bool avr_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
>                * to force tlb_fill to be called for the next access.
>                */
>               if (probe) {
> -                page_size = 1;
> +                lg_page_size = 0;
>               } else {
>                   cpu_env(cs)->fullacc = 1;
> -                cpu_loop_exit_restore(cs, retaddr);
> +                cpu_loop_exit_restore(cs, ra);
>               }
>           }
>       }
>   
> -    tlb_set_page(cs, address, paddr, prot, mmu_idx, page_size);
> +    memset(out, 0, sizeof(*out));
> +    out->phys_addr = paddr;
> +    out->prot = prot;
> +    out->attrs = MEMTXATTRS_UNSPECIFIED;
> +    out->lg_page_size = lg_page_size;
> +
>       return true;
>   }
>   

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 36/54] target/i386: Convert to TCGCPUOps.tlb_fill_align

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   target/i386/tcg/helper-tcg.h         |  6 +++---
>   target/i386/tcg/sysemu/excp_helper.c | 28 ++++++++++++++++------------
>   target/i386/tcg/tcg-cpu.c            |  2 +-
>   3 files changed, 20 insertions(+), 16 deletions(-)
> 
> diff --git a/target/i386/tcg/helper-tcg.h b/target/i386/tcg/helper-tcg.h
> index 696d6ef016..b2164f41e6 100644
> --- a/target/i386/tcg/helper-tcg.h
> +++ b/target/i386/tcg/helper-tcg.h
> @@ -79,9 +79,9 @@ void x86_cpu_record_sigsegv(CPUState *cs, vaddr addr,
>   void x86_cpu_record_sigbus(CPUState *cs, vaddr addr,
>                              MMUAccessType access_type, uintptr_t ra);
>   #else
> -bool x86_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                      MMUAccessType access_type, int mmu_idx,
> -                      bool probe, uintptr_t retaddr);
> +bool x86_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out, vaddr addr,
> +                            MMUAccessType access_type, int mmu_idx,
> +                            MemOp memop, int size, bool probe, uintptr_t ra);
>   G_NORETURN void x86_cpu_do_unaligned_access(CPUState *cs, vaddr vaddr,
>                                               MMUAccessType access_type,
>                                               int mmu_idx, uintptr_t retaddr);
> diff --git a/target/i386/tcg/sysemu/excp_helper.c b/target/i386/tcg/sysemu/excp_helper.c
> index 168ff8e5f3..d23d28fef5 100644
> --- a/target/i386/tcg/sysemu/excp_helper.c
> +++ b/target/i386/tcg/sysemu/excp_helper.c
> @@ -601,25 +601,29 @@ static bool get_physical_address(CPUX86State *env, vaddr addr,
>       return true;
>   }
>   
> -bool x86_cpu_tlb_fill(CPUState *cs, vaddr addr, int size,
> -                      MMUAccessType access_type, int mmu_idx,
> -                      bool probe, uintptr_t retaddr)
> +bool x86_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *full, vaddr addr,
> +                            MMUAccessType access_type, int mmu_idx,
> +                            MemOp memop, int size, bool probe,
> +                            uintptr_t retaddr)
>   {
>       CPUX86State *env = cpu_env(cs);
>       TranslateResult out;
>       TranslateFault err;
>   
> +    if (addr & ((1 << memop_alignment_bits(memop)) - 1)) {
> +        if (probe) {
> +            return false;
> +        }
> +        x86_cpu_do_unaligned_access(cs, addr, access_type, mmu_idx, retaddr);
> +    }
> +
>       if (get_physical_address(env, addr, access_type, mmu_idx, &out, &err,
>                                retaddr)) {
> -        /*
> -         * Even if 4MB pages, we map only one 4KB page in the cache to
> -         * avoid filling it too fast.
> -         */
> -        assert(out.prot & (1 << access_type));
> -        tlb_set_page_with_attrs(cs, addr & TARGET_PAGE_MASK,
> -                                out.paddr & TARGET_PAGE_MASK,
> -                                cpu_get_mem_attrs(env),
> -                                out.prot, mmu_idx, out.page_size);
> +        memset(full, 0, sizeof(*full));
> +        full->phys_addr = out.paddr;
> +        full->prot = out.prot;
> +        full->lg_page_size = ctz32(out.page_size);
> +        full->attrs = cpu_get_mem_attrs(env);
>           return true;
>       }
>   
> diff --git a/target/i386/tcg/tcg-cpu.c b/target/i386/tcg/tcg-cpu.c
> index cca19cd40e..6fce6227c7 100644
> --- a/target/i386/tcg/tcg-cpu.c
> +++ b/target/i386/tcg/tcg-cpu.c
> @@ -117,7 +117,7 @@ static const TCGCPUOps x86_tcg_ops = {
>       .record_sigsegv = x86_cpu_record_sigsegv,
>       .record_sigbus = x86_cpu_record_sigbus,
>   #else
> -    .tlb_fill = x86_cpu_tlb_fill,
> +    .tlb_fill_align = x86_cpu_tlb_fill_align,
>       .do_interrupt = x86_cpu_do_interrupt,
>       .cpu_exec_halt = x86_cpu_exec_halt,
>       .cpu_exec_interrupt = x86_cpu_exec_interrupt,

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 37/54] target/loongarch: Convert to TCGCPUOps.tlb_fill_align

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   target/loongarch/internals.h      |  7 ++++---
>   target/loongarch/cpu.c            |  2 +-
>   target/loongarch/tcg/tlb_helper.c | 17 +++++++++++------
>   3 files changed, 16 insertions(+), 10 deletions(-)
> 
> diff --git a/target/loongarch/internals.h b/target/loongarch/internals.h
> index 1a02427627..a9f73f27b2 100644
> --- a/target/loongarch/internals.h
> +++ b/target/loongarch/internals.h
> @@ -60,9 +60,10 @@ int get_physical_address(CPULoongArchState *env, hwaddr *physical,
>   hwaddr loongarch_cpu_get_phys_page_debug(CPUState *cpu, vaddr addr);
>   
>   #ifdef CONFIG_TCG
> -bool loongarch_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                            MMUAccessType access_type, int mmu_idx,
> -                            bool probe, uintptr_t retaddr);
> +bool loongarch_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
> +                                  vaddr addr, MMUAccessType access_type,
> +                                  int mmu_idx, MemOp memop, int size,
> +                                  bool probe, uintptr_t ra);
>   #endif
>   #endif /* !CONFIG_USER_ONLY */
>   
> diff --git a/target/loongarch/cpu.c b/target/loongarch/cpu.c
> index 57cc4f314b..47d69f1788 100644
> --- a/target/loongarch/cpu.c
> +++ b/target/loongarch/cpu.c
> @@ -798,7 +798,7 @@ static const TCGCPUOps loongarch_tcg_ops = {
>       .restore_state_to_opc = loongarch_restore_state_to_opc,
>   
>   #ifndef CONFIG_USER_ONLY
> -    .tlb_fill = loongarch_cpu_tlb_fill,
> +    .tlb_fill_align = loongarch_cpu_tlb_fill_align,
>       .cpu_exec_interrupt = loongarch_cpu_exec_interrupt,
>       .cpu_exec_halt = loongarch_cpu_has_work,
>       .do_interrupt = loongarch_cpu_do_interrupt,
> diff --git a/target/loongarch/tcg/tlb_helper.c b/target/loongarch/tcg/tlb_helper.c
> index 97f38fc391..94d5df08a4 100644
> --- a/target/loongarch/tcg/tlb_helper.c
> +++ b/target/loongarch/tcg/tlb_helper.c
> @@ -474,9 +474,10 @@ void helper_invtlb_page_asid_or_g(CPULoongArchState *env,
>       tlb_flush(env_cpu(env));
>   }
>   
> -bool loongarch_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                            MMUAccessType access_type, int mmu_idx,
> -                            bool probe, uintptr_t retaddr)
> +bool loongarch_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
> +                                  vaddr address, MMUAccessType access_type,
> +                                  int mmu_idx, MemOp memop, int size,
> +                                  bool probe, uintptr_t retaddr)
>   {
>       CPULoongArchState *env = cpu_env(cs);
>       hwaddr physical;
> @@ -488,12 +489,16 @@ bool loongarch_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
>                                  access_type, mmu_idx);
>   
>       if (ret == TLBRET_MATCH) {
> -        tlb_set_page(cs, address & TARGET_PAGE_MASK,
> -                     physical & TARGET_PAGE_MASK, prot,
> -                     mmu_idx, TARGET_PAGE_SIZE);
>           qemu_log_mask(CPU_LOG_MMU,
>                         "%s address=%" VADDR_PRIx " physical " HWADDR_FMT_plx
>                         " prot %d\n", __func__, address, physical, prot);
> +
> +        memset(out, 0, sizeof(*out));
> +        out->phys_addr = physical;
> +        out->prot = prot;
> +        out->attrs = MEMTXATTRS_UNSPECIFIED;
> +        out->lg_page_size = TARGET_PAGE_BITS;
> +
>           return true;
>       } else {
>           qemu_log_mask(CPU_LOG_MMU,

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 38/54] target/m68k: Convert to TCGCPUOps.tlb_fill_align

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   target/m68k/cpu.h    |  7 ++++---
>   target/m68k/cpu.c    |  2 +-
>   target/m68k/helper.c | 22 +++++++++++++---------
>   3 files changed, 18 insertions(+), 13 deletions(-)
> 
> diff --git a/target/m68k/cpu.h b/target/m68k/cpu.h
> index b5bbeedb7a..4401426a0b 100644
> --- a/target/m68k/cpu.h
> +++ b/target/m68k/cpu.h
> @@ -22,6 +22,7 @@
>   #define M68K_CPU_H
>   
>   #include "exec/cpu-defs.h"
> +#include "exec/memop.h"
>   #include "qemu/cpu-float.h"
>   #include "cpu-qom.h"
>   
> @@ -582,10 +583,10 @@ enum {
>   #define MMU_KERNEL_IDX 0
>   #define MMU_USER_IDX 1
>   
> -bool m68k_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                       MMUAccessType access_type, int mmu_idx,
> -                       bool probe, uintptr_t retaddr);
>   #ifndef CONFIG_USER_ONLY
> +bool m68k_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out, vaddr addr,
> +                             MMUAccessType access_type, int mmu_idx,
> +                             MemOp memop, int size, bool probe, uintptr_t ra);
>   void m68k_cpu_transaction_failed(CPUState *cs, hwaddr physaddr, vaddr addr,
>                                    unsigned size, MMUAccessType access_type,
>                                    int mmu_idx, MemTxAttrs attrs,
> diff --git a/target/m68k/cpu.c b/target/m68k/cpu.c
> index 5fe335558a..5316cf8922 100644
> --- a/target/m68k/cpu.c
> +++ b/target/m68k/cpu.c
> @@ -550,7 +550,7 @@ static const TCGCPUOps m68k_tcg_ops = {
>       .restore_state_to_opc = m68k_restore_state_to_opc,
>   
>   #ifndef CONFIG_USER_ONLY
> -    .tlb_fill = m68k_cpu_tlb_fill,
> +    .tlb_fill_align = m68k_cpu_tlb_fill_align,
>       .cpu_exec_interrupt = m68k_cpu_exec_interrupt,
>       .cpu_exec_halt = m68k_cpu_has_work,
>       .do_interrupt = m68k_cpu_do_interrupt,
> diff --git a/target/m68k/helper.c b/target/m68k/helper.c
> index 9bfc6ae97c..1decb6f39c 100644
> --- a/target/m68k/helper.c
> +++ b/target/m68k/helper.c
> @@ -950,9 +950,10 @@ void m68k_set_irq_level(M68kCPU *cpu, int level, uint8_t vector)
>       }
>   }
>   
> -bool m68k_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                       MMUAccessType qemu_access_type, int mmu_idx,
> -                       bool probe, uintptr_t retaddr)
> +bool m68k_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
> +                             vaddr address, MMUAccessType qemu_access_type,
> +                             int mmu_idx, MemOp memop, int size,
> +                             bool probe, uintptr_t retaddr)
>   {
>       CPUM68KState *env = cpu_env(cs);
>       hwaddr physical;
> @@ -961,12 +962,14 @@ bool m68k_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
>       int ret;
>       target_ulong page_size;
>   
> +    memset(out, 0, sizeof(*out));
> +    out->attrs = MEMTXATTRS_UNSPECIFIED;
> +
>       if ((env->mmu.tcr & M68K_TCR_ENABLED) == 0) {
>           /* MMU disabled */
> -        tlb_set_page(cs, address & TARGET_PAGE_MASK,
> -                     address & TARGET_PAGE_MASK,
> -                     PAGE_READ | PAGE_WRITE | PAGE_EXEC,
> -                     mmu_idx, TARGET_PAGE_SIZE);
> +        out->phys_addr = address;
> +        out->prot = PAGE_READ | PAGE_WRITE | PAGE_EXEC;
> +        out->lg_page_size = TARGET_PAGE_BITS;
>           return true;
>       }
>   
> @@ -985,8 +988,9 @@ bool m68k_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
>       ret = get_physical_address(env, &physical, &prot,
>                                  address, access_type, &page_size);
>       if (likely(ret == 0)) {
> -        tlb_set_page(cs, address & TARGET_PAGE_MASK,
> -                     physical & TARGET_PAGE_MASK, prot, mmu_idx, page_size);
> +        out->phys_addr = physical;
> +        out->prot = prot;
> +        out->lg_page_size = ctz32(page_size);
>           return true;
>       }
>   

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 39/54] target/m68k: Do not call tlb_set_page in helper_ptest

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> The entire operation of ptest is performed within
> get_physical_address as part of ACCESS_PTEST.
> There is no need to install the page into softmmu.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   target/m68k/helper.c | 10 +---------
>   1 file changed, 1 insertion(+), 9 deletions(-)
> 
> diff --git a/target/m68k/helper.c b/target/m68k/helper.c
> index 1decb6f39c..0a54eca9bb 100644
> --- a/target/m68k/helper.c
> +++ b/target/m68k/helper.c
> @@ -1460,7 +1460,6 @@ void HELPER(ptest)(CPUM68KState *env, uint32_t addr, uint32_t is_read)
>       hwaddr physical;
>       int access_type;
>       int prot;
> -    int ret;
>       target_ulong page_size;
>   
>       access_type = ACCESS_PTEST;
> @@ -1476,14 +1475,7 @@ void HELPER(ptest)(CPUM68KState *env, uint32_t addr, uint32_t is_read)
>   
>       env->mmu.mmusr = 0;
>       env->mmu.ssw = 0;
> -    ret = get_physical_address(env, &physical, &prot, addr,
> -                               access_type, &page_size);
> -    if (ret == 0) {
> -        tlb_set_page(env_cpu(env), addr & TARGET_PAGE_MASK,
> -                     physical & TARGET_PAGE_MASK,
> -                     prot, access_type & ACCESS_SUPER ?
> -                     MMU_KERNEL_IDX : MMU_USER_IDX, page_size);
> -    }
> +    get_physical_address(env, &physical, &prot, addr, access_type, &page_size);
>   }
>   
>   void HELPER(pflush)(CPUM68KState *env, uint32_t addr, uint32_t opmode)

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 40/54] target/microblaze: Convert to TCGCPUOps.tlb_fill_align

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   target/microblaze/cpu.h    |  7 +++----
>   target/microblaze/cpu.c    |  2 +-
>   target/microblaze/helper.c | 33 ++++++++++++++++++++-------------
>   3 files changed, 24 insertions(+), 18 deletions(-)
> 
> diff --git a/target/microblaze/cpu.h b/target/microblaze/cpu.h
> index 3e5a3e5c60..b0eadfd9b1 100644
> --- a/target/microblaze/cpu.h
> +++ b/target/microblaze/cpu.h
> @@ -421,10 +421,9 @@ static inline void cpu_get_tb_cpu_state(CPUMBState *env, vaddr *pc,
>   }
>   
>   #if !defined(CONFIG_USER_ONLY)
> -bool mb_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                     MMUAccessType access_type, int mmu_idx,
> -                     bool probe, uintptr_t retaddr);
> -
> +bool mb_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out, vaddr address,
> +                            MMUAccessType access_type, int mmu_idx,
> +                           MemOp memop, int size, bool probe, uintptr_t ra);
>   void mb_cpu_transaction_failed(CPUState *cs, hwaddr physaddr, vaddr addr,
>                                  unsigned size, MMUAccessType access_type,
>                                  int mmu_idx, MemTxAttrs attrs,
> diff --git a/target/microblaze/cpu.c b/target/microblaze/cpu.c
> index 710eb1146c..212cad2143 100644
> --- a/target/microblaze/cpu.c
> +++ b/target/microblaze/cpu.c
> @@ -425,7 +425,7 @@ static const TCGCPUOps mb_tcg_ops = {
>       .restore_state_to_opc = mb_restore_state_to_opc,
>   
>   #ifndef CONFIG_USER_ONLY
> -    .tlb_fill = mb_cpu_tlb_fill,
> +    .tlb_fill_align = mb_cpu_tlb_fill_align,
>       .cpu_exec_interrupt = mb_cpu_exec_interrupt,
>       .cpu_exec_halt = mb_cpu_has_work,
>       .do_interrupt = mb_cpu_do_interrupt,
> diff --git a/target/microblaze/helper.c b/target/microblaze/helper.c
> index 5d3259ce31..b6375564b4 100644
> --- a/target/microblaze/helper.c
> +++ b/target/microblaze/helper.c
> @@ -36,37 +36,44 @@ static bool mb_cpu_access_is_secure(MicroBlazeCPU *cpu,
>       }
>   }
>   
> -bool mb_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                     MMUAccessType access_type, int mmu_idx,
> -                     bool probe, uintptr_t retaddr)
> +bool mb_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out, vaddr address,
> +                           MMUAccessType access_type, int mmu_idx,
> +                           MemOp memop, int size,
> +                           bool probe, uintptr_t retaddr)
>   {
>       MicroBlazeCPU *cpu = MICROBLAZE_CPU(cs);
>       CPUMBState *env = &cpu->env;
>       MicroBlazeMMULookup lu;
>       unsigned int hit;
> -    int prot;
> -    MemTxAttrs attrs = {};
>   
> -    attrs.secure = mb_cpu_access_is_secure(cpu, access_type);
> +    if (address & ((1 << memop_alignment_bits(memop)) - 1)) {
> +        if (probe) {
> +            return false;
> +        }
> +        mb_cpu_do_unaligned_access(cs, address, access_type, mmu_idx, retaddr);
> +    }
> +
> +    memset(out, 0, sizeof(*out));
> +    out->attrs.secure = mb_cpu_access_is_secure(cpu, access_type);
> +    out->lg_page_size = TARGET_PAGE_BITS;
>   
>       if (mmu_idx == MMU_NOMMU_IDX) {
>           /* MMU disabled or not available.  */
> -        address &= TARGET_PAGE_MASK;
> -        prot = PAGE_RWX;
> -        tlb_set_page_with_attrs(cs, address, address, attrs, prot, mmu_idx,
> -                                TARGET_PAGE_SIZE);
> +        out->phys_addr = address;
> +        out->prot = PAGE_RWX;
>           return true;
>       }
>   
>       hit = mmu_translate(cpu, &lu, address, access_type, mmu_idx);
>       if (likely(hit)) {
> -        uint32_t vaddr = address & TARGET_PAGE_MASK;
> +        uint32_t vaddr = address;
>           uint32_t paddr = lu.paddr + vaddr - lu.vaddr;
>   
>           qemu_log_mask(CPU_LOG_MMU, "MMU map mmu=%d v=%x p=%x prot=%x\n",
>                         mmu_idx, vaddr, paddr, lu.prot);
> -        tlb_set_page_with_attrs(cs, vaddr, paddr, attrs, lu.prot, mmu_idx,
> -                                TARGET_PAGE_SIZE);
> +
> +        out->phys_addr = paddr;
> +        out->prot = lu.prot;
>           return true;
>       }
>   

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 41/54] target/mips: Convert to TCGCPUOps.tlb_fill_align

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   target/mips/tcg/tcg-internal.h      |  6 +++---
>   target/mips/cpu.c                   |  2 +-
>   target/mips/tcg/sysemu/tlb_helper.c | 29 ++++++++++++++++++++---------
>   3 files changed, 24 insertions(+), 13 deletions(-)
> 
> diff --git a/target/mips/tcg/tcg-internal.h b/target/mips/tcg/tcg-internal.h
> index aef032c48d..f4b00354af 100644
> --- a/target/mips/tcg/tcg-internal.h
> +++ b/target/mips/tcg/tcg-internal.h
> @@ -61,9 +61,9 @@ void mips_cpu_do_transaction_failed(CPUState *cs, hwaddr physaddr,
>                                       MemTxResult response, uintptr_t retaddr);
>   void cpu_mips_tlb_flush(CPUMIPSState *env);
>   
> -bool mips_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                       MMUAccessType access_type, int mmu_idx,
> -                       bool probe, uintptr_t retaddr);
> +bool mips_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out, vaddr address,
> +                             MMUAccessType access_type, int mmu_idx,
> +                             MemOp memop, int size, bool probe, uintptr_t ra);
>   
>   void mips_semihosting(CPUMIPSState *env);
>   
> diff --git a/target/mips/cpu.c b/target/mips/cpu.c
> index d0a43b6d5c..3a453c9285 100644
> --- a/target/mips/cpu.c
> +++ b/target/mips/cpu.c
> @@ -556,7 +556,7 @@ static const TCGCPUOps mips_tcg_ops = {
>       .restore_state_to_opc = mips_restore_state_to_opc,
>   
>   #if !defined(CONFIG_USER_ONLY)
> -    .tlb_fill = mips_cpu_tlb_fill,
> +    .tlb_fill_align = mips_cpu_tlb_fill_align,
>       .cpu_exec_interrupt = mips_cpu_exec_interrupt,
>       .cpu_exec_halt = mips_cpu_has_work,
>       .do_interrupt = mips_cpu_do_interrupt,
> diff --git a/target/mips/tcg/sysemu/tlb_helper.c b/target/mips/tcg/sysemu/tlb_helper.c
> index e98bb95951..ac76396525 100644
> --- a/target/mips/tcg/sysemu/tlb_helper.c
> +++ b/target/mips/tcg/sysemu/tlb_helper.c
> @@ -904,15 +904,28 @@ refill:
>   }
>   #endif
>   
> -bool mips_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                       MMUAccessType access_type, int mmu_idx,
> -                       bool probe, uintptr_t retaddr)
> +bool mips_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out, vaddr address,
> +                             MMUAccessType access_type, int mmu_idx,
> +                             MemOp memop, int size,
> +                             bool probe, uintptr_t retaddr)
>   {
>       CPUMIPSState *env = cpu_env(cs);
>       hwaddr physical;
>       int prot;
>       int ret = TLBRET_BADADDR;
>   
> +    if (address & ((1 << memop_alignment_bits(memop)) - 1)) {
> +        if (probe) {
> +            return false;
> +        }
> +        mips_cpu_do_unaligned_access(cs, address, access_type,
> +                                     mmu_idx, retaddr);
> +    }
> +
> +    memset(out, 0, sizeof(*out));
> +    out->attrs = MEMTXATTRS_UNSPECIFIED;
> +    out->lg_page_size = TARGET_PAGE_BITS;
> +
>       /* data access */
>       /* XXX: put correct access by using cpu_restore_state() correctly */
>       ret = get_physical_address(env, &physical, &prot, address,
> @@ -930,9 +943,8 @@ bool mips_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
>           break;
>       }
>       if (ret == TLBRET_MATCH) {
> -        tlb_set_page(cs, address & TARGET_PAGE_MASK,
> -                     physical & TARGET_PAGE_MASK, prot,
> -                     mmu_idx, TARGET_PAGE_SIZE);
> +        out->phys_addr = physical;
> +        out->prot = prot;
>           return true;
>       }
>   #if !defined(TARGET_MIPS64)
> @@ -948,9 +960,8 @@ bool mips_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
>               ret = get_physical_address(env, &physical, &prot, address,
>                                          access_type, mmu_idx);
>               if (ret == TLBRET_MATCH) {
> -                tlb_set_page(cs, address & TARGET_PAGE_MASK,
> -                             physical & TARGET_PAGE_MASK, prot,
> -                             mmu_idx, TARGET_PAGE_SIZE);
> +                out->phys_addr = physical;
> +                out->prot = prot;
>                   return true;
>               }
>           }

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 42/54] target/openrisc: Convert to TCGCPUOps.tlb_fill_align

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   target/openrisc/cpu.h |  8 +++++---
>   target/openrisc/cpu.c |  2 +-
>   target/openrisc/mmu.c | 39 +++++++++++++++++++++------------------
>   3 files changed, 27 insertions(+), 22 deletions(-)
> 
> diff --git a/target/openrisc/cpu.h b/target/openrisc/cpu.h
> index c9fe9ae12d..e177ad8b84 100644
> --- a/target/openrisc/cpu.h
> +++ b/target/openrisc/cpu.h
> @@ -22,6 +22,7 @@
>   
>   #include "cpu-qom.h"
>   #include "exec/cpu-defs.h"
> +#include "exec/memop.h"
>   #include "fpu/softfloat-types.h"
>   
>   /**
> @@ -306,9 +307,10 @@ int print_insn_or1k(bfd_vma addr, disassemble_info *info);
>   #ifndef CONFIG_USER_ONLY
>   hwaddr openrisc_cpu_get_phys_page_debug(CPUState *cpu, vaddr addr);
>   
> -bool openrisc_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                           MMUAccessType access_type, int mmu_idx,
> -                           bool probe, uintptr_t retaddr);
> +bool openrisc_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
> +                                 vaddr addr, MMUAccessType access_type,
> +                                 int mmu_idx, MemOp memop, int size,
> +                                 bool probe, uintptr_t ra);
>   
>   extern const VMStateDescription vmstate_openrisc_cpu;
>   
> diff --git a/target/openrisc/cpu.c b/target/openrisc/cpu.c
> index b96561d1f2..6aa04ff7d3 100644
> --- a/target/openrisc/cpu.c
> +++ b/target/openrisc/cpu.c
> @@ -237,7 +237,7 @@ static const TCGCPUOps openrisc_tcg_ops = {
>       .restore_state_to_opc = openrisc_restore_state_to_opc,
>   
>   #ifndef CONFIG_USER_ONLY
> -    .tlb_fill = openrisc_cpu_tlb_fill,
> +    .tlb_fill_align = openrisc_cpu_tlb_fill_align,
>       .cpu_exec_interrupt = openrisc_cpu_exec_interrupt,
>       .cpu_exec_halt = openrisc_cpu_has_work,
>       .do_interrupt = openrisc_cpu_do_interrupt,
> diff --git a/target/openrisc/mmu.c b/target/openrisc/mmu.c
> index c632d5230b..eafab356a6 100644
> --- a/target/openrisc/mmu.c
> +++ b/target/openrisc/mmu.c
> @@ -104,39 +104,42 @@ static void raise_mmu_exception(OpenRISCCPU *cpu, target_ulong address,
>       cpu->env.lock_addr = -1;
>   }
>   
> -bool openrisc_cpu_tlb_fill(CPUState *cs, vaddr addr, int size,
> -                           MMUAccessType access_type, int mmu_idx,
> -                           bool probe, uintptr_t retaddr)
> +bool openrisc_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
> +                                 vaddr addr, MMUAccessType access_type,
> +                                 int mmu_idx, MemOp memop, int size,
> +                                 bool probe, uintptr_t retaddr)
>   {
>       OpenRISCCPU *cpu = OPENRISC_CPU(cs);
> -    int excp = EXCP_DPF;
>       int prot;
>       hwaddr phys_addr;
>   
> +    /* TODO: alignment faults not currently handled. */
> +
>       if (mmu_idx == MMU_NOMMU_IDX) {
>           /* The mmu is disabled; lookups never fail.  */
>           get_phys_nommu(&phys_addr, &prot, addr);
> -        excp = 0;
>       } else {
>           bool super = mmu_idx == MMU_SUPERVISOR_IDX;
>           int need = (access_type == MMU_INST_FETCH ? PAGE_EXEC
>                       : access_type == MMU_DATA_STORE ? PAGE_WRITE
>                       : PAGE_READ);
> -        excp = get_phys_mmu(cpu, &phys_addr, &prot, addr, need, super);
> +        int excp = get_phys_mmu(cpu, &phys_addr, &prot, addr, need, super);
> +
> +        if (unlikely(excp)) {
> +            if (probe) {
> +                return false;
> +            }
> +            raise_mmu_exception(cpu, addr, excp);
> +            cpu_loop_exit_restore(cs, retaddr);
> +        }
>       }
>   
> -    if (likely(excp == 0)) {
> -        tlb_set_page(cs, addr & TARGET_PAGE_MASK,
> -                     phys_addr & TARGET_PAGE_MASK, prot,
> -                     mmu_idx, TARGET_PAGE_SIZE);
> -        return true;
> -    }
> -    if (probe) {
> -        return false;
> -    }
> -
> -    raise_mmu_exception(cpu, addr, excp);
> -    cpu_loop_exit_restore(cs, retaddr);
> +    memset(out, 0, sizeof(*out));
> +    out->phys_addr = phys_addr;
> +    out->prot = prot;
> +    out->lg_page_size = TARGET_PAGE_BITS;
> +    out->attrs = MEMTXATTRS_UNSPECIFIED;
> +    return true;
>   }
>   
>   hwaddr openrisc_cpu_get_phys_page_debug(CPUState *cs, vaddr addr)

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 43/54] target/ppc: Convert to TCGCPUOps.tlb_fill_align

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   target/ppc/internal.h   |  7 ++++---
>   target/ppc/cpu_init.c   |  2 +-
>   target/ppc/mmu_helper.c | 21 ++++++++++++++++-----
>   3 files changed, 21 insertions(+), 9 deletions(-)
> 
> diff --git a/target/ppc/internal.h b/target/ppc/internal.h
> index 20fb2ec593..9d132d35a1 100644
> --- a/target/ppc/internal.h
> +++ b/target/ppc/internal.h
> @@ -273,9 +273,10 @@ void ppc_cpu_record_sigsegv(CPUState *cs, vaddr addr,
>                               MMUAccessType access_type,
>                               bool maperr, uintptr_t ra);
>   #else
> -bool ppc_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                      MMUAccessType access_type, int mmu_idx,
> -                      bool probe, uintptr_t retaddr);
> +bool ppc_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
> +                            vaddr addr, MMUAccessType access_type,
> +                            int mmu_idx, MemOp memop, int size,
> +                            bool probe, uintptr_t ra);
>   G_NORETURN void ppc_cpu_do_unaligned_access(CPUState *cs, vaddr addr,
>                                               MMUAccessType access_type, int mmu_idx,
>                                               uintptr_t retaddr);
> diff --git a/target/ppc/cpu_init.c b/target/ppc/cpu_init.c
> index efcb80d1c2..387c7ff2da 100644
> --- a/target/ppc/cpu_init.c
> +++ b/target/ppc/cpu_init.c
> @@ -7422,7 +7422,7 @@ static const TCGCPUOps ppc_tcg_ops = {
>   #ifdef CONFIG_USER_ONLY
>     .record_sigsegv = ppc_cpu_record_sigsegv,
>   #else
> -  .tlb_fill = ppc_cpu_tlb_fill,
> +  .tlb_fill_align = ppc_cpu_tlb_fill_align,
>     .cpu_exec_interrupt = ppc_cpu_exec_interrupt,
>     .cpu_exec_halt = ppc_cpu_has_work,
>     .do_interrupt = ppc_cpu_do_interrupt,
> diff --git a/target/ppc/mmu_helper.c b/target/ppc/mmu_helper.c
> index b167b37e0a..bf98e0efb0 100644
> --- a/target/ppc/mmu_helper.c
> +++ b/target/ppc/mmu_helper.c
> @@ -1357,18 +1357,29 @@ void helper_check_tlb_flush_global(CPUPPCState *env)
>   }
>   
>   
> -bool ppc_cpu_tlb_fill(CPUState *cs, vaddr eaddr, int size,
> -                      MMUAccessType access_type, int mmu_idx,
> -                      bool probe, uintptr_t retaddr)
> +bool ppc_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
> +                            vaddr eaddr, MMUAccessType access_type,
> +                            int mmu_idx, MemOp memop, int size,
> +                            bool probe, uintptr_t retaddr)
>   {
>       PowerPCCPU *cpu = POWERPC_CPU(cs);
>       hwaddr raddr;
>       int page_size, prot;
>   
> +    if (eaddr & ((1 << memop_alignment_bits(memop)) - 1)) {
> +        if (probe) {
> +            return false;
> +        }
> +        ppc_cpu_do_unaligned_access(cs, eaddr, access_type, mmu_idx, retaddr);
> +    }
> +
>       if (ppc_xlate(cpu, eaddr, access_type, &raddr,
>                     &page_size, &prot, mmu_idx, !probe)) {
> -        tlb_set_page(cs, eaddr & TARGET_PAGE_MASK, raddr & TARGET_PAGE_MASK,
> -                     prot, mmu_idx, 1UL << page_size);
> +        memset(out, 0, sizeof(*out));
> +        out->phys_addr = raddr;
> +        out->prot = prot;
> +        out->lg_page_size = page_size;
> +        out->attrs = MEMTXATTRS_UNSPECIFIED;
>           return true;
>       }
>       if (probe) {

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 44/54] target/riscv: Convert to TCGCPUOps.tlb_fill_align

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   target/riscv/cpu.h         |  8 +++++---
>   target/riscv/cpu_helper.c  | 22 +++++++++++++++++-----
>   target/riscv/tcg/tcg-cpu.c |  2 +-
>   3 files changed, 23 insertions(+), 9 deletions(-)
> 
> diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h
> index 284b112821..f97c4f3410 100644
> --- a/target/riscv/cpu.h
> +++ b/target/riscv/cpu.h
> @@ -25,6 +25,7 @@
>   #include "hw/qdev-properties.h"
>   #include "exec/cpu-defs.h"
>   #include "exec/gdbstub.h"
> +#include "exec/memop.h"
>   #include "qemu/cpu-float.h"
>   #include "qom/object.h"
>   #include "qemu/int128.h"
> @@ -563,9 +564,10 @@ bool cpu_get_bcfien(CPURISCVState *env);
>   G_NORETURN void  riscv_cpu_do_unaligned_access(CPUState *cs, vaddr addr,
>                                                  MMUAccessType access_type,
>                                                  int mmu_idx, uintptr_t retaddr);
> -bool riscv_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                        MMUAccessType access_type, int mmu_idx,
> -                        bool probe, uintptr_t retaddr);
> +bool riscv_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
> +                              vaddr addr, MMUAccessType access_type,
> +                              int mmu_idx, MemOp memop, int size,
> +                              bool probe, uintptr_t ra);
>   char *riscv_isa_string(RISCVCPU *cpu);
>   int riscv_cpu_max_xlen(RISCVCPUClass *mcc);
>   bool riscv_cpu_option_set(const char *optname);
> diff --git a/target/riscv/cpu_helper.c b/target/riscv/cpu_helper.c
> index 0a3ead69ea..edb2edfc55 100644
> --- a/target/riscv/cpu_helper.c
> +++ b/target/riscv/cpu_helper.c
> @@ -1429,9 +1429,10 @@ static void pmu_tlb_fill_incr_ctr(RISCVCPU *cpu, MMUAccessType access_type)
>       riscv_pmu_incr_ctr(cpu, pmu_event_type);
>   }
>   
> -bool riscv_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                        MMUAccessType access_type, int mmu_idx,
> -                        bool probe, uintptr_t retaddr)
> +bool riscv_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
> +                              vaddr address, MMUAccessType access_type,
> +                              int mmu_idx, MemOp memop, int size,
> +                              bool probe, uintptr_t retaddr)
>   {
>       RISCVCPU *cpu = RISCV_CPU(cs);
>       CPURISCVState *env = &cpu->env;
> @@ -1452,6 +1453,14 @@ bool riscv_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
>       qemu_log_mask(CPU_LOG_MMU, "%s ad %" VADDR_PRIx " rw %d mmu_idx %d\n",
>                     __func__, address, access_type, mmu_idx);
>   
> +    if (address & ((1 << memop_alignment_bits(memop)) - 1)) {
> +        if (probe) {
> +            return false;
> +        }
> +        riscv_cpu_do_unaligned_access(cs, address, access_type,
> +                                      mmu_idx, retaddr);
> +    }
> +
>       pmu_tlb_fill_incr_ctr(cpu, access_type);
>       if (two_stage_lookup) {
>           /* Two stage lookup */
> @@ -1544,8 +1553,11 @@ bool riscv_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
>       }
>   
>       if (ret == TRANSLATE_SUCCESS) {
> -        tlb_set_page(cs, address & ~(tlb_size - 1), pa & ~(tlb_size - 1),
> -                     prot, mmu_idx, tlb_size);
> +        memset(out, 0, sizeof(*out));
> +        out->phys_addr = pa;
> +        out->prot = prot;
> +        out->lg_page_size = ctz64(tlb_size);
> +        out->attrs = MEMTXATTRS_UNSPECIFIED;
>           return true;
>       } else if (probe) {
>           return false;
> diff --git a/target/riscv/tcg/tcg-cpu.c b/target/riscv/tcg/tcg-cpu.c
> index c62c221696..f3b436bb86 100644
> --- a/target/riscv/tcg/tcg-cpu.c
> +++ b/target/riscv/tcg/tcg-cpu.c
> @@ -138,7 +138,7 @@ static const TCGCPUOps riscv_tcg_ops = {
>       .restore_state_to_opc = riscv_restore_state_to_opc,
>   
>   #ifndef CONFIG_USER_ONLY
> -    .tlb_fill = riscv_cpu_tlb_fill,
> +    .tlb_fill_align = riscv_cpu_tlb_fill_align,
>       .cpu_exec_interrupt = riscv_cpu_exec_interrupt,
>       .cpu_exec_halt = riscv_cpu_has_work,
>       .do_interrupt = riscv_cpu_do_interrupt,

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 45/54] target/rx: Convert to TCGCPUOps.tlb_fill_align

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   target/rx/cpu.c | 19 +++++++++++--------
>   1 file changed, 11 insertions(+), 8 deletions(-)
> 
> diff --git a/target/rx/cpu.c b/target/rx/cpu.c
> index 65a74ce720..c83a582141 100644
> --- a/target/rx/cpu.c
> +++ b/target/rx/cpu.c
> @@ -161,16 +161,19 @@ static void rx_cpu_disas_set_info(CPUState *cpu, disassemble_info *info)
>       info->print_insn = print_insn_rx;
>   }
>   
> -static bool rx_cpu_tlb_fill(CPUState *cs, vaddr addr, int size,
> -                            MMUAccessType access_type, int mmu_idx,
> -                            bool probe, uintptr_t retaddr)
> +static bool rx_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
> +                                  vaddr addr, MMUAccessType access_type,
> +                                  int mmu_idx, MemOp memop, int size,
> +                                  bool probe, uintptr_t retaddr)
>   {
> -    uint32_t address, physical, prot;
> +    /* TODO: alignment faults not currently handled. */
>   
>       /* Linear mapping */
> -    address = physical = addr & TARGET_PAGE_MASK;
> -    prot = PAGE_READ | PAGE_WRITE | PAGE_EXEC;
> -    tlb_set_page(cs, address, physical, prot, mmu_idx, TARGET_PAGE_SIZE);
> +    memset(out, 0, sizeof(*out));
> +    out->phys_addr = addr;
> +    out->prot = PAGE_READ | PAGE_WRITE | PAGE_EXEC;
> +    out->lg_page_size = TARGET_PAGE_BITS;
> +    out->attrs = MEMTXATTRS_UNSPECIFIED;
>       return true;
>   }
>   
> @@ -195,7 +198,7 @@ static const TCGCPUOps rx_tcg_ops = {
>       .initialize = rx_translate_init,
>       .synchronize_from_tb = rx_cpu_synchronize_from_tb,
>       .restore_state_to_opc = rx_restore_state_to_opc,
> -    .tlb_fill = rx_cpu_tlb_fill,
> +    .tlb_fill_align = rx_cpu_tlb_fill_align,
>   
>   #ifndef CONFIG_USER_ONLY
>       .cpu_exec_interrupt = rx_cpu_exec_interrupt,

Re: [PATCH v2 45/54] target/rx: Convert to TCGCPUOps.tlb_fill_align

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   target/rx/cpu.c | 19 +++++++++++--------
>   1 file changed, 11 insertions(+), 8 deletions(-)
> 
> diff --git a/target/rx/cpu.c b/target/rx/cpu.c
> index 65a74ce720..c83a582141 100644
> --- a/target/rx/cpu.c
> +++ b/target/rx/cpu.c
> @@ -161,16 +161,19 @@ static void rx_cpu_disas_set_info(CPUState *cpu, disassemble_info *info)
>       info->print_insn = print_insn_rx;
>   }
>   
> -static bool rx_cpu_tlb_fill(CPUState *cs, vaddr addr, int size,
> -                            MMUAccessType access_type, int mmu_idx,
> -                            bool probe, uintptr_t retaddr)
> +static bool rx_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
> +                                  vaddr addr, MMUAccessType access_type,
> +                                  int mmu_idx, MemOp memop, int size,
> +                                  bool probe, uintptr_t retaddr)
>   {
> -    uint32_t address, physical, prot;
> +    /* TODO: alignment faults not currently handled. */
>   
>       /* Linear mapping */
> -    address = physical = addr & TARGET_PAGE_MASK;
> -    prot = PAGE_READ | PAGE_WRITE | PAGE_EXEC;
> -    tlb_set_page(cs, address, physical, prot, mmu_idx, TARGET_PAGE_SIZE);
> +    memset(out, 0, sizeof(*out));
> +    out->phys_addr = addr;
> +    out->prot = PAGE_READ | PAGE_WRITE | PAGE_EXEC;
> +    out->lg_page_size = TARGET_PAGE_BITS;
> +    out->attrs = MEMTXATTRS_UNSPECIFIED;
>       return true;
>   }
>   
> @@ -195,7 +198,7 @@ static const TCGCPUOps rx_tcg_ops = {
>       .initialize = rx_translate_init,
>       .synchronize_from_tb = rx_cpu_synchronize_from_tb,
>       .restore_state_to_opc = rx_restore_state_to_opc,
> -    .tlb_fill = rx_cpu_tlb_fill,
> +    .tlb_fill_align = rx_cpu_tlb_fill_align,
>   
>   #ifndef CONFIG_USER_ONLY
>       .cpu_exec_interrupt = rx_cpu_exec_interrupt,

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 46/54] target/s390x: Convert to TCGCPUOps.tlb_fill_align

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   target/s390x/s390x-internal.h  |  7 ++++---
>   target/s390x/cpu.c             |  4 ++--
>   target/s390x/tcg/excp_helper.c | 23 ++++++++++++++++++-----
>   3 files changed, 24 insertions(+), 10 deletions(-)
> 
> diff --git a/target/s390x/s390x-internal.h b/target/s390x/s390x-internal.h
> index 825252d728..eb6fe24c9a 100644
> --- a/target/s390x/s390x-internal.h
> +++ b/target/s390x/s390x-internal.h
> @@ -278,9 +278,10 @@ void s390_cpu_record_sigsegv(CPUState *cs, vaddr address,
>   void s390_cpu_record_sigbus(CPUState *cs, vaddr address,
>                               MMUAccessType access_type, uintptr_t retaddr);
>   #else
> -bool s390_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                       MMUAccessType access_type, int mmu_idx,
> -                       bool probe, uintptr_t retaddr);
> +bool s390x_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
> +                              vaddr addr, MMUAccessType access_type,
> +                              int mmu_idx, MemOp memop, int size,
> +                              bool probe, uintptr_t retaddr);
>   G_NORETURN void s390x_cpu_do_unaligned_access(CPUState *cs, vaddr addr,
>                                                 MMUAccessType access_type, int mmu_idx,
>                                                 uintptr_t retaddr);
> diff --git a/target/s390x/cpu.c b/target/s390x/cpu.c
> index 514c70f301..4d0eb129e3 100644
> --- a/target/s390x/cpu.c
> +++ b/target/s390x/cpu.c
> @@ -330,7 +330,7 @@ void cpu_get_tb_cpu_state(CPUS390XState *env, vaddr *pc,
>            * Instructions must be at even addresses.
>            * This needs to be checked before address translation.
>            */
> -        env->int_pgm_ilen = 2; /* see s390_cpu_tlb_fill() */
> +        env->int_pgm_ilen = 2; /* see s390x_cpu_tlb_fill_align() */
>           tcg_s390_program_interrupt(env, PGM_SPECIFICATION, 0);
>       }
>   
> @@ -364,7 +364,7 @@ static const TCGCPUOps s390_tcg_ops = {
>       .record_sigsegv = s390_cpu_record_sigsegv,
>       .record_sigbus = s390_cpu_record_sigbus,
>   #else
> -    .tlb_fill = s390_cpu_tlb_fill,
> +    .tlb_fill_align = s390x_cpu_tlb_fill_align,
>       .cpu_exec_interrupt = s390_cpu_exec_interrupt,
>       .cpu_exec_halt = s390_cpu_has_work,
>       .do_interrupt = s390_cpu_do_interrupt,
> diff --git a/target/s390x/tcg/excp_helper.c b/target/s390x/tcg/excp_helper.c
> index 4c0b692c9e..6d61032a4a 100644
> --- a/target/s390x/tcg/excp_helper.c
> +++ b/target/s390x/tcg/excp_helper.c
> @@ -139,9 +139,10 @@ static inline uint64_t cpu_mmu_idx_to_asc(int mmu_idx)
>       }
>   }
>   
> -bool s390_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                       MMUAccessType access_type, int mmu_idx,
> -                       bool probe, uintptr_t retaddr)
> +bool s390x_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
> +                              vaddr address, MMUAccessType access_type,
> +                              int mmu_idx, MemOp memop, int size,
> +                              bool probe, uintptr_t retaddr)
>   {
>       CPUS390XState *env = cpu_env(cs);
>       target_ulong vaddr, raddr;
> @@ -151,6 +152,14 @@ bool s390_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
>       qemu_log_mask(CPU_LOG_MMU, "%s: addr 0x%" VADDR_PRIx " rw %d mmu_idx %d\n",
>                     __func__, address, access_type, mmu_idx);
>   
> +    if (address & ((1 << memop_alignment_bits(memop)) - 1)) {
> +        if (probe) {
> +            return false;
> +        }
> +        s390x_cpu_do_unaligned_access(cs, address, access_type,
> +                                      mmu_idx, retaddr);
> +    }
> +
>       vaddr = address;
>   
>       if (mmu_idx < MMU_REAL_IDX) {
> @@ -177,8 +186,12 @@ bool s390_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
>           qemu_log_mask(CPU_LOG_MMU,
>                         "%s: set tlb %" PRIx64 " -> %" PRIx64 " (%x)\n",
>                         __func__, (uint64_t)vaddr, (uint64_t)raddr, prot);
> -        tlb_set_page(cs, address & TARGET_PAGE_MASK, raddr, prot,
> -                     mmu_idx, TARGET_PAGE_SIZE);
> +
> +        memset(out, 0, sizeof(*out));
> +        out->phys_addr = raddr;
> +        out->prot = prot;
> +        out->lg_page_size = TARGET_PAGE_BITS;
> +        out->attrs = MEMTXATTRS_UNSPECIFIED;
>           return true;
>       }
>       if (probe) {

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 47/54] target/sh4: Convert to TCGCPUOps.tlb_fill_align

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   target/sh4/cpu.h    |  8 +++++---
>   target/sh4/cpu.c    |  2 +-
>   target/sh4/helper.c | 24 +++++++++++++++++-------
>   3 files changed, 23 insertions(+), 11 deletions(-)
> 
> diff --git a/target/sh4/cpu.h b/target/sh4/cpu.h
> index d928bcf006..161efdefcf 100644
> --- a/target/sh4/cpu.h
> +++ b/target/sh4/cpu.h
> @@ -22,6 +22,7 @@
>   
>   #include "cpu-qom.h"
>   #include "exec/cpu-defs.h"
> +#include "exec/memop.h"
>   #include "qemu/cpu-float.h"
>   
>   /* CPU Subtypes */
> @@ -251,9 +252,10 @@ void sh4_translate_init(void);
>   
>   #if !defined(CONFIG_USER_ONLY)
>   hwaddr superh_cpu_get_phys_page_debug(CPUState *cpu, vaddr addr);
> -bool superh_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                         MMUAccessType access_type, int mmu_idx,
> -                         bool probe, uintptr_t retaddr);
> +bool superh_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
> +                               vaddr addr, MMUAccessType access_type,
> +                               int mmu_idx, MemOp memop, int size,
> +                               bool probe, uintptr_t retaddr);
>   void superh_cpu_do_interrupt(CPUState *cpu);
>   bool superh_cpu_exec_interrupt(CPUState *cpu, int int_req);
>   void cpu_sh4_invalidate_tlb(CPUSH4State *s);
> diff --git a/target/sh4/cpu.c b/target/sh4/cpu.c
> index 8f07261dcf..8ca8b90e3c 100644
> --- a/target/sh4/cpu.c
> +++ b/target/sh4/cpu.c
> @@ -252,7 +252,7 @@ static const TCGCPUOps superh_tcg_ops = {
>       .restore_state_to_opc = superh_restore_state_to_opc,
>   
>   #ifndef CONFIG_USER_ONLY
> -    .tlb_fill = superh_cpu_tlb_fill,
> +    .tlb_fill_align = superh_cpu_tlb_fill_align,
>       .cpu_exec_interrupt = superh_cpu_exec_interrupt,
>       .cpu_exec_halt = superh_cpu_has_work,
>       .do_interrupt = superh_cpu_do_interrupt,
> diff --git a/target/sh4/helper.c b/target/sh4/helper.c
> index 9659c69550..543ac1b843 100644
> --- a/target/sh4/helper.c
> +++ b/target/sh4/helper.c
> @@ -792,22 +792,32 @@ bool superh_cpu_exec_interrupt(CPUState *cs, int interrupt_request)
>       return false;
>   }
>   
> -bool superh_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                         MMUAccessType access_type, int mmu_idx,
> -                         bool probe, uintptr_t retaddr)
> +bool superh_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
> +                               vaddr address, MMUAccessType access_type,
> +                               int mmu_idx, MemOp memop, int size,
> +                               bool probe, uintptr_t retaddr)
>   {
>       CPUSH4State *env = cpu_env(cs);
>       int ret;
> -
>       target_ulong physical;
>       int prot;
>   
> +    if (address & ((1 << memop_alignment_bits(memop)) - 1)) {
> +        if (probe) {
> +            return false;
> +        }
> +        superh_cpu_do_unaligned_access(cs, address, access_type,
> +                                       mmu_idx, retaddr);
> +    }
> +
>       ret = get_physical_address(env, &physical, &prot, address, access_type);
>   
>       if (ret == MMU_OK) {
> -        address &= TARGET_PAGE_MASK;
> -        physical &= TARGET_PAGE_MASK;
> -        tlb_set_page(cs, address, physical, prot, mmu_idx, TARGET_PAGE_SIZE);
> +        memset(out, 0, sizeof(*out));
> +        out->phys_addr = physical;
> +        out->prot = prot;
> +        out->lg_page_size = TARGET_PAGE_BITS;
> +        out->attrs = MEMTXATTRS_UNSPECIFIED;
>           return true;
>       }
>       if (probe) {

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 48/54] target/sparc: Convert to TCGCPUOps.tlb_fill_align

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   target/sparc/cpu.h        |  8 ++++---
>   target/sparc/cpu.c        |  2 +-
>   target/sparc/mmu_helper.c | 44 +++++++++++++++++++++++++--------------
>   3 files changed, 34 insertions(+), 20 deletions(-)
> 
> diff --git a/target/sparc/cpu.h b/target/sparc/cpu.h
> index f517e5a383..4c8927e9fa 100644
> --- a/target/sparc/cpu.h
> +++ b/target/sparc/cpu.h
> @@ -4,6 +4,7 @@
>   #include "qemu/bswap.h"
>   #include "cpu-qom.h"
>   #include "exec/cpu-defs.h"
> +#include "exec/memop.h"
>   #include "qemu/cpu-float.h"
>   
>   #if !defined(TARGET_SPARC64)
> @@ -596,9 +597,10 @@ G_NORETURN void cpu_raise_exception_ra(CPUSPARCState *, int, uintptr_t);
>   void cpu_sparc_set_id(CPUSPARCState *env, unsigned int cpu);
>   void sparc_cpu_list(void);
>   /* mmu_helper.c */
> -bool sparc_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                        MMUAccessType access_type, int mmu_idx,
> -                        bool probe, uintptr_t retaddr);
> +bool sparc_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
> +                              vaddr addr, MMUAccessType access_type,
> +                              int mmu_idx, MemOp memop, int size,
> +                              bool probe, uintptr_t retaddr);
>   target_ulong mmu_probe(CPUSPARCState *env, target_ulong address, int mmulev);
>   void dump_mmu(CPUSPARCState *env);
>   
> diff --git a/target/sparc/cpu.c b/target/sparc/cpu.c
> index dd7af86de7..57ae53bd71 100644
> --- a/target/sparc/cpu.c
> +++ b/target/sparc/cpu.c
> @@ -932,7 +932,7 @@ static const TCGCPUOps sparc_tcg_ops = {
>       .restore_state_to_opc = sparc_restore_state_to_opc,
>   
>   #ifndef CONFIG_USER_ONLY
> -    .tlb_fill = sparc_cpu_tlb_fill,
> +    .tlb_fill_align = sparc_cpu_tlb_fill_align,
>       .cpu_exec_interrupt = sparc_cpu_exec_interrupt,
>       .cpu_exec_halt = sparc_cpu_has_work,
>       .do_interrupt = sparc_cpu_do_interrupt,
> diff --git a/target/sparc/mmu_helper.c b/target/sparc/mmu_helper.c
> index 9ff06026b8..32766a37d6 100644
> --- a/target/sparc/mmu_helper.c
> +++ b/target/sparc/mmu_helper.c
> @@ -203,12 +203,12 @@ static int get_physical_address(CPUSPARCState *env, CPUTLBEntryFull *full,
>   }
>   
>   /* Perform address translation */
> -bool sparc_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                        MMUAccessType access_type, int mmu_idx,
> -                        bool probe, uintptr_t retaddr)
> +bool sparc_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
> +                              vaddr address, MMUAccessType access_type,
> +                              int mmu_idx, MemOp memop, int size,
> +                              bool probe, uintptr_t retaddr)
>   {
>       CPUSPARCState *env = cpu_env(cs);
> -    CPUTLBEntryFull full = {};
>       target_ulong vaddr;
>       int error_code = 0, access_index;
>   
> @@ -220,16 +220,21 @@ bool sparc_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
>        */
>       assert(!probe);
>   
> +    if (address & ((1 << memop_alignment_bits(memop)) - 1)) {
> +        sparc_cpu_do_unaligned_access(cs, address, access_type,
> +                                      mmu_idx, retaddr);
> +    }
> +
> +    memset(out, 0, sizeof(*out));
>       address &= TARGET_PAGE_MASK;
> -    error_code = get_physical_address(env, &full, &access_index,
> +    error_code = get_physical_address(env, out, &access_index,
>                                         address, access_type, mmu_idx);
>       vaddr = address;
>       if (likely(error_code == 0)) {
>           qemu_log_mask(CPU_LOG_MMU,
>                         "Translate at %" VADDR_PRIx " -> "
>                         HWADDR_FMT_plx ", vaddr " TARGET_FMT_lx "\n",
> -                      address, full.phys_addr, vaddr);
> -        tlb_set_page_full(cs, mmu_idx, vaddr, &full);
> +                      address, out->phys_addr, vaddr);
>           return true;
>       }
>   
> @@ -244,8 +249,7 @@ bool sparc_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
>              permissions. If no mapping is available, redirect accesses to
>              neverland. Fake/overridden mappings will be flushed when
>              switching to normal mode. */
> -        full.prot = PAGE_READ | PAGE_WRITE | PAGE_EXEC;
> -        tlb_set_page_full(cs, mmu_idx, vaddr, &full);
> +        out->prot = PAGE_READ | PAGE_WRITE | PAGE_EXEC;
>           return true;
>       } else {
>           if (access_type == MMU_INST_FETCH) {
> @@ -754,22 +758,30 @@ static int get_physical_address(CPUSPARCState *env, CPUTLBEntryFull *full,
>   }
>   
>   /* Perform address translation */
> -bool sparc_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                        MMUAccessType access_type, int mmu_idx,
> -                        bool probe, uintptr_t retaddr)
> +bool sparc_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
> +                              vaddr address, MMUAccessType access_type,
> +                              int mmu_idx, MemOp memop, int size,
> +                              bool probe, uintptr_t retaddr)
>   {
>       CPUSPARCState *env = cpu_env(cs);
> -    CPUTLBEntryFull full = {};
>       int error_code = 0, access_index;
>   
> +    if (address & ((1 << memop_alignment_bits(memop)) - 1)) {
> +        if (probe) {
> +            return false;
> +        }
> +        sparc_cpu_do_unaligned_access(cs, address, access_type,
> +                                      mmu_idx, retaddr);
> +    }
> +
> +    memset(out, 0, sizeof(*out));
>       address &= TARGET_PAGE_MASK;
> -    error_code = get_physical_address(env, &full, &access_index,
> +    error_code = get_physical_address(env, out, &access_index,
>                                         address, access_type, mmu_idx);
>       if (likely(error_code == 0)) {
> -        trace_mmu_helper_mmu_fault(address, full.phys_addr, mmu_idx, env->tl,
> +        trace_mmu_helper_mmu_fault(address, out->phys_addr, mmu_idx, env->tl,
>                                      env->dmmu.mmu_primary_context,
>                                      env->dmmu.mmu_secondary_context);
> -        tlb_set_page_full(cs, mmu_idx, address, &full);
>           return true;
>       }
>       if (probe) {

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 49/54] target/tricore: Convert to TCGCPUOps.tlb_fill_align

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   target/tricore/cpu.h    |  7 ++++---
>   target/tricore/cpu.c    |  2 +-
>   target/tricore/helper.c | 19 ++++++++++++-------
>   3 files changed, 17 insertions(+), 11 deletions(-)
> 
> diff --git a/target/tricore/cpu.h b/target/tricore/cpu.h
> index 220af69fc2..5f141ce8f3 100644
> --- a/target/tricore/cpu.h
> +++ b/target/tricore/cpu.h
> @@ -268,8 +268,9 @@ static inline void cpu_get_tb_cpu_state(CPUTriCoreState *env, vaddr *pc,
>   #define CPU_RESOLVING_TYPE TYPE_TRICORE_CPU
>   
>   /* helpers.c */
> -bool tricore_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                          MMUAccessType access_type, int mmu_idx,
> -                          bool probe, uintptr_t retaddr);
> +bool tricore_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
> +                                vaddr addr, MMUAccessType access_type,
> +                                int mmu_idx, MemOp memop, int size,
> +                                bool probe, uintptr_t retaddr);
>   
>   #endif /* TRICORE_CPU_H */
> diff --git a/target/tricore/cpu.c b/target/tricore/cpu.c
> index 1a26171590..29e0b5d129 100644
> --- a/target/tricore/cpu.c
> +++ b/target/tricore/cpu.c
> @@ -173,7 +173,7 @@ static const TCGCPUOps tricore_tcg_ops = {
>       .initialize = tricore_tcg_init,
>       .synchronize_from_tb = tricore_cpu_synchronize_from_tb,
>       .restore_state_to_opc = tricore_restore_state_to_opc,
> -    .tlb_fill = tricore_cpu_tlb_fill,
> +    .tlb_fill_align = tricore_cpu_tlb_fill_align,
>       .cpu_exec_interrupt = tricore_cpu_exec_interrupt,
>       .cpu_exec_halt = tricore_cpu_has_work,
>   };
> diff --git a/target/tricore/helper.c b/target/tricore/helper.c
> index 7014255f77..8c6bf63298 100644
> --- a/target/tricore/helper.c
> +++ b/target/tricore/helper.c
> @@ -64,16 +64,19 @@ static void raise_mmu_exception(CPUTriCoreState *env, target_ulong address,
>   {
>   }
>   
> -bool tricore_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                          MMUAccessType rw, int mmu_idx,
> -                          bool probe, uintptr_t retaddr)
> +bool tricore_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
> +                                vaddr address, MMUAccessType access_type,
> +                                int mmu_idx, MemOp memop, int size,
> +                                bool probe, uintptr_t retaddr)
>   {
>       CPUTriCoreState *env = cpu_env(cs);
>       hwaddr physical;
>       int prot;
>       int ret = 0;
> +    int rw = access_type & 1;
> +
> +    /* TODO: alignment faults not currently handled. */
>   
> -    rw &= 1;
>       ret = get_physical_address(env, &physical, &prot,
>                                  address, rw, mmu_idx);
>   
> @@ -82,9 +85,11 @@ bool tricore_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
>                     __func__, address, ret, physical, prot);
>   
>       if (ret == TLBRET_MATCH) {
> -        tlb_set_page(cs, address & TARGET_PAGE_MASK,
> -                     physical & TARGET_PAGE_MASK, prot | PAGE_EXEC,
> -                     mmu_idx, TARGET_PAGE_SIZE);
> +        memset(out, 0, sizeof(*out));
> +        out->phys_addr = physical;
> +        out->prot = prot | PAGE_EXEC;
> +        out->lg_page_size = TARGET_PAGE_BITS;
> +        out->attrs = MEMTXATTRS_UNSPECIFIED;
>           return true;
>       } else {
>           assert(ret < 0);

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 50/54] target/xtensa: Convert to TCGCPUOps.tlb_fill_align

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   target/xtensa/cpu.h    |  8 +++++---
>   target/xtensa/cpu.c    |  2 +-
>   target/xtensa/helper.c | 28 ++++++++++++++++++++--------
>   3 files changed, 26 insertions(+), 12 deletions(-)
> 
> diff --git a/target/xtensa/cpu.h b/target/xtensa/cpu.h
> index 77e48eef19..68c3d90d41 100644
> --- a/target/xtensa/cpu.h
> +++ b/target/xtensa/cpu.h
> @@ -31,6 +31,7 @@
>   #include "cpu-qom.h"
>   #include "qemu/cpu-float.h"
>   #include "exec/cpu-defs.h"
> +#include "exec/memop.h"
>   #include "hw/clock.h"
>   #include "xtensa-isa.h"
>   
> @@ -580,9 +581,10 @@ struct XtensaCPUClass {
>   };
>   
>   #ifndef CONFIG_USER_ONLY
> -bool xtensa_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                         MMUAccessType access_type, int mmu_idx,
> -                         bool probe, uintptr_t retaddr);
> +bool xtensa_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
> +                               vaddr addr, MMUAccessType access_type,
> +                               int mmu_idx, MemOp memop, int size,
> +                               bool probe, uintptr_t retaddr);
>   void xtensa_cpu_do_interrupt(CPUState *cpu);
>   bool xtensa_cpu_exec_interrupt(CPUState *cpu, int interrupt_request);
>   void xtensa_cpu_do_transaction_failed(CPUState *cs, hwaddr physaddr, vaddr addr,
> diff --git a/target/xtensa/cpu.c b/target/xtensa/cpu.c
> index 6f9039abae..3e4ec97e0e 100644
> --- a/target/xtensa/cpu.c
> +++ b/target/xtensa/cpu.c
> @@ -232,7 +232,7 @@ static const TCGCPUOps xtensa_tcg_ops = {
>       .restore_state_to_opc = xtensa_restore_state_to_opc,
>   
>   #ifndef CONFIG_USER_ONLY
> -    .tlb_fill = xtensa_cpu_tlb_fill,
> +    .tlb_fill_align = xtensa_cpu_tlb_fill_align,
>       .cpu_exec_interrupt = xtensa_cpu_exec_interrupt,
>       .cpu_exec_halt = xtensa_cpu_has_work,
>       .do_interrupt = xtensa_cpu_do_interrupt,
> diff --git a/target/xtensa/helper.c b/target/xtensa/helper.c
> index ca214b948a..69b0e661c8 100644
> --- a/target/xtensa/helper.c
> +++ b/target/xtensa/helper.c
> @@ -261,15 +261,26 @@ void xtensa_cpu_do_unaligned_access(CPUState *cs,
>                                     addr);
>   }
>   
> -bool xtensa_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
> -                         MMUAccessType access_type, int mmu_idx,
> -                         bool probe, uintptr_t retaddr)
> +bool xtensa_cpu_tlb_fill_align(CPUState *cs, CPUTLBEntryFull *out,
> +                               vaddr address, MMUAccessType access_type,
> +                               int mmu_idx, MemOp memop, int size,
> +                               bool probe, uintptr_t retaddr)
>   {
>       CPUXtensaState *env = cpu_env(cs);
>       uint32_t paddr;
>       uint32_t page_size;
>       unsigned access;
> -    int ret = xtensa_get_physical_addr(env, true, address, access_type,
> +    int ret;
> +
> +    if (address & ((1 << memop_alignment_bits(memop)) - 1)) {
> +        if (probe) {
> +            return false;
> +        }
> +        xtensa_cpu_do_unaligned_access(cs, address, access_type,
> +                                       mmu_idx, retaddr);
> +    }
> +
> +    ret = xtensa_get_physical_addr(env, true, address, access_type,
>                                          mmu_idx, &paddr, &page_size, &access);
>   
>       qemu_log_mask(CPU_LOG_MMU, "%s(%08" VADDR_PRIx
> @@ -277,10 +288,11 @@ bool xtensa_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
>                     __func__, address, access_type, mmu_idx, paddr, ret);
>   
>       if (ret == 0) {
> -        tlb_set_page(cs,
> -                     address & TARGET_PAGE_MASK,
> -                     paddr & TARGET_PAGE_MASK,
> -                     access, mmu_idx, page_size);
> +        memset(out, 0, sizeof(*out));
> +        out->phys_addr = paddr;
> +        out->prot = access;
> +        out->lg_page_size = ctz32(page_size);
> +        out->attrs = MEMTXATTRS_UNSPECIFIED;
>           return true;
>       } else if (probe) {
>           return false;

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 51/54] accel/tcg: Drop TCGCPUOps.tlb_fill

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Now that all targets have been converted to tlb_fill_align,
> remove the tlb_fill hook.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   include/hw/core/tcg-cpu-ops.h | 10 ----------
>   accel/tcg/cputlb.c            | 19 ++++---------------
>   2 files changed, 4 insertions(+), 25 deletions(-)
> 
> diff --git a/include/hw/core/tcg-cpu-ops.h b/include/hw/core/tcg-cpu-ops.h
> index 663efb9133..70cafcc6cd 100644
> --- a/include/hw/core/tcg-cpu-ops.h
> +++ b/include/hw/core/tcg-cpu-ops.h
> @@ -157,16 +157,6 @@ struct TCGCPUOps {
>       bool (*tlb_fill_align)(CPUState *cpu, CPUTLBEntryFull *out, vaddr addr,
>                              MMUAccessType access_type, int mmu_idx,
>                              MemOp memop, int size, bool probe, uintptr_t ra);
> -    /**
> -     * @tlb_fill: Handle a softmmu tlb miss
> -     *
> -     * If the access is valid, call tlb_set_page and return true;
> -     * if the access is invalid and probe is true, return false;
> -     * otherwise raise an exception and do not return.
> -     */
> -    bool (*tlb_fill)(CPUState *cpu, vaddr address, int size,
> -                     MMUAccessType access_type, int mmu_idx,
> -                     bool probe, uintptr_t retaddr);
>       /**
>        * @do_transaction_failed: Callback for handling failed memory transactions
>        * (ie bus faults or external aborts; not MMU faults)
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index 7f63dc3fd8..ec597ed6f5 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -1222,23 +1222,12 @@ static bool tlb_fill_align(CPUState *cpu, vaddr addr, MMUAccessType type,
>                              int mmu_idx, MemOp memop, int size,
>                              bool probe, uintptr_t ra)
>   {
> -    const TCGCPUOps *ops = cpu->cc->tcg_ops;
>       CPUTLBEntryFull full;
>   
> -    if (ops->tlb_fill_align) {
> -        if (ops->tlb_fill_align(cpu, &full, addr, type, mmu_idx,
> -                                memop, size, probe, ra)) {
> -            tlb_set_page_full(cpu, mmu_idx, addr, &full);
> -            return true;
> -        }
> -    } else {
> -        /* Legacy behaviour is alignment before paging. */
> -        if (addr & ((1u << memop_alignment_bits(memop)) - 1)) {
> -            ops->do_unaligned_access(cpu, addr, type, mmu_idx, ra);
> -        }
> -        if (ops->tlb_fill(cpu, addr, size, type, mmu_idx, probe, ra)) {
> -            return true;
> -        }
> +    if (cpu->cc->tcg_ops->tlb_fill_align(cpu, &full, addr, type, mmu_idx,
> +                                         memop, size, probe, ra)) {
> +        tlb_set_page_full(cpu, mmu_idx, addr, &full);
> +        return true;
>       }
>       assert(probe);
>       return false;

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 52/54] accel/tcg: Unexport tlb_set_page*

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> The new tlb_fill_align hook returns page data via structure
> rather than by function call, so we can make tlb_set_page_full
> be local to cputlb.c.  There are no users of tlb_set_page
> or tlb_set_page_with_attrs, so those can be eliminated.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   include/exec/exec-all.h | 57 -----------------------------------------
>   accel/tcg/cputlb.c      | 27 ++-----------------
>   2 files changed, 2 insertions(+), 82 deletions(-)
> 
> diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
> index 69bdb77584..b65fc547bd 100644
> --- a/include/exec/exec-all.h
> +++ b/include/exec/exec-all.h
> @@ -184,63 +184,6 @@ void tlb_flush_range_by_mmuidx_all_cpus_synced(CPUState *cpu,
>                                                  vaddr len,
>                                                  uint16_t idxmap,
>                                                  unsigned bits);
> -
> -/**
> - * tlb_set_page_full:
> - * @cpu: CPU context
> - * @mmu_idx: mmu index of the tlb to modify
> - * @addr: virtual address of the entry to add
> - * @full: the details of the tlb entry
> - *
> - * Add an entry to @cpu tlb index @mmu_idx.  All of the fields of
> - * @full must be filled, except for xlat_section, and constitute
> - * the complete description of the translated page.
> - *
> - * This is generally called by the target tlb_fill function after
> - * having performed a successful page table walk to find the physical
> - * address and attributes for the translation.
> - *
> - * At most one entry for a given virtual address is permitted. Only a
> - * single TARGET_PAGE_SIZE region is mapped; @full->lg_page_size is only
> - * used by tlb_flush_page.
> - */
> -void tlb_set_page_full(CPUState *cpu, int mmu_idx, vaddr addr,
> -                       CPUTLBEntryFull *full);
> -
> -/**
> - * tlb_set_page_with_attrs:
> - * @cpu: CPU to add this TLB entry for
> - * @addr: virtual address of page to add entry for
> - * @paddr: physical address of the page
> - * @attrs: memory transaction attributes
> - * @prot: access permissions (PAGE_READ/PAGE_WRITE/PAGE_EXEC bits)
> - * @mmu_idx: MMU index to insert TLB entry for
> - * @size: size of the page in bytes
> - *
> - * Add an entry to this CPU's TLB (a mapping from virtual address
> - * @addr to physical address @paddr) with the specified memory
> - * transaction attributes. This is generally called by the target CPU
> - * specific code after it has been called through the tlb_fill()
> - * entry point and performed a successful page table walk to find
> - * the physical address and attributes for the virtual address
> - * which provoked the TLB miss.
> - *
> - * At most one entry for a given virtual address is permitted. Only a
> - * single TARGET_PAGE_SIZE region is mapped; the supplied @size is only
> - * used by tlb_flush_page.
> - */
> -void tlb_set_page_with_attrs(CPUState *cpu, vaddr addr,
> -                             hwaddr paddr, MemTxAttrs attrs,
> -                             int prot, int mmu_idx, vaddr size);
> -/* tlb_set_page:
> - *
> - * This function is equivalent to calling tlb_set_page_with_attrs()
> - * with an @attrs argument of MEMTXATTRS_UNSPECIFIED. It's provided
> - * as a convenience for CPUs which don't use memory transaction attributes.
> - */
> -void tlb_set_page(CPUState *cpu, vaddr addr,
> -                  hwaddr paddr, int prot,
> -                  int mmu_idx, vaddr size);
>   #else
>   static inline void tlb_init(CPUState *cpu)
>   {
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index ec597ed6f5..3d731b8f3d 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -1037,8 +1037,8 @@ static inline void tlb_set_compare(CPUTLBEntryFull *full, CPUTLBEntry *ent,
>    * Called from TCG-generated code, which is under an RCU read-side
>    * critical section.
>    */
> -void tlb_set_page_full(CPUState *cpu, int mmu_idx,
> -                       vaddr addr, CPUTLBEntryFull *full)
> +static void tlb_set_page_full(CPUState *cpu, int mmu_idx,
> +                              vaddr addr, CPUTLBEntryFull *full)
>   {
>       CPUTLB *tlb = &cpu->neg.tlb;
>       CPUTLBDesc *desc = &tlb->d[mmu_idx];
> @@ -1189,29 +1189,6 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx,
>       qemu_spin_unlock(&tlb->c.lock);
>   }
>   
> -void tlb_set_page_with_attrs(CPUState *cpu, vaddr addr,
> -                             hwaddr paddr, MemTxAttrs attrs, int prot,
> -                             int mmu_idx, uint64_t size)
> -{
> -    CPUTLBEntryFull full = {
> -        .phys_addr = paddr,
> -        .attrs = attrs,
> -        .prot = prot,
> -        .lg_page_size = ctz64(size)
> -    };
> -
> -    assert(is_power_of_2(size));
> -    tlb_set_page_full(cpu, mmu_idx, addr, &full);
> -}
> -
> -void tlb_set_page(CPUState *cpu, vaddr addr,
> -                  hwaddr paddr, int prot,
> -                  int mmu_idx, uint64_t size)
> -{
> -    tlb_set_page_with_attrs(cpu, addr, paddr, MEMTXATTRS_UNSPECIFIED,
> -                            prot, mmu_idx, size);
> -}
> -
>   /*
>    * Note: tlb_fill_align() can trigger a resize of the TLB.
>    * This means that all of the caller's prior references to the TLB table

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 53/54] accel/tcg: Merge tlb_fill_align into callers

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> In tlb_lookup, we still call tlb_set_page_full.
> In atomic_mmu_lookup, we're expecting noreturn.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 31 ++++++-------------------------
>   1 file changed, 6 insertions(+), 25 deletions(-)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index 3d731b8f3d..20af48c6c5 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -1189,27 +1189,6 @@ static void tlb_set_page_full(CPUState *cpu, int mmu_idx,
>       qemu_spin_unlock(&tlb->c.lock);
>   }
>   
> -/*
> - * Note: tlb_fill_align() can trigger a resize of the TLB.
> - * This means that all of the caller's prior references to the TLB table
> - * (e.g. CPUTLBEntry pointers) must be discarded and looked up again
> - * (e.g. via tlb_entry()).
> - */
> -static bool tlb_fill_align(CPUState *cpu, vaddr addr, MMUAccessType type,
> -                           int mmu_idx, MemOp memop, int size,
> -                           bool probe, uintptr_t ra)
> -{
> -    CPUTLBEntryFull full;
> -
> -    if (cpu->cc->tcg_ops->tlb_fill_align(cpu, &full, addr, type, mmu_idx,
> -                                         memop, size, probe, ra)) {
> -        tlb_set_page_full(cpu, mmu_idx, addr, &full);
> -        return true;
> -    }
> -    assert(probe);
> -    return false;
> -}
> -
>   static inline void cpu_unaligned_access(CPUState *cpu, vaddr addr,
>                                           MMUAccessType access_type,
>                                           int mmu_idx, uintptr_t retaddr)
> @@ -1281,11 +1260,13 @@ static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
>       }
>   
>       /* Finally, query the target hook. */
> -    if (!tlb_fill_align(cpu, addr, access_type, i->mmu_idx,
> -                        memop, i->size, probe, i->ra)) {
> +    if (!cpu->cc->tcg_ops->tlb_fill_align(cpu, &o->full, addr, access_type,
> +                                          i->mmu_idx, memop, i->size,
> +                                          probe, i->ra)) {
>           tcg_debug_assert(probe);
>           return false;
>       }
> +    tlb_set_page_full(cpu, i->mmu_idx, addr, &o->full);
>       o->did_tlb_fill = true;
>   
>       if (access_type == MMU_INST_FETCH) {
> @@ -1794,8 +1775,8 @@ static void *atomic_mmu_lookup(CPUState *cpu, vaddr addr, MemOpIdx oi,
>        * We have just verified that the page is writable.
>        */
>       if (unlikely(!(o.full.prot & PAGE_READ))) {
> -        tlb_fill_align(cpu, addr, MMU_DATA_LOAD, i.mmu_idx,
> -                       0, i.size, false, i.ra);
> +        cpu->cc->tcg_ops->tlb_fill_align(cpu, &o.full, addr, MMU_DATA_LOAD,
> +                                         i.mmu_idx, 0, i.size, false, i.ra);
>           /*
>            * Since we don't support reads and writes to different
>            * addresses, and we do have the proper page loaded for

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 54/54] accel/tcg: Return CPUTLBEntryTree from tlb_set_page_full

[Pierrick Bouvier]

On 11/14/24 08:01, Richard Henderson wrote:
> Avoid a lookup to find the node that we have just inserted.
> 
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>   accel/tcg/cputlb.c | 14 ++++++--------
>   1 file changed, 6 insertions(+), 8 deletions(-)
> 
> diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
> index 20af48c6c5..6d316e8767 100644
> --- a/accel/tcg/cputlb.c
> +++ b/accel/tcg/cputlb.c
> @@ -1037,8 +1037,8 @@ static inline void tlb_set_compare(CPUTLBEntryFull *full, CPUTLBEntry *ent,
>    * Called from TCG-generated code, which is under an RCU read-side
>    * critical section.
>    */
> -static void tlb_set_page_full(CPUState *cpu, int mmu_idx,
> -                              vaddr addr, CPUTLBEntryFull *full)
> +static CPUTLBEntryTree *tlb_set_page_full(CPUState *cpu, int mmu_idx,
> +                                          vaddr addr, CPUTLBEntryFull *full)
>   {
>       CPUTLB *tlb = &cpu->neg.tlb;
>       CPUTLBDesc *desc = &tlb->d[mmu_idx];
> @@ -1187,6 +1187,8 @@ static void tlb_set_page_full(CPUState *cpu, int mmu_idx,
>       copy_tlb_helper_locked(te, &node->copy);
>       desc->n_used_entries++;
>       qemu_spin_unlock(&tlb->c.lock);
> +
> +    return node;
>   }
>   
>   static inline void cpu_unaligned_access(CPUState *cpu, vaddr addr,
> @@ -1266,18 +1268,14 @@ static bool tlb_lookup(CPUState *cpu, TLBLookupOutput *o,
>           tcg_debug_assert(probe);
>           return false;
>       }
> -    tlb_set_page_full(cpu, i->mmu_idx, addr, &o->full);
> +    node = tlb_set_page_full(cpu, i->mmu_idx, addr, &o->full);
>       o->did_tlb_fill = true;
>   
>       if (access_type == MMU_INST_FETCH) {
> -        node = tlbtree_lookup_addr(desc, addr);
> -        tcg_debug_assert(node);
>           goto found_code;
>       }
>   
> -    entry = tlbfast_entry(fast, addr);
> -    cmp = tlb_read_idx(entry, access_type);
> -    node = entry->tree;
> +    cmp = tlb_read_idx(&node->copy, access_type);
>       /*
>        * With PAGE_WRITE_INV, we set TLB_INVALID_MASK immediately,
>        * to force the next access through tlb_fill_align.  We've just

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH for-10.0 v2 00/54] accel/tcg: Convert victim tlb to IntervalTree

[Pierrick Bouvier]

On 11/14/24 08:00, Richard Henderson wrote:
> v1: 20241009150855.804605-1-richard.henderson@linaro.org
> 
> The initial idea was: how much can we do with an intelligent data
> structure for the same cost as a linear search through an array?
> 
> 
> r~
> 
> 
> Richard Henderson (54):
>    util/interval-tree: Introduce interval_tree_free_nodes
>    accel/tcg: Split out tlbfast_flush_locked
>    accel/tcg: Split out tlbfast_{index,entry}
>    accel/tcg: Split out tlbfast_flush_range_locked
>    accel/tcg: Fix flags usage in mmu_lookup1, atomic_mmu_lookup
>    accel/tcg: Assert non-zero length in tlb_flush_range_by_mmuidx*
>    accel/tcg: Assert bits in range in tlb_flush_range_by_mmuidx*
>    accel/tcg: Flush entire tlb when a masked range wraps
>    accel/tcg: Add IntervalTreeRoot to CPUTLBDesc
>    accel/tcg: Populate IntervalTree in tlb_set_page_full
>    accel/tcg: Remove IntervalTree entry in tlb_flush_page_locked
>    accel/tcg: Remove IntervalTree entries in tlb_flush_range_locked
>    accel/tcg: Process IntervalTree entries in tlb_reset_dirty
>    accel/tcg: Process IntervalTree entries in tlb_set_dirty
>    accel/tcg: Use tlb_hit_page in victim_tlb_hit
>    accel/tcg: Pass full addr to victim_tlb_hit
>    accel/tcg: Replace victim_tlb_hit with tlbtree_hit
>    accel/tcg: Remove the victim tlb
>    accel/tcg: Remove tlb_n_used_entries_inc
>    include/exec/tlb-common: Move CPUTLBEntryFull from hw/core/cpu.h
>    accel/tcg: Delay plugin adjustment in probe_access_internal
>    accel/tcg: Call cpu_ld*_code_mmu from cpu_ld*_code
>    accel/tcg: Check original prot bits for read in atomic_mmu_lookup
>    accel/tcg: Preserve tlb flags in tlb_set_compare
>    accel/tcg: Return CPUTLBEntryFull not pointer in probe_access_full_mmu
>    accel/tcg: Return CPUTLBEntryFull not pointer in probe_access_full
>    accel/tcg: Return CPUTLBEntryFull not pointer in probe_access_internal
>    accel/tcg: Introduce tlb_lookup
>    accel/tcg: Partially unify MMULookupPageData and TLBLookupOutput
>    accel/tcg: Merge mmu_lookup1 into mmu_lookup
>    accel/tcg: Always use IntervalTree for code lookups
>    accel/tcg: Link CPUTLBEntry to CPUTLBEntryTree
>    accel/tcg: Remove CPUTLBDesc.fulltlb
>    target/alpha: Convert to TCGCPUOps.tlb_fill_align
>    target/avr: Convert to TCGCPUOps.tlb_fill_align
>    target/i386: Convert to TCGCPUOps.tlb_fill_align
>    target/loongarch: Convert to TCGCPUOps.tlb_fill_align
>    target/m68k: Convert to TCGCPUOps.tlb_fill_align
>    target/m68k: Do not call tlb_set_page in helper_ptest
>    target/microblaze: Convert to TCGCPUOps.tlb_fill_align
>    target/mips: Convert to TCGCPUOps.tlb_fill_align
>    target/openrisc: Convert to TCGCPUOps.tlb_fill_align
>    target/ppc: Convert to TCGCPUOps.tlb_fill_align
>    target/riscv: Convert to TCGCPUOps.tlb_fill_align
>    target/rx: Convert to TCGCPUOps.tlb_fill_align
>    target/s390x: Convert to TCGCPUOps.tlb_fill_align
>    target/sh4: Convert to TCGCPUOps.tlb_fill_align
>    target/sparc: Convert to TCGCPUOps.tlb_fill_align
>    target/tricore: Convert to TCGCPUOps.tlb_fill_align
>    target/xtensa: Convert to TCGCPUOps.tlb_fill_align
>    accel/tcg: Drop TCGCPUOps.tlb_fill
>    accel/tcg: Unexport tlb_set_page*
>    accel/tcg: Merge tlb_fill_align into callers
>    accel/tcg: Return CPUTLBEntryTree from tlb_set_page_full
> 
>   include/exec/cpu-all.h               |   3 +
>   include/exec/exec-all.h              |  65 +-
>   include/exec/tlb-common.h            |  68 +-
>   include/hw/core/cpu.h                |  75 +-
>   include/hw/core/tcg-cpu-ops.h        |  10 -
>   include/qemu/interval-tree.h         |  11 +
>   target/alpha/cpu.h                   |   6 +-
>   target/avr/cpu.h                     |   7 +-
>   target/i386/tcg/helper-tcg.h         |   6 +-
>   target/loongarch/internals.h         |   7 +-
>   target/m68k/cpu.h                    |   7 +-
>   target/microblaze/cpu.h              |   7 +-
>   target/mips/tcg/tcg-internal.h       |   6 +-
>   target/openrisc/cpu.h                |   8 +-
>   target/ppc/internal.h                |   7 +-
>   target/riscv/cpu.h                   |   8 +-
>   target/s390x/s390x-internal.h        |   7 +-
>   target/sh4/cpu.h                     |   8 +-
>   target/sparc/cpu.h                   |   8 +-
>   target/tricore/cpu.h                 |   7 +-
>   target/xtensa/cpu.h                  |   8 +-
>   accel/tcg/cputlb.c                   | 994 +++++++++++++--------------
>   target/alpha/cpu.c                   |   2 +-
>   target/alpha/helper.c                |  23 +-
>   target/arm/ptw.c                     |  10 +-
>   target/arm/tcg/helper-a64.c          |   4 +-
>   target/arm/tcg/mte_helper.c          |  15 +-
>   target/arm/tcg/sve_helper.c          |   6 +-
>   target/avr/cpu.c                     |   2 +-
>   target/avr/helper.c                  |  19 +-
>   target/i386/tcg/sysemu/excp_helper.c |  36 +-
>   target/i386/tcg/tcg-cpu.c            |   2 +-
>   target/loongarch/cpu.c               |   2 +-
>   target/loongarch/tcg/tlb_helper.c    |  17 +-
>   target/m68k/cpu.c                    |   2 +-
>   target/m68k/helper.c                 |  32 +-
>   target/microblaze/cpu.c              |   2 +-
>   target/microblaze/helper.c           |  33 +-
>   target/mips/cpu.c                    |   2 +-
>   target/mips/tcg/sysemu/tlb_helper.c  |  29 +-
>   target/openrisc/cpu.c                |   2 +-
>   target/openrisc/mmu.c                |  39 +-
>   target/ppc/cpu_init.c                |   2 +-
>   target/ppc/mmu_helper.c              |  21 +-
>   target/riscv/cpu_helper.c            |  22 +-
>   target/riscv/tcg/tcg-cpu.c           |   2 +-
>   target/rx/cpu.c                      |  19 +-
>   target/s390x/cpu.c                   |   4 +-
>   target/s390x/tcg/excp_helper.c       |  23 +-
>   target/sh4/cpu.c                     |   2 +-
>   target/sh4/helper.c                  |  24 +-
>   target/sparc/cpu.c                   |   2 +-
>   target/sparc/mmu_helper.c            |  44 +-
>   target/tricore/cpu.c                 |   2 +-
>   target/tricore/helper.c              |  19 +-
>   target/xtensa/cpu.c                  |   2 +-
>   target/xtensa/helper.c               |  28 +-
>   util/interval-tree.c                 |  20 +
>   util/selfmap.c                       |  13 +-
>   59 files changed, 938 insertions(+), 923 deletions(-)
> 

I tested this change by booting a debian x86_64 image, it works as expected.

I noticed that this change does not come for free (64s before, 82s after 
- 1.3x). Is that acceptable?

Pierrick

Re: [PATCH for-10.0 v2 00/54] accel/tcg: Convert victim tlb to IntervalTree

[Richard Henderson]

On 11/14/24 11:56, Pierrick Bouvier wrote:
> I tested this change by booting a debian x86_64 image, it works as expected.
> 
> I noticed that this change does not come for free (64s before, 82s after - 1.3x). Is that 
> acceptable?
Well, no.  But I didn't notice any change during boot tests.  I used hyperfine over 'make 
check-functional'.

I would only expect benefits to be seen during longer lived vm's, since a boot test 
doesn't run applications long enough to see tlb entries accumulate.  I have not attempted 
to create a reproducible test for that so far.


r~

Re: [PATCH for-10.0 v2 00/54] accel/tcg: Convert victim tlb to IntervalTree

[Pierrick Bouvier]

On 11/14/24 12:58, Richard Henderson wrote:
> On 11/14/24 11:56, Pierrick Bouvier wrote:
>> I tested this change by booting a debian x86_64 image, it works as expected.
>>
>> I noticed that this change does not come for free (64s before, 82s after - 1.3x). Is that
>> acceptable?
> Well, no.  But I didn't notice any change during boot tests.  I used hyperfine over 'make
> check-functional'.
> 
> I would only expect benefits to be seen during longer lived vm's, since a boot test
> doesn't run applications long enough to see tlb entries accumulate.  I have not attempted
> to create a reproducible test for that so far.
> 
> 

I didn't use check-functional neither.
I used a vanilla debian bookworm install, with a modified /etc/rc.local 
calling poweroff, and ran 3 times with/without change with turbo 
disabled on my cpu.

> r~

Re: [PATCH for-10.0 v2 00/54] accel/tcg: Convert victim tlb to IntervalTree

[Alex Bennée]

Pierrick Bouvier <pierrick.bouvier@linaro.org> writes:

> On 11/14/24 12:58, Richard Henderson wrote:
>> On 11/14/24 11:56, Pierrick Bouvier wrote:
>>> I tested this change by booting a debian x86_64 image, it works as expected.
>>>
>>> I noticed that this change does not come for free (64s before, 82s after - 1.3x). Is that
>>> acceptable?
>> Well, no.  But I didn't notice any change during boot tests.  I used hyperfine over 'make
>> check-functional'.
>> I would only expect benefits to be seen during longer lived vm's,
>> since a boot test
>> doesn't run applications long enough to see tlb entries accumulate.  I have not attempted
>> to create a reproducible test for that so far.
>> 
>
> I didn't use check-functional neither.
> I used a vanilla debian bookworm install, with a modified
> /etc/rc.local calling poweroff, and ran 3 times with/without change
> with turbo disabled on my cpu.

If you want to really stress the VM handling you should use stress-ng to
exercise page faulting and recovery. Wrap it up in a systemd unit for a
reproducible test:

  cat /etc/systemd/system/benchmark-stress-ng.service 
  # A benchmark target
  #
  # This shutsdown once the boot has completed

  [Unit]
  Description=Default
  Requires=basic.target
  After=basic.target
  AllowIsolate=yes

  [Service]
  Type=oneshot
  ExecStart=stress-ng --perf --iomix 4 --vm 2 --timeout 10s
  ExecStartPost=/sbin/poweroff

  [Install]
  WantedBy=multi-user.target

and then call with something like:

  -append "root=/dev/sda2 console=ttyAMA0 systemd.unit=benchmark-stress-ng.service"

>
>> r~

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH for-10.0 v2 00/54] accel/tcg: Convert victim tlb to IntervalTree

[Pierrick Bouvier]

On 11/15/24 03:43, Alex Bennée wrote:
> Pierrick Bouvier <pierrick.bouvier@linaro.org> writes:
> 
>> On 11/14/24 12:58, Richard Henderson wrote:
>>> On 11/14/24 11:56, Pierrick Bouvier wrote:
>>>> I tested this change by booting a debian x86_64 image, it works as expected.
>>>>
>>>> I noticed that this change does not come for free (64s before, 82s after - 1.3x). Is that
>>>> acceptable?
>>> Well, no.  But I didn't notice any change during boot tests.  I used hyperfine over 'make
>>> check-functional'.
>>> I would only expect benefits to be seen during longer lived vm's,
>>> since a boot test
>>> doesn't run applications long enough to see tlb entries accumulate.  I have not attempted
>>> to create a reproducible test for that so far.
>>>
>>
>> I didn't use check-functional neither.
>> I used a vanilla debian bookworm install, with a modified
>> /etc/rc.local calling poweroff, and ran 3 times with/without change
>> with turbo disabled on my cpu.
> 
> If you want to really stress the VM handling you should use stress-ng to
> exercise page faulting and recovery. Wrap it up in a systemd unit for a
> reproducible test:
> 
>    cat /etc/systemd/system/benchmark-stress-ng.service
>    # A benchmark target
>    #
>    # This shutsdown once the boot has completed
> 
>    [Unit]
>    Description=Default
>    Requires=basic.target
>    After=basic.target
>    AllowIsolate=yes
> 
>    [Service]
>    Type=oneshot
>    ExecStart=stress-ng --perf --iomix 4 --vm 2 --timeout 10s
>    ExecStartPost=/sbin/poweroff
> 
>    [Install]
>    WantedBy=multi-user.target
> 
> and then call with something like:
> 
>    -append "root=/dev/sda2 console=ttyAMA0 systemd.unit=benchmark-stress-ng.service"
> 

Thanks for the advice.

>>
>>> r~
>