[RFC PATCH 0/7] accel: Add per-accelerator vCPUs queue

[RFC PATCH 0/7] accel: Add per-accelerator vCPUs queue

[Philippe Mathieu-Daudé]

Hi,

Currently we register all vCPUs to the global 'cpus_queue' queue,
however we can not discriminate per accelerator or per target
architecture (which might happen in a soon future).

This series tries to add an accelerator discriminator, so
accelerator specific code can iterate on its own vCPUs. This
is required to run a pair of HW + SW accelerators like the
(HVF, TCG) or (KVM, TCG) combinations. Otherwise, i.e. the
HVF core code could iterate on TCG vCPUs...
To keep it simple and not refactor heavily the code base,
we introduce the CPU_FOREACH_TCG/HVF/KVM() macros, only
defined for each accelerator.

This is just a RFC to get some thoughts whether this is
heading in the correct direction or not ;)

Regards,

Phil.

Philippe Mathieu-Daudé (7):
  cpus: Restrict CPU_FOREACH_SAFE() to user emulation
  cpus: Introduce AccelOpsClass::get_cpus_queue()
  accel/tcg: Implement tcg_get_cpus_queue()
  accel/tcg: Use CPU_FOREACH_TCG()
  accel/hw: Implement hw_accel_get_cpus_queue()
  accel/hvf: Use CPU_FOREACH_HVF()
  accel/kvm: Use CPU_FOREACH_KVM()

 accel/tcg/tcg-accel-ops.h         | 10 ++++++++++
 include/hw/core/cpu.h             | 11 +++++++++++
 include/system/accel-ops.h        |  6 ++++++
 include/system/hvf_int.h          |  4 ++++
 include/system/hw_accel.h         |  9 +++++++++
 include/system/kvm_int.h          |  3 +++
 accel/accel-system.c              |  8 ++++++++
 accel/hvf/hvf-accel-ops.c         |  9 +++++----
 accel/kvm/kvm-accel-ops.c         |  1 +
 accel/kvm/kvm-all.c               | 14 +++++++-------
 accel/tcg/cputlb.c                |  7 ++++---
 accel/tcg/monitor.c               |  3 ++-
 accel/tcg/tb-maint.c              |  7 ++++---
 accel/tcg/tcg-accel-ops-rr.c      | 10 +++++-----
 accel/tcg/tcg-accel-ops.c         | 16 ++++++++++++----
 accel/tcg/user-exec-stub.c        |  5 +++++
 accel/xen/xen-all.c               |  1 +
 cpu-common.c                      | 10 ++++++++++
 hw/i386/kvm/clock.c               |  3 ++-
 hw/intc/spapr_xive_kvm.c          |  5 +++--
 hw/intc/xics_kvm.c                |  5 +++--
 system/cpus.c                     |  5 +++++
 target/arm/hvf/hvf.c              |  4 ++--
 target/i386/kvm/kvm.c             |  4 ++--
 target/i386/kvm/xen-emu.c         |  2 +-
 target/i386/nvmm/nvmm-accel-ops.c |  1 +
 target/i386/whpx/whpx-accel-ops.c |  1 +
 target/s390x/kvm/kvm.c            |  2 +-
 target/s390x/kvm/stsi-topology.c  |  3 ++-
 29 files changed, 130 insertions(+), 39 deletions(-)

-- 
2.47.1

[RFC PATCH 1/7] cpus: Restrict CPU_FOREACH_SAFE() to user emulation

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 include/hw/core/cpu.h | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
index c3ca0babcb3..48d90f50a71 100644
--- a/include/hw/core/cpu.h
+++ b/include/hw/core/cpu.h
@@ -594,8 +594,11 @@ extern CPUTailQ cpus_queue;
 #define first_cpu        QTAILQ_FIRST_RCU(&cpus_queue)
 #define CPU_NEXT(cpu)    QTAILQ_NEXT_RCU(cpu, node)
 #define CPU_FOREACH(cpu) QTAILQ_FOREACH_RCU(cpu, &cpus_queue, node)
+
+#if defined(CONFIG_USER_ONLY)
 #define CPU_FOREACH_SAFE(cpu, next_cpu) \
     QTAILQ_FOREACH_SAFE_RCU(cpu, &cpus_queue, node, next_cpu)
+#endif
 
 extern __thread CPUState *current_cpu;
 
-- 
2.47.1

Re: [RFC PATCH 1/7] cpus: Restrict CPU_FOREACH_SAFE() to user emulation

[Daniel Henrique Barboza]

Perhaps add in the commit msg something like "it's only being used in
bsd-user and linux-user code"

On 1/6/25 5:02 PM, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---

Reviewed-by: Daniel Henrique Barboza <dbarboza@ventanamicro.com>

>   include/hw/core/cpu.h | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
> index c3ca0babcb3..48d90f50a71 100644
> --- a/include/hw/core/cpu.h
> +++ b/include/hw/core/cpu.h
> @@ -594,8 +594,11 @@ extern CPUTailQ cpus_queue;
>   #define first_cpu        QTAILQ_FIRST_RCU(&cpus_queue)
>   #define CPU_NEXT(cpu)    QTAILQ_NEXT_RCU(cpu, node)
>   #define CPU_FOREACH(cpu) QTAILQ_FOREACH_RCU(cpu, &cpus_queue, node)
> +
> +#if defined(CONFIG_USER_ONLY)
>   #define CPU_FOREACH_SAFE(cpu, next_cpu) \
>       QTAILQ_FOREACH_SAFE_RCU(cpu, &cpus_queue, node, next_cpu)
> +#endif
>   
>   extern __thread CPUState *current_cpu;
>   

[RFC PATCH 2/7] cpus: Introduce AccelOpsClass::get_cpus_queue()

[Philippe Mathieu-Daudé]

We want the ability to iterate over vCPUs of a specific
accelerator.  Introduce cpus_get_accel_cpus_queue() to
be used by accelerator common code, and expose the
get_cpus_queue() in AccelOpsClass, so each accelerator
can register its own queue of vCPUs.

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 include/hw/core/cpu.h      |  8 ++++++++
 include/system/accel-ops.h |  6 ++++++
 accel/tcg/user-exec-stub.c |  5 +++++
 cpu-common.c               | 10 ++++++++++
 system/cpus.c              |  5 +++++
 5 files changed, 34 insertions(+)

diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
index 48d90f50a71..5ae9bb64d6e 100644
--- a/include/hw/core/cpu.h
+++ b/include/hw/core/cpu.h
@@ -591,6 +591,14 @@ static inline CPUArchState *cpu_env(CPUState *cpu)
 typedef QTAILQ_HEAD(CPUTailQ, CPUState) CPUTailQ;
 extern CPUTailQ cpus_queue;
 
+/**
+ * cpus_get_accel_cpus_queue:
+ * @cpu: The vCPU to get the accelerator #CPUTailQ.
+ *
+ * Returns the #CPUTailQ associated with the accelerator of the vCPU.
+ */
+CPUTailQ *cpus_get_accel_cpus_queue(CPUState *cpu);
+
 #define first_cpu        QTAILQ_FIRST_RCU(&cpus_queue)
 #define CPU_NEXT(cpu)    QTAILQ_NEXT_RCU(cpu, node)
 #define CPU_FOREACH(cpu) QTAILQ_FOREACH_RCU(cpu, &cpus_queue, node)
diff --git a/include/system/accel-ops.h b/include/system/accel-ops.h
index 137fb96d444..fe59f728bfc 100644
--- a/include/system/accel-ops.h
+++ b/include/system/accel-ops.h
@@ -12,6 +12,7 @@
 
 #include "exec/vaddr.h"
 #include "qom/object.h"
+#include "hw/core/cpu.h"
 
 #define ACCEL_OPS_SUFFIX "-ops"
 #define TYPE_ACCEL_OPS "accel" ACCEL_OPS_SUFFIX
@@ -37,6 +38,11 @@ struct AccelOpsClass {
     bool (*cpus_are_resettable)(void);
     void (*cpu_reset_hold)(CPUState *cpu);
 
+    /**
+     * get_cpus_queue:
+     * Returns the #CPUTailQ maintained by this accelerator.
+     */
+    CPUTailQ *(*get_cpus_queue)(void);
     void (*create_vcpu_thread)(CPUState *cpu); /* MANDATORY NON-NULL */
     void (*kick_vcpu_thread)(CPUState *cpu);
     bool (*cpu_thread_is_idle)(CPUState *cpu);
diff --git a/accel/tcg/user-exec-stub.c b/accel/tcg/user-exec-stub.c
index 4fbe2dbdc88..cb76cec76be 100644
--- a/accel/tcg/user-exec-stub.c
+++ b/accel/tcg/user-exec-stub.c
@@ -18,6 +18,11 @@ void cpu_exec_reset_hold(CPUState *cpu)
 {
 }
 
+CPUTailQ *cpus_get_accel_cpus_queue(CPUState *cpu)
+{
+    return NULL;
+}
+
 /* User mode emulation does not support record/replay yet.  */
 
 bool replay_exception(void)
diff --git a/cpu-common.c b/cpu-common.c
index 4248b2d727e..ff8db9c7f9d 100644
--- a/cpu-common.c
+++ b/cpu-common.c
@@ -82,6 +82,7 @@ unsigned int cpu_list_generation_id_get(void)
 void cpu_list_add(CPUState *cpu)
 {
     static bool cpu_index_auto_assigned;
+    CPUTailQ *accel_cpus_queue = cpus_get_accel_cpus_queue(cpu);
 
     QEMU_LOCK_GUARD(&qemu_cpu_list_lock);
     if (cpu->cpu_index == UNASSIGNED_CPU_INDEX) {
@@ -92,17 +93,26 @@ void cpu_list_add(CPUState *cpu)
         assert(!cpu_index_auto_assigned);
     }
     QTAILQ_INSERT_TAIL_RCU(&cpus_queue, cpu, node);
+    if (accel_cpus_queue) {
+        QTAILQ_INSERT_TAIL_RCU(accel_cpus_queue, cpu, node);
+    }
+
     cpu_list_generation_id++;
 }
 
 void cpu_list_remove(CPUState *cpu)
 {
+    CPUTailQ *accel_cpus_queue = cpus_get_accel_cpus_queue(cpu);
+
     QEMU_LOCK_GUARD(&qemu_cpu_list_lock);
     if (!QTAILQ_IN_USE(cpu, node)) {
         /* there is nothing to undo since cpu_exec_init() hasn't been called */
         return;
     }
 
+    if (accel_cpus_queue) {
+        QTAILQ_REMOVE_RCU(accel_cpus_queue, cpu, node);
+    }
     QTAILQ_REMOVE_RCU(&cpus_queue, cpu, node);
     cpu->cpu_index = UNASSIGNED_CPU_INDEX;
     cpu_list_generation_id++;
diff --git a/system/cpus.c b/system/cpus.c
index 99f83806c16..972df6ab061 100644
--- a/system/cpus.c
+++ b/system/cpus.c
@@ -209,6 +209,11 @@ void cpu_exec_reset_hold(CPUState *cpu)
     }
 }
 
+CPUTailQ *cpus_get_accel_cpus_queue(CPUState *cpu)
+{
+    return cpus_accel ? cpus_accel->get_cpus_queue() : NULL;
+}
+
 int64_t cpus_get_virtual_clock(void)
 {
     /*
-- 
2.47.1

[RFC PATCH 3/7] accel/tcg: Implement tcg_get_cpus_queue()

[Philippe Mathieu-Daudé]

Use a specific vCPUs queue for our unique software accelerator.
Register the AccelOpsClass::get_cpus_queue() handler.

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/tcg/tcg-accel-ops.h | 10 ++++++++++
 accel/tcg/tcg-accel-ops.c |  8 ++++++++
 2 files changed, 18 insertions(+)

diff --git a/accel/tcg/tcg-accel-ops.h b/accel/tcg/tcg-accel-ops.h
index 6feeb3f3e9b..7b1d6288742 100644
--- a/accel/tcg/tcg-accel-ops.h
+++ b/accel/tcg/tcg-accel-ops.h
@@ -13,10 +13,20 @@
 #define TCG_ACCEL_OPS_H
 
 #include "system/cpus.h"
+#include "hw/core/cpu.h"
 
 void tcg_cpu_destroy(CPUState *cpu);
 int tcg_cpu_exec(CPUState *cpu);
 void tcg_handle_interrupt(CPUState *cpu, int mask);
 void tcg_cpu_init_cflags(CPUState *cpu, bool parallel);
 
+#ifdef CONFIG_USER_ONLY
+#define tcg_cpus_queue cpus_queue
+#else
+/* Guard with qemu_cpu_list_lock */
+extern CPUTailQ tcg_cpus_queue;
+#endif
+
+#define CPU_FOREACH_TCG(cpu) QTAILQ_FOREACH_RCU(cpu, &tcg_cpus_queue, node)
+
 #endif /* TCG_ACCEL_OPS_H */
diff --git a/accel/tcg/tcg-accel-ops.c b/accel/tcg/tcg-accel-ops.c
index 6e3f1fa92b2..1fb077f7b38 100644
--- a/accel/tcg/tcg-accel-ops.c
+++ b/accel/tcg/tcg-accel-ops.c
@@ -47,6 +47,13 @@
 
 /* common functionality among all TCG variants */
 
+CPUTailQ tcg_cpus_queue = QTAILQ_HEAD_INITIALIZER(tcg_cpus_queue);
+
+static CPUTailQ *tcg_get_cpus_queue(void)
+{
+    return &tcg_cpus_queue;
+}
+
 void tcg_cpu_init_cflags(CPUState *cpu, bool parallel)
 {
     uint32_t cflags;
@@ -199,6 +206,7 @@ static inline void tcg_remove_all_breakpoints(CPUState *cpu)
 
 static void tcg_accel_ops_init(AccelOpsClass *ops)
 {
+    ops->get_cpus_queue = tcg_get_cpus_queue;
     if (qemu_tcg_mttcg_enabled()) {
         ops->create_vcpu_thread = mttcg_start_vcpu_thread;
         ops->kick_vcpu_thread = mttcg_kick_vcpu_thread;
-- 
2.47.1

[RFC PATCH 4/7] accel/tcg: Use CPU_FOREACH_TCG()

[Philippe Mathieu-Daudé]

Only iterate over TCG vCPUs when running TCG specific code.

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/tcg/cputlb.c           |  7 ++++---
 accel/tcg/monitor.c          |  3 ++-
 accel/tcg/tb-maint.c         |  7 ++++---
 accel/tcg/tcg-accel-ops-rr.c | 10 +++++-----
 accel/tcg/tcg-accel-ops.c    |  8 ++++----
 5 files changed, 19 insertions(+), 16 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index b4ccf0cdcb7..06f34df808b 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -48,6 +48,7 @@
 #endif
 #include "tcg/tcg-ldst.h"
 #include "tcg/oversized-guest.h"
+#include "tcg-accel-ops.h"
 
 /* DEBUG defines, enable DEBUG_TLB_LOG to log to the CPU_LOG_MMU target */
 /* #define DEBUG_TLB */
@@ -368,7 +369,7 @@ static void flush_all_helper(CPUState *src, run_on_cpu_func fn,
 {
     CPUState *cpu;
 
-    CPU_FOREACH(cpu) {
+    CPU_FOREACH_TCG(cpu) {
         if (cpu != src) {
             async_run_on_cpu(cpu, fn, d);
         }
@@ -646,7 +647,7 @@ void tlb_flush_page_by_mmuidx_all_cpus_synced(CPUState *src_cpu,
         TLBFlushPageByMMUIdxData *d;
 
         /* Allocate a separate data block for each destination cpu.  */
-        CPU_FOREACH(dst_cpu) {
+        CPU_FOREACH_TCG(dst_cpu) {
             if (dst_cpu != src_cpu) {
                 d = g_new(TLBFlushPageByMMUIdxData, 1);
                 d->addr = addr;
@@ -839,7 +840,7 @@ void tlb_flush_range_by_mmuidx_all_cpus_synced(CPUState *src_cpu,
     d.bits = bits;
 
     /* Allocate a separate data block for each destination cpu.  */
-    CPU_FOREACH(dst_cpu) {
+    CPU_FOREACH_TCG(dst_cpu) {
         if (dst_cpu != src_cpu) {
             p = g_memdup(&d, sizeof(d));
             async_run_on_cpu(dst_cpu, tlb_flush_range_by_mmuidx_async_1,
diff --git a/accel/tcg/monitor.c b/accel/tcg/monitor.c
index ae1dbeb79f8..98bd937ae20 100644
--- a/accel/tcg/monitor.c
+++ b/accel/tcg/monitor.c
@@ -19,6 +19,7 @@
 #include "tcg/tcg.h"
 #include "internal-common.h"
 #include "tb-context.h"
+#include "tcg-accel-ops.h"
 
 
 static void dump_drift_info(GString *buf)
@@ -131,7 +132,7 @@ static void tlb_flush_counts(size_t *pfull, size_t *ppart, size_t *pelide)
     CPUState *cpu;
     size_t full = 0, part = 0, elide = 0;
 
-    CPU_FOREACH(cpu) {
+    CPU_FOREACH_TCG(cpu) {
         full += qatomic_read(&cpu->neg.tlb.c.full_flush_count);
         part += qatomic_read(&cpu->neg.tlb.c.part_flush_count);
         elide += qatomic_read(&cpu->neg.tlb.c.elide_flush_count);
diff --git a/accel/tcg/tb-maint.c b/accel/tcg/tb-maint.c
index 3f1bebf6ab5..8598c59654f 100644
--- a/accel/tcg/tb-maint.c
+++ b/accel/tcg/tb-maint.c
@@ -36,6 +36,7 @@
 #ifdef CONFIG_USER_ONLY
 #include "user/page-protection.h"
 #endif
+#include "tcg-accel-ops.h"
 
 
 /* List iterators for lists of tagged pointers in TranslationBlock. */
@@ -771,7 +772,7 @@ static void do_tb_flush(CPUState *cpu, run_on_cpu_data tb_flush_count)
     }
     did_flush = true;
 
-    CPU_FOREACH(cpu) {
+    CPU_FOREACH_TCG(cpu) {
         tcg_flush_jmp_cache(cpu);
     }
 
@@ -885,13 +886,13 @@ static void tb_jmp_cache_inval_tb(TranslationBlock *tb)
 
     if (tb_cflags(tb) & CF_PCREL) {
         /* A TB may be at any virtual address */
-        CPU_FOREACH(cpu) {
+        CPU_FOREACH_TCG(cpu) {
             tcg_flush_jmp_cache(cpu);
         }
     } else {
         uint32_t h = tb_jmp_cache_hash_func(tb->pc);
 
-        CPU_FOREACH(cpu) {
+        CPU_FOREACH_TCG(cpu) {
             CPUJumpCache *jc = cpu->tb_jmp_cache;
 
             if (qatomic_read(&jc->array[h].tb) == tb) {
diff --git a/accel/tcg/tcg-accel-ops-rr.c b/accel/tcg/tcg-accel-ops-rr.c
index 028b385af9a..e5ce285efb9 100644
--- a/accel/tcg/tcg-accel-ops-rr.c
+++ b/accel/tcg/tcg-accel-ops-rr.c
@@ -42,7 +42,7 @@ void rr_kick_vcpu_thread(CPUState *unused)
 {
     CPUState *cpu;
 
-    CPU_FOREACH(cpu) {
+    CPU_FOREACH_TCG(cpu) {
         cpu_exit(cpu);
     };
 }
@@ -116,7 +116,7 @@ static void rr_wait_io_event(void)
 
     rr_start_kick_timer();
 
-    CPU_FOREACH(cpu) {
+    CPU_FOREACH_TCG(cpu) {
         qemu_wait_io_event_common(cpu);
     }
 }
@@ -129,7 +129,7 @@ static void rr_deal_with_unplugged_cpus(void)
 {
     CPUState *cpu;
 
-    CPU_FOREACH(cpu) {
+    CPU_FOREACH_TCG(cpu) {
         if (cpu->unplug && !cpu_can_run(cpu)) {
             tcg_cpu_destroy(cpu);
             break;
@@ -160,7 +160,7 @@ static int rr_cpu_count(void)
 
     if (cpu_list_generation_id_get() != last_gen_id) {
         cpu_count = 0;
-        CPU_FOREACH(cpu) {
+        CPU_FOREACH_TCG(cpu) {
             ++cpu_count;
         }
         last_gen_id = cpu_list_generation_id_get();
@@ -201,7 +201,7 @@ static void *rr_cpu_thread_fn(void *arg)
         qemu_cond_wait_bql(first_cpu->halt_cond);
 
         /* process any pending work */
-        CPU_FOREACH(cpu) {
+        CPU_FOREACH_TCG(cpu) {
             current_cpu = cpu;
             qemu_wait_io_event_common(cpu);
         }
diff --git a/accel/tcg/tcg-accel-ops.c b/accel/tcg/tcg-accel-ops.c
index 1fb077f7b38..371bbaa0307 100644
--- a/accel/tcg/tcg-accel-ops.c
+++ b/accel/tcg/tcg-accel-ops.c
@@ -144,7 +144,7 @@ static int tcg_insert_breakpoint(CPUState *cs, int type, vaddr addr, vaddr len)
     switch (type) {
     case GDB_BREAKPOINT_SW:
     case GDB_BREAKPOINT_HW:
-        CPU_FOREACH(cpu) {
+        CPU_FOREACH_TCG(cpu) {
             err = cpu_breakpoint_insert(cpu, addr, BP_GDB, NULL);
             if (err) {
                 break;
@@ -154,7 +154,7 @@ static int tcg_insert_breakpoint(CPUState *cs, int type, vaddr addr, vaddr len)
     case GDB_WATCHPOINT_WRITE:
     case GDB_WATCHPOINT_READ:
     case GDB_WATCHPOINT_ACCESS:
-        CPU_FOREACH(cpu) {
+        CPU_FOREACH_TCG(cpu) {
             err = cpu_watchpoint_insert(cpu, addr, len,
                                         xlat_gdb_type(cpu, type), NULL);
             if (err) {
@@ -175,7 +175,7 @@ static int tcg_remove_breakpoint(CPUState *cs, int type, vaddr addr, vaddr len)
     switch (type) {
     case GDB_BREAKPOINT_SW:
     case GDB_BREAKPOINT_HW:
-        CPU_FOREACH(cpu) {
+        CPU_FOREACH_TCG(cpu) {
             err = cpu_breakpoint_remove(cpu, addr, BP_GDB);
             if (err) {
                 break;
@@ -185,7 +185,7 @@ static int tcg_remove_breakpoint(CPUState *cs, int type, vaddr addr, vaddr len)
     case GDB_WATCHPOINT_WRITE:
     case GDB_WATCHPOINT_READ:
     case GDB_WATCHPOINT_ACCESS:
-        CPU_FOREACH(cpu) {
+        CPU_FOREACH_TCG(cpu) {
             err = cpu_watchpoint_remove(cpu, addr, len,
                                         xlat_gdb_type(cpu, type));
             if (err) {
-- 
2.47.1

[RFC PATCH 5/7] accel/hw: Implement hw_accel_get_cpus_queue()

[Philippe Mathieu-Daudé]

We can only run a single hardware accelerator at a time,
so add a generic hw_accel_get_cpus_queue() helper in
accel-system.c, a file common to all HW accelerators.

Register AccelOpsClass::get_cpus_queue() for each HW
accelerator. Add a generic CPU_FOREACH_HWACCEL() macro.

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 include/system/hw_accel.h         | 9 +++++++++
 accel/accel-system.c              | 8 ++++++++
 accel/kvm/kvm-accel-ops.c         | 1 +
 accel/xen/xen-all.c               | 1 +
 target/i386/nvmm/nvmm-accel-ops.c | 1 +
 target/i386/whpx/whpx-accel-ops.c | 1 +
 6 files changed, 21 insertions(+)

diff --git a/include/system/hw_accel.h b/include/system/hw_accel.h
index 380e9e640b6..12664cac6f9 100644
--- a/include/system/hw_accel.h
+++ b/include/system/hw_accel.h
@@ -2,6 +2,7 @@
  * QEMU Hardware accelerators support
  *
  * Copyright 2016 Google, Inc.
+ * Copyright 2025 Linaro Ltd.
  *
  * This work is licensed under the terms of the GNU GPL, version 2 or later.
  * See the COPYING file in the top-level directory.
@@ -17,6 +18,14 @@
 #include "system/whpx.h"
 #include "system/nvmm.h"
 
+/* Guard with qemu_cpu_list_lock */
+extern CPUTailQ hw_accel_cpus_queue;
+
+#define CPU_FOREACH_HWACCEL(cpu) \
+            QTAILQ_FOREACH_RCU(cpu, &hw_accel_cpus_queue, node)
+
+CPUTailQ *hw_accel_get_cpus_queue(void);
+
 void cpu_synchronize_state(CPUState *cpu);
 void cpu_synchronize_post_reset(CPUState *cpu);
 void cpu_synchronize_post_init(CPUState *cpu);
diff --git a/accel/accel-system.c b/accel/accel-system.c
index a7596aef59d..60877ea7a28 100644
--- a/accel/accel-system.c
+++ b/accel/accel-system.c
@@ -27,9 +27,17 @@
 #include "qemu/accel.h"
 #include "hw/boards.h"
 #include "system/cpus.h"
+#include "system/hw_accel.h"
 #include "qemu/error-report.h"
 #include "accel-system.h"
 
+CPUTailQ hw_accel_cpus_queue = QTAILQ_HEAD_INITIALIZER(hw_accel_cpus_queue);
+
+CPUTailQ *hw_accel_get_cpus_queue(void)
+{
+    return &hw_accel_cpus_queue;
+}
+
 int accel_init_machine(AccelState *accel, MachineState *ms)
 {
     AccelClass *acc = ACCEL_GET_CLASS(accel);
diff --git a/accel/kvm/kvm-accel-ops.c b/accel/kvm/kvm-accel-ops.c
index a81e8f3b03b..5f4001860d5 100644
--- a/accel/kvm/kvm-accel-ops.c
+++ b/accel/kvm/kvm-accel-ops.c
@@ -93,6 +93,7 @@ static void kvm_accel_ops_class_init(ObjectClass *oc, void *data)
 {
     AccelOpsClass *ops = ACCEL_OPS_CLASS(oc);
 
+    ops->get_cpus_queue = hw_accel_get_cpus_queue;
     ops->create_vcpu_thread = kvm_start_vcpu_thread;
     ops->cpu_thread_is_idle = kvm_vcpu_thread_is_idle;
     ops->cpus_are_resettable = kvm_cpus_are_resettable;
diff --git a/accel/xen/xen-all.c b/accel/xen/xen-all.c
index 852e9fbe5fe..ac5ed2dfb80 100644
--- a/accel/xen/xen-all.c
+++ b/accel/xen/xen-all.c
@@ -150,6 +150,7 @@ static void xen_accel_ops_class_init(ObjectClass *oc, void *data)
 {
     AccelOpsClass *ops = ACCEL_OPS_CLASS(oc);
 
+    ops->get_cpus_queue = hw_accel_get_cpus_queue;
     ops->create_vcpu_thread = dummy_start_vcpu_thread;
 }
 
diff --git a/target/i386/nvmm/nvmm-accel-ops.c b/target/i386/nvmm/nvmm-accel-ops.c
index e7b56662fee..bb407c68e14 100644
--- a/target/i386/nvmm/nvmm-accel-ops.c
+++ b/target/i386/nvmm/nvmm-accel-ops.c
@@ -84,6 +84,7 @@ static void nvmm_accel_ops_class_init(ObjectClass *oc, void *data)
 {
     AccelOpsClass *ops = ACCEL_OPS_CLASS(oc);
 
+    ops->get_cpus_queue = hw_accel_get_cpus_queue;
     ops->create_vcpu_thread = nvmm_start_vcpu_thread;
     ops->kick_vcpu_thread = nvmm_kick_vcpu_thread;
 
diff --git a/target/i386/whpx/whpx-accel-ops.c b/target/i386/whpx/whpx-accel-ops.c
index ab2e014c9ea..191214ca81d 100644
--- a/target/i386/whpx/whpx-accel-ops.c
+++ b/target/i386/whpx/whpx-accel-ops.c
@@ -86,6 +86,7 @@ static void whpx_accel_ops_class_init(ObjectClass *oc, void *data)
 {
     AccelOpsClass *ops = ACCEL_OPS_CLASS(oc);
 
+    ops->get_cpus_queue = hw_accel_get_cpus_queue;
     ops->create_vcpu_thread = whpx_start_vcpu_thread;
     ops->kick_vcpu_thread = whpx_kick_vcpu_thread;
     ops->cpu_thread_is_idle = whpx_vcpu_thread_is_idle;
-- 
2.47.1

[RFC PATCH 6/7] accel/hvf: Use CPU_FOREACH_HVF()

[Philippe Mathieu-Daudé]

Only iterate over HVF vCPUs when running HVF specific code.

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 include/system/hvf_int.h  | 4 ++++
 accel/hvf/hvf-accel-ops.c | 9 +++++----
 target/arm/hvf/hvf.c      | 4 ++--
 3 files changed, 11 insertions(+), 6 deletions(-)

diff --git a/include/system/hvf_int.h b/include/system/hvf_int.h
index 42ae18433f0..3cf64faabd1 100644
--- a/include/system/hvf_int.h
+++ b/include/system/hvf_int.h
@@ -11,6 +11,8 @@
 #ifndef HVF_INT_H
 #define HVF_INT_H
 
+#include "system/hw_accel.h"
+
 #ifdef __aarch64__
 #include <Hypervisor/Hypervisor.h>
 typedef hv_vcpu_t hvf_vcpuid;
@@ -74,4 +76,6 @@ int hvf_put_registers(CPUState *);
 int hvf_get_registers(CPUState *);
 void hvf_kick_vcpu_thread(CPUState *cpu);
 
+#define CPU_FOREACH_HVF(cpu) CPU_FOREACH_HWACCEL(cpu)
+
 #endif
diff --git a/accel/hvf/hvf-accel-ops.c b/accel/hvf/hvf-accel-ops.c
index 945ba720513..bbbe2f8d45b 100644
--- a/accel/hvf/hvf-accel-ops.c
+++ b/accel/hvf/hvf-accel-ops.c
@@ -504,7 +504,7 @@ static int hvf_insert_breakpoint(CPUState *cpu, int type, vaddr addr, vaddr len)
         }
     }
 
-    CPU_FOREACH(cpu) {
+    CPU_FOREACH_HVF(cpu) {
         err = hvf_update_guest_debug(cpu);
         if (err) {
             return err;
@@ -543,7 +543,7 @@ static int hvf_remove_breakpoint(CPUState *cpu, int type, vaddr addr, vaddr len)
         }
     }
 
-    CPU_FOREACH(cpu) {
+    CPU_FOREACH_HVF(cpu) {
         err = hvf_update_guest_debug(cpu);
         if (err) {
             return err;
@@ -560,7 +560,7 @@ static void hvf_remove_all_breakpoints(CPUState *cpu)
     QTAILQ_FOREACH_SAFE(bp, &hvf_state->hvf_sw_breakpoints, entry, next) {
         if (hvf_arch_remove_sw_breakpoint(cpu, bp) != 0) {
             /* Try harder to find a CPU that currently sees the breakpoint. */
-            CPU_FOREACH(tmpcpu)
+            CPU_FOREACH_HVF(tmpcpu)
             {
                 if (hvf_arch_remove_sw_breakpoint(tmpcpu, bp) == 0) {
                     break;
@@ -572,7 +572,7 @@ static void hvf_remove_all_breakpoints(CPUState *cpu)
     }
     hvf_arch_remove_all_hw_breakpoints();
 
-    CPU_FOREACH(cpu) {
+    CPU_FOREACH_HVF(cpu) {
         hvf_update_guest_debug(cpu);
     }
 }
@@ -581,6 +581,7 @@ static void hvf_accel_ops_class_init(ObjectClass *oc, void *data)
 {
     AccelOpsClass *ops = ACCEL_OPS_CLASS(oc);
 
+    ops->get_cpus_queue = hw_accel_get_cpus_queue;
     ops->create_vcpu_thread = hvf_start_vcpu_thread;
     ops->kick_vcpu_thread = hvf_kick_vcpu_thread;
 
diff --git a/target/arm/hvf/hvf.c b/target/arm/hvf/hvf.c
index 0afd96018e0..13400ff0d5f 100644
--- a/target/arm/hvf/hvf.c
+++ b/target/arm/hvf/hvf.c
@@ -2269,10 +2269,10 @@ static void hvf_arch_set_traps(void)
 
     /* Check whether guest debugging is enabled for at least one vCPU; if it
      * is, enable exiting the guest on all vCPUs */
-    CPU_FOREACH(cpu) {
+    CPU_FOREACH_HVF(cpu) {
         should_enable_traps |= cpu->accel->guest_debug_enabled;
     }
-    CPU_FOREACH(cpu) {
+    CPU_FOREACH_HVF(cpu) {
         /* Set whether debug exceptions exit the guest */
         r = hv_vcpu_set_trap_debug_exceptions(cpu->accel->fd,
                                               should_enable_traps);
-- 
2.47.1

Re: [RFC PATCH 6/7] accel/hvf: Use CPU_FOREACH_HVF()

[Daniel Henrique Barboza]

On 1/6/25 5:02 PM, Philippe Mathieu-Daudé wrote:
> Only iterate over HVF vCPUs when running HVF specific code.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   include/system/hvf_int.h  | 4 ++++
>   accel/hvf/hvf-accel-ops.c | 9 +++++----
>   target/arm/hvf/hvf.c      | 4 ++--
>   3 files changed, 11 insertions(+), 6 deletions(-)
> 
> diff --git a/include/system/hvf_int.h b/include/system/hvf_int.h
> index 42ae18433f0..3cf64faabd1 100644
> --- a/include/system/hvf_int.h
> +++ b/include/system/hvf_int.h
> @@ -11,6 +11,8 @@
>   #ifndef HVF_INT_H
>   #define HVF_INT_H
>   
> +#include "system/hw_accel.h"
> +
>   #ifdef __aarch64__
>   #include <Hypervisor/Hypervisor.h>
>   typedef hv_vcpu_t hvf_vcpuid;
> @@ -74,4 +76,6 @@ int hvf_put_registers(CPUState *);
>   int hvf_get_registers(CPUState *);
>   void hvf_kick_vcpu_thread(CPUState *cpu);
>   
> +#define CPU_FOREACH_HVF(cpu) CPU_FOREACH_HWACCEL(cpu)


Cosmetic comment: given that this is HVF specific code and we only support one hw
accelerator at a time, I'd skip this alias and use CPU_FOREACH_HWACCEL(cpu) directly.
It would make it easier when grepping to see where and how the macro is being used.
Same thing in the next patch with CPU_FOREACH_KVM().


LGTM otherwise. Thanks,

Daniel


> +
>   #endif
> diff --git a/accel/hvf/hvf-accel-ops.c b/accel/hvf/hvf-accel-ops.c
> index 945ba720513..bbbe2f8d45b 100644
> --- a/accel/hvf/hvf-accel-ops.c
> +++ b/accel/hvf/hvf-accel-ops.c
> @@ -504,7 +504,7 @@ static int hvf_insert_breakpoint(CPUState *cpu, int type, vaddr addr, vaddr len)
>           }
>       }
>   
> -    CPU_FOREACH(cpu) {
> +    CPU_FOREACH_HVF(cpu) {
>           err = hvf_update_guest_debug(cpu);
>           if (err) {
>               return err;
> @@ -543,7 +543,7 @@ static int hvf_remove_breakpoint(CPUState *cpu, int type, vaddr addr, vaddr len)
>           }
>       }
>   
> -    CPU_FOREACH(cpu) {
> +    CPU_FOREACH_HVF(cpu) {
>           err = hvf_update_guest_debug(cpu);
>           if (err) {
>               return err;
> @@ -560,7 +560,7 @@ static void hvf_remove_all_breakpoints(CPUState *cpu)
>       QTAILQ_FOREACH_SAFE(bp, &hvf_state->hvf_sw_breakpoints, entry, next) {
>           if (hvf_arch_remove_sw_breakpoint(cpu, bp) != 0) {
>               /* Try harder to find a CPU that currently sees the breakpoint. */
> -            CPU_FOREACH(tmpcpu)
> +            CPU_FOREACH_HVF(tmpcpu)
>               {
>                   if (hvf_arch_remove_sw_breakpoint(tmpcpu, bp) == 0) {
>                       break;
> @@ -572,7 +572,7 @@ static void hvf_remove_all_breakpoints(CPUState *cpu)
>       }
>       hvf_arch_remove_all_hw_breakpoints();
>   
> -    CPU_FOREACH(cpu) {
> +    CPU_FOREACH_HVF(cpu) {
>           hvf_update_guest_debug(cpu);
>       }
>   }
> @@ -581,6 +581,7 @@ static void hvf_accel_ops_class_init(ObjectClass *oc, void *data)
>   {
>       AccelOpsClass *ops = ACCEL_OPS_CLASS(oc);
>   
> +    ops->get_cpus_queue = hw_accel_get_cpus_queue;
>       ops->create_vcpu_thread = hvf_start_vcpu_thread;
>       ops->kick_vcpu_thread = hvf_kick_vcpu_thread;
>   
> diff --git a/target/arm/hvf/hvf.c b/target/arm/hvf/hvf.c
> index 0afd96018e0..13400ff0d5f 100644
> --- a/target/arm/hvf/hvf.c
> +++ b/target/arm/hvf/hvf.c
> @@ -2269,10 +2269,10 @@ static void hvf_arch_set_traps(void)
>   
>       /* Check whether guest debugging is enabled for at least one vCPU; if it
>        * is, enable exiting the guest on all vCPUs */
> -    CPU_FOREACH(cpu) {
> +    CPU_FOREACH_HVF(cpu) {
>           should_enable_traps |= cpu->accel->guest_debug_enabled;
>       }
> -    CPU_FOREACH(cpu) {
> +    CPU_FOREACH_HVF(cpu) {
>           /* Set whether debug exceptions exit the guest */
>           r = hv_vcpu_set_trap_debug_exceptions(cpu->accel->fd,
>                                                 should_enable_traps);

Re: [RFC PATCH 6/7] accel/hvf: Use CPU_FOREACH_HVF()

[Philippe Mathieu-Daudé]

On 6/1/25 21:33, Daniel Henrique Barboza wrote:
> 
> 
> On 1/6/25 5:02 PM, Philippe Mathieu-Daudé wrote:
>> Only iterate over HVF vCPUs when running HVF specific code.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
>> ---
>>   include/system/hvf_int.h  | 4 ++++
>>   accel/hvf/hvf-accel-ops.c | 9 +++++----
>>   target/arm/hvf/hvf.c      | 4 ++--
>>   3 files changed, 11 insertions(+), 6 deletions(-)
>>
>> diff --git a/include/system/hvf_int.h b/include/system/hvf_int.h
>> index 42ae18433f0..3cf64faabd1 100644
>> --- a/include/system/hvf_int.h
>> +++ b/include/system/hvf_int.h
>> @@ -11,6 +11,8 @@
>>   #ifndef HVF_INT_H
>>   #define HVF_INT_H
>> +#include "system/hw_accel.h"
>> +
>>   #ifdef __aarch64__
>>   #include <Hypervisor/Hypervisor.h>
>>   typedef hv_vcpu_t hvf_vcpuid;
>> @@ -74,4 +76,6 @@ int hvf_put_registers(CPUState *);
>>   int hvf_get_registers(CPUState *);
>>   void hvf_kick_vcpu_thread(CPUState *cpu);
>> +#define CPU_FOREACH_HVF(cpu) CPU_FOREACH_HWACCEL(cpu)
> 
> 
> Cosmetic comment: given that this is HVF specific code and we only 
> support one hw
> accelerator at a time, I'd skip this alias and use 
> CPU_FOREACH_HWACCEL(cpu) directly.
> It would make it easier when grepping to see where and how the macro is 
> being used.

I find it more useful to grep for a particular accelerator, or for
all of them:

$ git grep CPU_FOREACH_
accel/hvf/hvf-accel-ops.c:507:    CPU_FOREACH_HVF(cpu) {
accel/hvf/hvf-accel-ops.c:546:    CPU_FOREACH_HVF(cpu) {
accel/kvm/kvm-all.c:875:        CPU_FOREACH_KVM(cpu) {
accel/kvm/kvm-all.c:938:    CPU_FOREACH_KVM(cpu) {
accel/tcg/cputlb.c:372:    CPU_FOREACH_TCG(cpu) {
accel/tcg/cputlb.c:650:        CPU_FOREACH_TCG(dst_cpu) {

Re: [RFC PATCH 6/7] accel/hvf: Use CPU_FOREACH_HVF()

[BALATON Zoltan]

[-- Attachment #1: Type: text/plain, Size: 2130 bytes --]

On Mon, 6 Jan 2025, Philippe Mathieu-Daudé wrote:
> On 6/1/25 21:33, Daniel Henrique Barboza wrote:
>> 
>> 
>> On 1/6/25 5:02 PM, Philippe Mathieu-Daudé wrote:
>>> Only iterate over HVF vCPUs when running HVF specific code.
>>> 
>>> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
>>> ---
>>>   include/system/hvf_int.h  | 4 ++++
>>>   accel/hvf/hvf-accel-ops.c | 9 +++++----
>>>   target/arm/hvf/hvf.c      | 4 ++--
>>>   3 files changed, 11 insertions(+), 6 deletions(-)
>>> 
>>> diff --git a/include/system/hvf_int.h b/include/system/hvf_int.h
>>> index 42ae18433f0..3cf64faabd1 100644
>>> --- a/include/system/hvf_int.h
>>> +++ b/include/system/hvf_int.h
>>> @@ -11,6 +11,8 @@
>>>   #ifndef HVF_INT_H
>>>   #define HVF_INT_H
>>> +#include "system/hw_accel.h"
>>> +
>>>   #ifdef __aarch64__
>>>   #include <Hypervisor/Hypervisor.h>
>>>   typedef hv_vcpu_t hvf_vcpuid;
>>> @@ -74,4 +76,6 @@ int hvf_put_registers(CPUState *);
>>>   int hvf_get_registers(CPUState *);
>>>   void hvf_kick_vcpu_thread(CPUState *cpu);
>>> +#define CPU_FOREACH_HVF(cpu) CPU_FOREACH_HWACCEL(cpu)
>> 
>> 
>> Cosmetic comment: given that this is HVF specific code and we only support 
>> one hw
>> accelerator at a time, I'd skip this alias and use CPU_FOREACH_HWACCEL(cpu) 
>> directly.
>> It would make it easier when grepping to see where and how the macro is 
>> being used.
>
> I find it more useful to grep for a particular accelerator, or for
> all of them:
>
> $ git grep CPU_FOREACH_
> accel/hvf/hvf-accel-ops.c:507:    CPU_FOREACH_HVF(cpu) {
> accel/hvf/hvf-accel-ops.c:546:    CPU_FOREACH_HVF(cpu) {
> accel/kvm/kvm-all.c:875:        CPU_FOREACH_KVM(cpu) {
> accel/kvm/kvm-all.c:938:    CPU_FOREACH_KVM(cpu) {
> accel/tcg/cputlb.c:372:    CPU_FOREACH_TCG(cpu) {
> accel/tcg/cputlb.c:650:        CPU_FOREACH_TCG(dst_cpu) {

But then you need to define a new macro for every new accelerator. Maybe 
it's simpler to have CPU_FOREACH take the queue as a parameter so no 
separate macro is needed for each accel (and they cannot get inconsistent 
by changing only one of them in the future).

Regards,
BALATON Zoltan

[RFC PATCH 7/7] accel/kvm: Use CPU_FOREACH_KVM()

[Philippe Mathieu-Daudé]

Only iterate over KVM vCPUs when running KVM specific code.

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 include/system/kvm_int.h         |  3 +++
 accel/kvm/kvm-all.c              | 14 +++++++-------
 hw/i386/kvm/clock.c              |  3 ++-
 hw/intc/spapr_xive_kvm.c         |  5 +++--
 hw/intc/xics_kvm.c               |  5 +++--
 target/i386/kvm/kvm.c            |  4 ++--
 target/i386/kvm/xen-emu.c        |  2 +-
 target/s390x/kvm/kvm.c           |  2 +-
 target/s390x/kvm/stsi-topology.c |  3 ++-
 9 files changed, 24 insertions(+), 17 deletions(-)

diff --git a/include/system/kvm_int.h b/include/system/kvm_int.h
index 4de6106869b..0ef4c336b18 100644
--- a/include/system/kvm_int.h
+++ b/include/system/kvm_int.h
@@ -13,6 +13,7 @@
 #include "qapi/qapi-types-common.h"
 #include "qemu/accel.h"
 #include "qemu/queue.h"
+#include "system/hw_accel.h"
 #include "system/kvm.h"
 #include "hw/boards.h"
 #include "hw/i386/topology.h"
@@ -168,6 +169,8 @@ struct KVMState
     char *device;
 };
 
+#define CPU_FOREACH_KVM(cpu) CPU_FOREACH_HWACCEL(cpu)
+
 void kvm_memory_listener_register(KVMState *s, KVMMemoryListener *kml,
                                   AddressSpace *as, int as_id, const char *name);
 
diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c
index c65b790433c..9b26b286865 100644
--- a/accel/kvm/kvm-all.c
+++ b/accel/kvm/kvm-all.c
@@ -872,7 +872,7 @@ static uint64_t kvm_dirty_ring_reap_locked(KVMState *s, CPUState* cpu)
     if (cpu) {
         total = kvm_dirty_ring_reap_one(s, cpu);
     } else {
-        CPU_FOREACH(cpu) {
+        CPU_FOREACH_KVM(cpu) {
             total += kvm_dirty_ring_reap_one(s, cpu);
         }
     }
@@ -935,7 +935,7 @@ static void kvm_cpu_synchronize_kick_all(void)
 {
     CPUState *cpu;
 
-    CPU_FOREACH(cpu) {
+    CPU_FOREACH_KVM(cpu) {
         run_on_cpu(cpu, do_kvm_cpu_synchronize_kick, RUN_ON_CPU_NULL);
     }
 }
@@ -3535,7 +3535,7 @@ int kvm_insert_breakpoint(CPUState *cpu, int type, vaddr addr, vaddr len)
         }
     }
 
-    CPU_FOREACH(cpu) {
+    CPU_FOREACH_KVM(cpu) {
         err = kvm_update_guest_debug(cpu, 0);
         if (err) {
             return err;
@@ -3574,7 +3574,7 @@ int kvm_remove_breakpoint(CPUState *cpu, int type, vaddr addr, vaddr len)
         }
     }
 
-    CPU_FOREACH(cpu) {
+    CPU_FOREACH_KVM(cpu) {
         err = kvm_update_guest_debug(cpu, 0);
         if (err) {
             return err;
@@ -3592,7 +3592,7 @@ void kvm_remove_all_breakpoints(CPUState *cpu)
     QTAILQ_FOREACH_SAFE(bp, &s->kvm_sw_breakpoints, entry, next) {
         if (kvm_arch_remove_sw_breakpoint(cpu, bp) != 0) {
             /* Try harder to find a CPU that currently sees the breakpoint. */
-            CPU_FOREACH(tmpcpu) {
+            CPU_FOREACH_KVM(tmpcpu) {
                 if (kvm_arch_remove_sw_breakpoint(tmpcpu, bp) == 0) {
                     break;
                 }
@@ -3603,7 +3603,7 @@ void kvm_remove_all_breakpoints(CPUState *cpu)
     }
     kvm_arch_remove_all_hw_breakpoints();
 
-    CPU_FOREACH(cpu) {
+    CPU_FOREACH_KVM(cpu) {
         kvm_update_guest_debug(cpu, 0);
     }
 }
@@ -4384,7 +4384,7 @@ static void query_stats_cb(StatsResultList **result, StatsTarget target,
         stats_args.result.stats = result;
         stats_args.names = names;
         stats_args.errp = errp;
-        CPU_FOREACH(cpu) {
+        CPU_FOREACH_KVM(cpu) {
             if (!apply_str_list_filter(cpu->parent_obj.canonical_path, targets)) {
                 continue;
             }
diff --git a/hw/i386/kvm/clock.c b/hw/i386/kvm/clock.c
index 63be5088420..f2638cf2c22 100644
--- a/hw/i386/kvm/clock.c
+++ b/hw/i386/kvm/clock.c
@@ -17,6 +17,7 @@
 #include "qemu/host-utils.h"
 #include "qemu/module.h"
 #include "system/kvm.h"
+#include "system/kvm_int.h"
 #include "system/runstate.h"
 #include "system/hw_accel.h"
 #include "kvm/kvm_i386.h"
@@ -196,7 +197,7 @@ static void kvmclock_vm_state_change(void *opaque, bool running,
         if (!cap_clock_ctrl) {
             return;
         }
-        CPU_FOREACH(cpu) {
+        CPU_FOREACH_KVM(cpu) {
             run_on_cpu(cpu, do_kvmclock_ctrl, RUN_ON_CPU_NULL);
         }
     } else {
diff --git a/hw/intc/spapr_xive_kvm.c b/hw/intc/spapr_xive_kvm.c
index 26d30b41c15..08354f08512 100644
--- a/hw/intc/spapr_xive_kvm.c
+++ b/hw/intc/spapr_xive_kvm.c
@@ -14,6 +14,7 @@
 #include "target/ppc/cpu.h"
 #include "system/cpus.h"
 #include "system/kvm.h"
+#include "system/kvm_int.h"
 #include "system/runstate.h"
 #include "hw/ppc/spapr.h"
 #include "hw/ppc/spapr_cpu_core.h"
@@ -678,7 +679,7 @@ int kvmppc_xive_post_load(SpaprXive *xive, int version_id)
      * 'post_load' handler of XiveTCTX because the machine is not
      * necessarily connected to the KVM device at that time.
      */
-    CPU_FOREACH(cs) {
+    CPU_FOREACH_KVM(cs) {
         PowerPCCPU *cpu = POWERPC_CPU(cs);
 
         ret = kvmppc_xive_cpu_set_state(spapr_cpu_state(cpu)->tctx, &local_err);
@@ -795,7 +796,7 @@ int kvmppc_xive_connect(SpaprInterruptController *intc, uint32_t nr_servers,
         kvmppc_xive_change_state_handler, xive);
 
     /* Connect the presenters to the initial VCPUs of the machine */
-    CPU_FOREACH(cs) {
+    CPU_FOREACH_KVM(cs) {
         PowerPCCPU *cpu = POWERPC_CPU(cs);
 
         ret = kvmppc_xive_cpu_connect(spapr_cpu_state(cpu)->tctx, errp);
diff --git a/hw/intc/xics_kvm.c b/hw/intc/xics_kvm.c
index ee72969f5f1..aed2ad44363 100644
--- a/hw/intc/xics_kvm.c
+++ b/hw/intc/xics_kvm.c
@@ -29,6 +29,7 @@
 #include "qapi/error.h"
 #include "trace.h"
 #include "system/kvm.h"
+#include "system/kvm_int.h"
 #include "hw/ppc/spapr.h"
 #include "hw/ppc/spapr_cpu_core.h"
 #include "hw/ppc/xics.h"
@@ -418,7 +419,7 @@ int xics_kvm_connect(SpaprInterruptController *intc, uint32_t nr_servers,
     kvm_gsi_direct_mapping = true;
 
     /* Create the presenters */
-    CPU_FOREACH(cs) {
+    CPU_FOREACH_KVM(cs) {
         PowerPCCPU *cpu = POWERPC_CPU(cs);
 
         icp_kvm_realize(DEVICE(spapr_cpu_state(cpu)->icp), &local_err);
@@ -434,7 +435,7 @@ int xics_kvm_connect(SpaprInterruptController *intc, uint32_t nr_servers,
     }
 
     /* Connect the presenters to the initial VCPUs of the machine */
-    CPU_FOREACH(cs) {
+    CPU_FOREACH_KVM(cs) {
         PowerPCCPU *cpu = POWERPC_CPU(cs);
         icp_set_kvm_state(spapr_cpu_state(cpu)->icp, &local_err);
         if (local_err) {
diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index 2f66e63b880..437911d6c6a 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -329,7 +329,7 @@ void kvm_synchronize_all_tsc(void)
     CPUState *cpu;
 
     if (kvm_enabled()) {
-        CPU_FOREACH(cpu) {
+        CPU_FOREACH_KVM(cpu) {
             run_on_cpu(cpu, do_kvm_synchronize_tsc, RUN_ON_CPU_NULL);
         }
     }
@@ -2847,7 +2847,7 @@ static void *kvm_msr_energy_thread(void *data)
          * Identify the vcpu threads
          * Calculate the number of vcpu per package
          */
-        CPU_FOREACH(cpu) {
+        CPU_FOREACH_KVM(cpu) {
             for (int i = 0; i < num_threads; i++) {
                 if (cpu->thread_id == thd_stat[i].thread_id) {
                     thd_stat[i].is_vcpu = true;
diff --git a/target/i386/kvm/xen-emu.c b/target/i386/kvm/xen-emu.c
index e81a2458812..36ae9c11252 100644
--- a/target/i386/kvm/xen-emu.c
+++ b/target/i386/kvm/xen-emu.c
@@ -1422,7 +1422,7 @@ int kvm_xen_soft_reset(void)
         return err;
     }
 
-    CPU_FOREACH(cpu) {
+    CPU_FOREACH_KVM(cpu) {
         async_run_on_cpu(cpu, do_vcpu_soft_reset, RUN_ON_CPU_NULL);
     }
 
diff --git a/target/s390x/kvm/kvm.c b/target/s390x/kvm/kvm.c
index db645a48133..a02e78ce807 100644
--- a/target/s390x/kvm/kvm.c
+++ b/target/s390x/kvm/kvm.c
@@ -1559,7 +1559,7 @@ static void handle_diag_318(S390CPU *cpu, struct kvm_run *run)
         return;
     }
 
-    CPU_FOREACH(t) {
+    CPU_FOREACH_KVM(t) {
         run_on_cpu(t, s390_do_cpu_set_diag318,
                    RUN_ON_CPU_HOST_ULONG(diag318_info));
     }
diff --git a/target/s390x/kvm/stsi-topology.c b/target/s390x/kvm/stsi-topology.c
index c8d6389cd87..cf1a9b5d218 100644
--- a/target/s390x/kvm/stsi-topology.c
+++ b/target/s390x/kvm/stsi-topology.c
@@ -10,6 +10,7 @@
 #include "cpu.h"
 #include "hw/s390x/sclp.h"
 #include "hw/s390x/cpu-topology.h"
+#include "system/kvm_int.h"
 
 QEMU_BUILD_BUG_ON(S390_CPU_ENTITLEMENT_LOW != 1);
 QEMU_BUILD_BUG_ON(S390_CPU_ENTITLEMENT_MEDIUM != 2);
@@ -256,7 +257,7 @@ static void s390_topology_fill_list_sorted(S390TopologyList *topology_list)
 
     QTAILQ_INSERT_HEAD(topology_list, &sentinel, next);
 
-    CPU_FOREACH(cs) {
+    CPU_FOREACH_KVM(cs) {
         S390TopologyId id = s390_topology_from_cpu(S390_CPU(cs));
         S390TopologyEntry *entry = NULL, *tmp;
 
-- 
2.47.1

Re: [RFC PATCH 0/7] accel: Add per-accelerator vCPUs queue

[Philippe Mathieu-Daudé]

Cc'ing Pierrick & Salil.

On 6/1/25 21:02, Philippe Mathieu-Daudé wrote:
> Hi,
> 
> Currently we register all vCPUs to the global 'cpus_queue' queue,
> however we can not discriminate per accelerator or per target
> architecture (which might happen in a soon future).
> 
> This series tries to add an accelerator discriminator, so
> accelerator specific code can iterate on its own vCPUs. This
> is required to run a pair of HW + SW accelerators like the
> (HVF, TCG) or (KVM, TCG) combinations. Otherwise, i.e. the
> HVF core code could iterate on TCG vCPUs...
> To keep it simple and not refactor heavily the code base,
> we introduce the CPU_FOREACH_TCG/HVF/KVM() macros, only
> defined for each accelerator.
> 
> This is just a RFC to get some thoughts whether this is
> heading in the correct direction or not ;)
> 
> Regards,
> 
> Phil.
> 
> Philippe Mathieu-Daudé (7):
>    cpus: Restrict CPU_FOREACH_SAFE() to user emulation
>    cpus: Introduce AccelOpsClass::get_cpus_queue()
>    accel/tcg: Implement tcg_get_cpus_queue()
>    accel/tcg: Use CPU_FOREACH_TCG()
>    accel/hw: Implement hw_accel_get_cpus_queue()
>    accel/hvf: Use CPU_FOREACH_HVF()
>    accel/kvm: Use CPU_FOREACH_KVM()
> 
>   accel/tcg/tcg-accel-ops.h         | 10 ++++++++++
>   include/hw/core/cpu.h             | 11 +++++++++++
>   include/system/accel-ops.h        |  6 ++++++
>   include/system/hvf_int.h          |  4 ++++
>   include/system/hw_accel.h         |  9 +++++++++
>   include/system/kvm_int.h          |  3 +++
>   accel/accel-system.c              |  8 ++++++++
>   accel/hvf/hvf-accel-ops.c         |  9 +++++----
>   accel/kvm/kvm-accel-ops.c         |  1 +
>   accel/kvm/kvm-all.c               | 14 +++++++-------
>   accel/tcg/cputlb.c                |  7 ++++---
>   accel/tcg/monitor.c               |  3 ++-
>   accel/tcg/tb-maint.c              |  7 ++++---
>   accel/tcg/tcg-accel-ops-rr.c      | 10 +++++-----
>   accel/tcg/tcg-accel-ops.c         | 16 ++++++++++++----
>   accel/tcg/user-exec-stub.c        |  5 +++++
>   accel/xen/xen-all.c               |  1 +
>   cpu-common.c                      | 10 ++++++++++
>   hw/i386/kvm/clock.c               |  3 ++-
>   hw/intc/spapr_xive_kvm.c          |  5 +++--
>   hw/intc/xics_kvm.c                |  5 +++--
>   system/cpus.c                     |  5 +++++
>   target/arm/hvf/hvf.c              |  4 ++--
>   target/i386/kvm/kvm.c             |  4 ++--
>   target/i386/kvm/xen-emu.c         |  2 +-
>   target/i386/nvmm/nvmm-accel-ops.c |  1 +
>   target/i386/whpx/whpx-accel-ops.c |  1 +
>   target/s390x/kvm/kvm.c            |  2 +-
>   target/s390x/kvm/stsi-topology.c  |  3 ++-
>   29 files changed, 130 insertions(+), 39 deletions(-)
>