[RFC PATCH RESEND 00/42] accel/split/arm: Run EL2 using TCG and EL1/EL0 in hardware with HVF

[RFC PATCH RESEND 00/42] accel/split/arm: Run EL2 using TCG and EL1/EL0 in hardware with HVF

[Philippe Mathieu-Daudé]

Hi,

This RFC is a proof-of-concept we can have QEMU run both software
emulator (TCG) and hardware accelerator (here HVF). Unfortunately
I'm exhausted so I'll let Alex explain what this is about,
otherwise I'll post the real cover letter next Monday after
getting some rest.

Full work available in the split-accel-rfc-v5 tag:
https://gitlab.com/philmd/qemu/-/commits/split-accel-rfc-v5

Regards,

Phil.

Julian Armistead (1):
  accel/split: Minimal stubs for split accelerator

Philippe Mathieu-Daudé (41):
  accel/split: Define SplitAccelState
  accel/split: Implement cpus_are_resettable()
  accel/split: Implement accel_init_machine()
  accel/split: Expose 'hw' and 'sw' properties
  accel/split: Empty setup_post()
  accel/split: Implement supports_guest_debug()
  accel/split: Implement gdbstub_supported_sstep_flags()
  accel/split: Add cpu_thread_routine() stub
  accel/split: Define and allocate AccelCPUState
  accel/split: Register MTTCG
  accel/split: Have thread routine ready to dispatch over HW/SW
  accel/split: Implement cpu_reset_hold()
  accel/split: Implement synchronize_post_init()
  accel/split: Implement synchronize_pre_resume()
  accel/split: Implement synchronize_state()
  accel/split: Implement synchronize_post_reset()
  accel/split: Implement synchronize_pre_loadvm()
  accel/split: Implement kick_vcpu_thread()
  accel/split: Implement cpu_common_realize()
  accel/split: Set use_hw in cpu_thread_routine() and switch over
  accel/split: Add few trace events in cpu_thread_routine handler
  target/arm: Implement SysemuCPUOps::can_accelerate() handler
  accel/split: Implement handle_interrupt()
  accel/split: Empty ops_init()
  accel/split: Empty set/get_virtual_clock()
  accel/split: Empty get_elapsed_ticks()
  accel/split: Empty cpu_thread_is_idle()
  accel/split: Kludge qemu_tcg_mttcg_enabled()
  accel/split: Implement remove_all_breakpoints()
  accel/split: Implement remove_breakpoint()
  accel/split: Implement insert_breakpoint()
  accel/split: Implement update_guest_debug()
  accel/split: Implement get_[vcpu]_stats()
  target/arm: Emulate EL2 under TCG
  target/arm: Have ERET switch to hw accel for EL0/EL1
  accel/hvf: Emulate HVC at EL2
  accel/tcg: Introduce TCGCPUOps::rebuild_tb_hflags handler
  target/arm: Implement TCGCPUOps::rebuild_tb_hflags()
  accel/split: Call TCGCPUOps::rebuild_tb_hflags()
  tests/functional: Add split_available() helper
  tests/functional: Test Aarch64 virt machine with split-accelerator

 meson.build                                 |   1 +
 accel/split/split-accel.h                   |  54 +++
 accel/split/trace.h                         |   2 +
 accel/tcg/internal-common.h                 |   1 +
 include/accel/tcg/cpu-ops.h                 |   2 +
 include/exec/cpu-common.h                   |   1 +
 include/system/accel-ops.h                  |   8 +
 target/arm/cpu.h                            |   2 +
 accel/split/split-accel-ops.c               | 381 ++++++++++++++++++++
 accel/split/split-all.c                     | 196 ++++++++++
 accel/tcg/cpu-exec.c                        |   9 +
 accel/tcg/tcg-accel-ops.c                   |   1 +
 accel/tcg/tcg-all.c                         |   1 +
 system/vl.c                                 |   4 +
 target/arm/cpu.c                            |   9 +
 target/arm/cpu64.c                          |   7 +-
 target/arm/hvf/hvf.c                        |   6 +
 target/arm/tcg/cpu-v7m.c                    |   1 +
 target/arm/tcg/helper-a64.c                 |   9 +
 target/arm/tcg/hflags.c                     |   5 +
 accel/Kconfig                               |   6 +
 accel/meson.build                           |   1 +
 accel/split/meson.build                     |   9 +
 accel/split/trace-events                    |   9 +
 python/qemu/utils/__init__.py               |   3 +-
 python/qemu/utils/accel.py                  |  10 +
 tests/functional/meson.build                |   1 +
 tests/functional/qemu_test/testcase.py      |   4 +-
 tests/functional/test_aarch64_virt_split.py |  69 ++++
 29 files changed, 808 insertions(+), 4 deletions(-)
 create mode 100644 accel/split/split-accel.h
 create mode 100644 accel/split/trace.h
 create mode 100644 accel/split/split-accel-ops.c
 create mode 100644 accel/split/split-all.c
 create mode 100644 accel/split/meson.build
 create mode 100644 accel/split/trace-events
 create mode 100644 tests/functional/test_aarch64_virt_split.py

-- 
2.49.0

[RFC PATCH RESEND 01/42] accel/split: Minimal stubs for split accelerator

[Philippe Mathieu-Daudé]

From: Julian Armistead <julian.armistead@linaro.org>

Signed-off-by: Julian Armistead <julian.armistead@linaro.org>
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel-ops.c | 161 ++++++++++++++++++++++++++++++++++
 accel/split/split-all.c       |  77 ++++++++++++++++
 system/vl.c                   |   4 +
 accel/Kconfig                 |   6 ++
 accel/meson.build             |   1 +
 accel/split/meson.build       |   9 ++
 6 files changed, 258 insertions(+)
 create mode 100644 accel/split/split-accel-ops.c
 create mode 100644 accel/split/split-all.c
 create mode 100644 accel/split/meson.build

diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
new file mode 100644
index 00000000000..e5c1d51d426
--- /dev/null
+++ b/accel/split/split-accel-ops.c
@@ -0,0 +1,161 @@
+/*
+ * QEMU "split" accelerator (HW + SW) ops
+ *
+ * Copyright (c) 2025 Linaro Ltd.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+#include "system/accel-ops.h"
+
+static void *split_cpu_thread_routine(void *arg)
+{
+    g_assert_not_reached();
+}
+
+static void split_ops_init(AccelClass *ac)
+{
+    g_assert_not_reached();
+}
+
+static bool split_cpu_common_realize(CPUState *cpu, Error **errp)
+{
+    g_assert_not_reached();
+}
+
+static void split_cpu_common_unrealize(CPUState *cpu)
+{
+    g_assert_not_reached();
+}
+
+static void split_cpu_reset_hold(CPUState *cpu)
+{
+    g_assert_not_reached();
+}
+
+static void split_kick_vcpu_thread(CPUState *cpu)
+{
+    g_assert_not_reached();
+}
+
+static bool split_cpu_thread_is_idle(CPUState *cpu)
+{
+    g_assert_not_reached();
+}
+
+static void split_synchronize_post_reset(CPUState *cpu)
+{
+    g_assert_not_reached();
+}
+
+static void split_synchronize_post_init(CPUState *cpu)
+{
+    g_assert_not_reached();
+}
+
+static void split_synchronize_state(CPUState *cpu)
+{
+    g_assert_not_reached();
+}
+
+static void split_synchronize_pre_loadvm(CPUState *cpu)
+{
+    g_assert_not_reached();
+}
+
+static void split_synchronize_pre_resume(bool step_pending)
+{
+    g_assert_not_reached();
+}
+
+static void split_handle_interrupt(CPUState *cpu, int mask)
+{
+    g_assert_not_reached();
+}
+
+static int64_t split_get_virtual_clock(void)
+{
+    g_assert_not_reached();
+}
+
+static void split_set_virtual_clock(int64_t time)
+{
+    g_assert_not_reached();
+}
+
+static int64_t split_get_elapsed_ticks(void)
+{
+    g_assert_not_reached();
+}
+
+static int split_update_guest_debug(CPUState *cpu)
+{
+    g_assert_not_reached();
+}
+
+static int split_insert_breakpoint(CPUState *cpu, int type,
+                                   vaddr addr, vaddr len)
+{
+    g_assert_not_reached();
+}
+
+static int split_remove_breakpoint(CPUState *cpu, int type,
+                                   vaddr addr, vaddr len)
+{
+    g_assert_not_reached();
+}
+
+static void split_remove_all_breakpoints(CPUState *cpu)
+{
+    g_assert_not_reached();
+}
+
+static void split_get_vcpu_stats(CPUState *cpu, GString *buf)
+{
+    g_assert_not_reached();
+}
+
+static void split_accel_ops_class_init(ObjectClass *oc, const void *data)
+{
+    AccelOpsClass *ops = ACCEL_OPS_CLASS(oc);
+
+    ops->ops_init = split_ops_init;
+    ops->cpu_common_realize = split_cpu_common_realize;
+    ops->cpu_common_unrealize = split_cpu_common_unrealize;
+    ops->cpu_reset_hold = split_cpu_reset_hold;
+    ops->cpu_thread_routine = split_cpu_thread_routine;
+    ops->kick_vcpu_thread = split_kick_vcpu_thread;
+    ops->cpu_thread_is_idle = split_cpu_thread_is_idle;
+
+    ops->synchronize_post_reset = split_synchronize_post_reset;
+    ops->synchronize_post_init = split_synchronize_post_init;
+    ops->synchronize_state = split_synchronize_state;
+    ops->synchronize_pre_loadvm = split_synchronize_pre_loadvm;
+    ops->synchronize_pre_resume = split_synchronize_pre_resume;
+
+    ops->handle_interrupt = split_handle_interrupt;
+    ops->get_vcpu_stats = split_get_vcpu_stats;
+
+    ops->get_virtual_clock = split_get_virtual_clock;
+    ops->set_virtual_clock = split_set_virtual_clock;
+    ops->get_elapsed_ticks = split_get_elapsed_ticks;
+
+    ops->update_guest_debug = split_update_guest_debug;
+    ops->insert_breakpoint = split_insert_breakpoint;
+    ops->remove_breakpoint = split_remove_breakpoint;
+    ops->remove_all_breakpoints = split_remove_all_breakpoints;
+}
+
+static const TypeInfo split_accel_ops_type = {
+    .name = ACCEL_OPS_NAME("split"),
+    .parent = TYPE_ACCEL_OPS,
+    .class_init = split_accel_ops_class_init,
+    .abstract = true,
+};
+
+static void split_accel_ops_register_types(void)
+{
+    type_register_static(&split_accel_ops_type);
+}
+type_init(split_accel_ops_register_types);
diff --git a/accel/split/split-all.c b/accel/split/split-all.c
new file mode 100644
index 00000000000..7e308f3c1e7
--- /dev/null
+++ b/accel/split/split-all.c
@@ -0,0 +1,77 @@
+/*
+ * QEMU "split" accelerator (HW + SW emulator)
+ *
+ * Copyright (c) 2025 Linaro Ltd.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/accel.h"
+
+bool split_allowed;
+
+static int split_accel_init_machine(MachineState *ms, AccelState *as)
+{
+    g_assert_not_reached();
+}
+
+static void split_setup_post(MachineState *ms, AccelState *accel)
+{
+    g_assert_not_reached();
+}
+
+static bool split_has_memory(MachineState *ms, AddressSpace *as,
+                             hwaddr start_addr, hwaddr size)
+{
+    g_assert_not_reached();
+}
+
+static bool split_cpus_are_resettable(AccelState *as)
+{
+    g_assert_not_reached();
+}
+
+static bool split_supports_guest_debug(AccelState *as)
+{
+    g_assert_not_reached();
+}
+
+static int split_gdbstub_supported_sstep_flags(AccelState *as)
+{
+    g_assert_not_reached();
+}
+
+static void split_get_stats(AccelState *as, GString *buf)
+{
+    g_assert_not_reached();
+}
+
+static void split_accel_class_init(ObjectClass *oc, const void *data)
+{
+    AccelClass *ac = ACCEL_CLASS(oc);
+
+    ac->name = "split";
+    ac->init_machine = split_accel_init_machine;
+    ac->setup_post = split_setup_post;
+    ac->has_memory = split_has_memory;
+    ac->cpus_are_resettable = split_cpus_are_resettable;
+    ac->supports_guest_debug = split_supports_guest_debug;
+    ac->gdbstub_supported_sstep_flags = split_gdbstub_supported_sstep_flags;
+    ac->get_stats = split_get_stats;
+    ac->allowed = &split_allowed;
+    ac->compat_props = NULL;
+}
+
+static const TypeInfo split_accel_type = {
+    .name = ACCEL_CLASS_NAME("split"),
+    .parent = TYPE_ACCEL,
+    .class_init = split_accel_class_init,
+};
+
+static void register_accel_types(void)
+{
+    type_register_static(&split_accel_type);
+}
+
+type_init(register_accel_types);
diff --git a/system/vl.c b/system/vl.c
index 3b7057e6c66..c7111d743ab 100644
--- a/system/vl.c
+++ b/system/vl.c
@@ -2433,6 +2433,7 @@ static void configure_accelerators(const char *progname)
             bool have_tcg = accel_find("tcg");
             bool have_kvm = accel_find("kvm");
             bool have_hvf = accel_find("hvf");
+            bool have_split = accel_find("split");
 
             if (have_tcg && have_kvm) {
                 if (g_str_has_suffix(progname, "kvm")) {
@@ -2447,6 +2448,9 @@ static void configure_accelerators(const char *progname)
                 accelerators = "tcg";
             } else if (have_hvf) {
                 accelerators = "hvf";
+            } else if (have_split) {
+                assert(have_tcg);
+                accelerators = "split";
             } else {
                 error_report("No accelerator selected and"
                              " no default accelerator available");
diff --git a/accel/Kconfig b/accel/Kconfig
index 4263cab7227..122ca9945ad 100644
--- a/accel/Kconfig
+++ b/accel/Kconfig
@@ -10,6 +10,12 @@ config HVF
 config TCG
     bool
 
+config SPLIT
+    bool
+    default y
+    depends on TCG
+    depends on HVF
+
 config KVM
     bool
 
diff --git a/accel/meson.build b/accel/meson.build
index 52909314bfa..b6d4740725f 100644
--- a/accel/meson.build
+++ b/accel/meson.build
@@ -9,6 +9,7 @@ if have_system
   subdir('qtest')
   subdir('kvm')
   subdir('xen')
+  subdir('split')
   subdir('stubs')
 endif
 
diff --git a/accel/split/meson.build b/accel/split/meson.build
new file mode 100644
index 00000000000..fbc68593180
--- /dev/null
+++ b/accel/split/meson.build
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+split_ss = ss.source_set()
+split_ss.add(files(
+  'split-all.c',
+  'split-accel-ops.c',
+))
+
+specific_ss.add_all(when: 'CONFIG_SPLIT', if_true: split_ss)
-- 
2.49.0

[RFC PATCH RESEND 02/42] accel/split: Define SplitAccelState

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel.h | 38 ++++++++++++++++++++++++++++++++++++++
 accel/split/split-all.c   |  5 ++++-
 2 files changed, 42 insertions(+), 1 deletion(-)
 create mode 100644 accel/split/split-accel.h

diff --git a/accel/split/split-accel.h b/accel/split/split-accel.h
new file mode 100644
index 00000000000..cc825a92a6d
--- /dev/null
+++ b/accel/split/split-accel.h
@@ -0,0 +1,38 @@
+/*
+ * QEMU "split" accelerator (HW + SW) state
+ *
+ * Copyright (c) 2025 Linaro Ltd.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#ifndef SPLIT_ACCEL_H
+#define SPLIT_ACCEL_H
+
+#include "qemu/accel.h"
+#include "qom/object.h"
+
+#define TYPE_SPLIT_ACCEL ACCEL_CLASS_NAME("split")
+
+typedef struct SplitAccelClass SplitAccelClass;
+DECLARE_CLASS_CHECKERS(SplitAccelClass, SPLIT_ACCEL, TYPE_SPLIT_ACCEL)
+
+typedef struct SplitAccelState SplitAccelState;
+DECLARE_INSTANCE_CHECKER(SplitAccelState, SPLIT_ACCEL,
+                         TYPE_SPLIT_ACCEL)
+
+struct SplitAccelClass {
+    AccelClass parent_obj;
+
+    AccelClass *hw;
+    AccelClass *sw;
+};
+
+struct SplitAccelState {
+    AccelState parent_obj;
+
+    AccelState *hw;
+    AccelState *sw;
+};
+
+#endif /* SPLIT_ACCEL_H */
diff --git a/accel/split/split-all.c b/accel/split/split-all.c
index 7e308f3c1e7..6bc95c7a7c8 100644
--- a/accel/split/split-all.c
+++ b/accel/split/split-all.c
@@ -8,6 +8,7 @@
 
 #include "qemu/osdep.h"
 #include "qemu/accel.h"
+#include "split-accel.h"
 
 bool split_allowed;
 
@@ -64,8 +65,10 @@ static void split_accel_class_init(ObjectClass *oc, const void *data)
 }
 
 static const TypeInfo split_accel_type = {
-    .name = ACCEL_CLASS_NAME("split"),
+    .name = TYPE_SPLIT_ACCEL,
     .parent = TYPE_ACCEL,
+    .instance_size = sizeof(SplitAccelState),
+    .class_size = sizeof(SplitAccelClass),
     .class_init = split_accel_class_init,
 };
 
-- 
2.49.0

[RFC PATCH RESEND 03/42] accel/split: Implement cpus_are_resettable()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-all.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/accel/split/split-all.c b/accel/split/split-all.c
index 6bc95c7a7c8..c86d0e8583a 100644
--- a/accel/split/split-all.c
+++ b/accel/split/split-all.c
@@ -30,7 +30,18 @@ static bool split_has_memory(MachineState *ms, AddressSpace *as,
 
 static bool split_cpus_are_resettable(AccelState *as)
 {
-    g_assert_not_reached();
+    SplitAccelState *sas = SPLIT_ACCEL(as);
+    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
+    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
+    bool rv = false;
+
+    if (hwc->cpus_are_resettable) {
+        rv |= hwc->cpus_are_resettable(sas->hw);
+    }
+    if (swc->cpus_are_resettable) {
+        rv |= swc->cpus_are_resettable(sas->sw);
+    }
+    return rv;
 }
 
 static bool split_supports_guest_debug(AccelState *as)
-- 
2.49.0

[RFC PATCH RESEND 04/42] accel/split: Implement accel_init_machine()

[Philippe Mathieu-Daudé]

Force to TCG + HVF for now.

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel.h |  3 +++
 accel/split/split-all.c   | 30 +++++++++++++++++++++++++++++-
 2 files changed, 32 insertions(+), 1 deletion(-)

diff --git a/accel/split/split-accel.h b/accel/split/split-accel.h
index cc825a92a6d..19fb530f207 100644
--- a/accel/split/split-accel.h
+++ b/accel/split/split-accel.h
@@ -33,6 +33,9 @@ struct SplitAccelState {
 
     AccelState *hw;
     AccelState *sw;
+
+    char *hw_name;
+    char *sw_name;
 };
 
 #endif /* SPLIT_ACCEL_H */
diff --git a/accel/split/split-all.c b/accel/split/split-all.c
index c86d0e8583a..7cbe32ea768 100644
--- a/accel/split/split-all.c
+++ b/accel/split/split-all.c
@@ -8,13 +8,32 @@
 
 #include "qemu/osdep.h"
 #include "qemu/accel.h"
+#include "hw/boards.h"
+#include "accel/accel-internal.h"
 #include "split-accel.h"
 
 bool split_allowed;
 
 static int split_accel_init_machine(MachineState *ms, AccelState *as)
 {
-    g_assert_not_reached();
+    SplitAccelState *sas = SPLIT_ACCEL(as);
+    AccelClass *hwc;
+    AccelClass *swc;
+
+    swc = accel_find(sas->sw_name);
+    accel_init_ops_interfaces(swc);
+    hwc = accel_find(sas->hw_name);
+    accel_init_ops_interfaces(hwc);
+
+    sas->sw = ACCEL(object_new_with_class(OBJECT_CLASS(swc)));
+    sas->hw = ACCEL(object_new_with_class(OBJECT_CLASS(hwc)));
+
+    accel_init_machine(sas->sw, ms);
+    accel_init_machine(sas->hw, ms);
+
+    ms->accelerator = as;
+
+    return 0;
 }
 
 static void split_setup_post(MachineState *ms, AccelState *accel)
@@ -59,6 +78,14 @@ static void split_get_stats(AccelState *as, GString *buf)
     g_assert_not_reached();
 }
 
+static void split_accel_instance_init(Object *obj)
+{
+    SplitAccelState *sas = SPLIT_ACCEL(obj);
+
+    sas->sw_name = g_strdup("tcg");
+    sas->hw_name = g_strdup("hvf");
+}
+
 static void split_accel_class_init(ObjectClass *oc, const void *data)
 {
     AccelClass *ac = ACCEL_CLASS(oc);
@@ -79,6 +106,7 @@ static const TypeInfo split_accel_type = {
     .name = TYPE_SPLIT_ACCEL,
     .parent = TYPE_ACCEL,
     .instance_size = sizeof(SplitAccelState),
+    .instance_init  = split_accel_instance_init,
     .class_size = sizeof(SplitAccelClass),
     .class_init = split_accel_class_init,
 };
-- 
2.49.0

[RFC PATCH RESEND 05/42] accel/split: Expose 'hw' and 'sw' properties

[Philippe Mathieu-Daudé]

In preparation of other accelerator (or potential emulator),
expose the "hw" and "sw" keys. Only HVF and TCG allowed so far.

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-all.c | 46 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/accel/split/split-all.c b/accel/split/split-all.c
index 7cbe32ea768..28f626d0ff4 100644
--- a/accel/split/split-all.c
+++ b/accel/split/split-all.c
@@ -8,6 +8,7 @@
 
 #include "qemu/osdep.h"
 #include "qemu/accel.h"
+#include "qapi/error.h"
 #include "hw/boards.h"
 #include "accel/accel-internal.h"
 #include "split-accel.h"
@@ -78,6 +79,42 @@ static void split_get_stats(AccelState *as, GString *buf)
     g_assert_not_reached();
 }
 
+static char *split_get_hw(Object *obj, Error **errp)
+{
+    SplitAccelState *s = SPLIT_ACCEL(obj);
+
+    return g_strdup(s->hw_name);
+}
+
+static void split_set_hw(Object *obj, const char *value, Error **errp)
+{
+    SplitAccelState *s = SPLIT_ACCEL(obj);
+
+    if (strcmp(value, "hvf") == 0) {
+        s->hw_name = g_strdup(value);
+    } else {
+        error_setg(errp, "'%s' accelerator no supported", value);
+    }
+}
+
+static char *split_get_sw(Object *obj, Error **errp)
+{
+    SplitAccelState *s = SPLIT_ACCEL(obj);
+
+    return g_strdup(s->sw_name);
+}
+
+static void split_set_sw(Object *obj, const char *value, Error **errp)
+{
+    SplitAccelState *s = SPLIT_ACCEL(obj);
+
+    if (strcmp(value, "tcg") == 0) {
+        s->hw_name = g_strdup(value);
+    } else {
+        error_setg(errp, "'%s' emulator no supported", value);
+    }
+}
+
 static void split_accel_instance_init(Object *obj)
 {
     SplitAccelState *sas = SPLIT_ACCEL(obj);
@@ -100,6 +137,15 @@ static void split_accel_class_init(ObjectClass *oc, const void *data)
     ac->get_stats = split_get_stats;
     ac->allowed = &split_allowed;
     ac->compat_props = NULL;
+
+    object_class_property_add_str(oc, "hw",
+                                  split_get_hw,
+                                  split_set_hw);
+    object_class_property_set_description(oc, "hw", "Hardware accelerator");
+    object_class_property_add_str(oc, "sw",
+                                  split_get_sw,
+                                  split_set_sw);
+    object_class_property_set_description(oc, "sw", "Software emulator");
 }
 
 static const TypeInfo split_accel_type = {
-- 
2.49.0

[RFC PATCH RESEND 06/42] accel/split: Empty setup_post()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-all.c | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/accel/split/split-all.c b/accel/split/split-all.c
index 28f626d0ff4..8b9f8ff77a4 100644
--- a/accel/split/split-all.c
+++ b/accel/split/split-all.c
@@ -37,11 +37,6 @@ static int split_accel_init_machine(MachineState *ms, AccelState *as)
     return 0;
 }
 
-static void split_setup_post(MachineState *ms, AccelState *accel)
-{
-    g_assert_not_reached();
-}
-
 static bool split_has_memory(MachineState *ms, AddressSpace *as,
                              hwaddr start_addr, hwaddr size)
 {
@@ -129,7 +124,6 @@ static void split_accel_class_init(ObjectClass *oc, const void *data)
 
     ac->name = "split";
     ac->init_machine = split_accel_init_machine;
-    ac->setup_post = split_setup_post;
     ac->has_memory = split_has_memory;
     ac->cpus_are_resettable = split_cpus_are_resettable;
     ac->supports_guest_debug = split_supports_guest_debug;
-- 
2.49.0

[RFC PATCH RESEND 07/42] accel/split: Implement supports_guest_debug()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-all.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/accel/split/split-all.c b/accel/split/split-all.c
index 8b9f8ff77a4..cb910300931 100644
--- a/accel/split/split-all.c
+++ b/accel/split/split-all.c
@@ -61,7 +61,18 @@ static bool split_cpus_are_resettable(AccelState *as)
 
 static bool split_supports_guest_debug(AccelState *as)
 {
-    g_assert_not_reached();
+    SplitAccelState *sas = SPLIT_ACCEL(as);
+    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
+    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
+    bool supported = false;
+
+    if (hwc->supports_guest_debug) {
+        supported |= hwc->supports_guest_debug(sas->hw);
+    }
+    if (swc->supports_guest_debug) {
+        supported |= swc->supports_guest_debug(sas->sw);
+    }
+    return supported;
 }
 
 static int split_gdbstub_supported_sstep_flags(AccelState *as)
-- 
2.49.0

[RFC PATCH RESEND 08/42] accel/split: Implement gdbstub_supported_sstep_flags()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-all.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/accel/split/split-all.c b/accel/split/split-all.c
index cb910300931..413954af96c 100644
--- a/accel/split/split-all.c
+++ b/accel/split/split-all.c
@@ -77,7 +77,19 @@ static bool split_supports_guest_debug(AccelState *as)
 
 static int split_gdbstub_supported_sstep_flags(AccelState *as)
 {
-    g_assert_not_reached();
+    SplitAccelState *sas = SPLIT_ACCEL(as);
+    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
+    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
+    int flags = -1;
+
+    if (hwc->gdbstub_supported_sstep_flags) {
+        flags &= hwc->gdbstub_supported_sstep_flags(sas->hw);
+    }
+    if (swc->gdbstub_supported_sstep_flags) {
+        flags &= swc->gdbstub_supported_sstep_flags(sas->sw);
+    }
+
+    return flags;
 }
 
 static void split_get_stats(AccelState *as, GString *buf)
-- 
2.49.0

[RFC PATCH RESEND 09/42] accel/split: Add cpu_thread_routine() stub

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel-ops.c | 70 ++++++++++++++++++++++++++++++++++-
 1 file changed, 69 insertions(+), 1 deletion(-)

diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index e5c1d51d426..294ea79420e 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -7,11 +7,79 @@
  */
 
 #include "qemu/osdep.h"
+#include "qemu/main-loop.h"
+#include "qemu/guest-random.h"
+#include "exec/cpu-common.h"
 #include "system/accel-ops.h"
+#include "system/cpus.h"
 
 static void *split_cpu_thread_routine(void *arg)
 {
-    g_assert_not_reached();
+    CPUState *cpu = arg;
+    int r;
+
+    /* TODO: check accel allowed */
+
+    rcu_register_thread();
+
+    bql_lock();
+    qemu_thread_get_self(cpu->thread);
+
+    cpu->thread_id = qemu_get_thread_id();
+    current_cpu = cpu;
+
+    /* TODO: init_vcpu_thread() */
+    /* TODO: allocate AccelCPUState */
+
+    /* signal CPU creation */
+    cpu_thread_signal_created(cpu);
+    qemu_guest_random_seed_thread_part2(cpu->random_seed);
+
+    /* process any pending work */
+    cpu->exit_request = 1;
+
+    do {
+        r = 0;
+
+        if (cpu_can_run(cpu)) {
+            r = 0; /* TODO: exec_vcpu_thread() */
+            switch (r) {
+            case 0:
+                break;
+            case EXCP_INTERRUPT:
+                break;
+            case EXCP_YIELD:
+                break;
+            case EXCP_DEBUG:
+                cpu_handle_guest_debug(cpu);
+                break;
+            case EXCP_HALTED:
+                /*
+                 * Usually cpu->halted is set, but may have already been
+                 * reset by another thread by the time we arrive here.
+                 */
+                break;
+            case EXCP_ATOMIC:
+                bql_unlock();
+                cpu_exec_step_atomic(cpu);
+                bql_lock();
+                break;
+            default:
+                /* Ignore everything else? */
+                break;
+            }
+        }
+        qemu_wait_io_event(cpu);
+    } while (!cpu->unplug || cpu_can_run(cpu));
+
+    /* TODO: destroy_vcpu_thread() */
+
+    cpu_thread_signal_destroyed(cpu);
+    bql_unlock();
+
+    rcu_unregister_thread();
+
+    return NULL;
 }
 
 static void split_ops_init(AccelClass *ac)
-- 
2.49.0

[RFC PATCH RESEND 10/42] accel/split: Define and allocate AccelCPUState

[Philippe Mathieu-Daudé]

AccelCPUState is where we store per-vCPU accelerator
related information.

FIXME: structures might clash so add padding (ok with TCG+HVF)

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel.h     | 7 +++++++
 accel/split/split-accel-ops.c | 9 ++++++++-
 2 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/accel/split/split-accel.h b/accel/split/split-accel.h
index 19fb530f207..87a08454ce6 100644
--- a/accel/split/split-accel.h
+++ b/accel/split/split-accel.h
@@ -38,4 +38,11 @@ struct SplitAccelState {
     char *sw_name;
 };
 
+struct AccelCPUState {
+    char pad[128];
+
+    AccelState *accel;
+    bool use_hw;
+};
+
 #endif /* SPLIT_ACCEL_H */
diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index 294ea79420e..efef0d60fde 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -12,10 +12,13 @@
 #include "exec/cpu-common.h"
 #include "system/accel-ops.h"
 #include "system/cpus.h"
+#include "split-accel.h"
 
 static void *split_cpu_thread_routine(void *arg)
 {
+    AccelState *as = current_accel();
     CPUState *cpu = arg;
+    AccelCPUState *acs;
     int r;
 
     /* TODO: check accel allowed */
@@ -29,7 +32,10 @@ static void *split_cpu_thread_routine(void *arg)
     current_cpu = cpu;
 
     /* TODO: init_vcpu_thread() */
-    /* TODO: allocate AccelCPUState */
+    cpu->accel = g_renew(AccelCPUState, cpu->accel, 1); /* XXX only with current TCG */
+    acs = cpu->accel;
+    acs->accel = as;
+    acs->use_hw = false;
 
     /* signal CPU creation */
     cpu_thread_signal_created(cpu);
@@ -60,6 +66,7 @@ static void *split_cpu_thread_routine(void *arg)
                  */
                 break;
             case EXCP_ATOMIC:
+                assert(!acs->use_hw);
                 bql_unlock();
                 cpu_exec_step_atomic(cpu);
                 bql_lock();
-- 
2.49.0

[RFC PATCH RESEND 11/42] accel/split: Register MTTCG

[Philippe Mathieu-Daudé]

FIXME: Use sw-hooks?

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel-ops.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index efef0d60fde..2c7945b6331 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -14,9 +14,12 @@
 #include "system/cpus.h"
 #include "split-accel.h"
 
+#include "accel/tcg/tcg-accel-ops-mttcg.h"
+
 static void *split_cpu_thread_routine(void *arg)
 {
     AccelState *as = current_accel();
+    void *sw_force_rcu;
     CPUState *cpu = arg;
     AccelCPUState *acs;
     int r;
@@ -24,6 +27,7 @@ static void *split_cpu_thread_routine(void *arg)
     /* TODO: check accel allowed */
 
     rcu_register_thread();
+    sw_force_rcu = mttcg_vcpu_register(cpu);
 
     bql_lock();
     qemu_thread_get_self(cpu->thread);
@@ -84,6 +88,7 @@ static void *split_cpu_thread_routine(void *arg)
     cpu_thread_signal_destroyed(cpu);
     bql_unlock();
 
+    mttcg_vcpu_unregister(cpu, sw_force_rcu);
     rcu_unregister_thread();
 
     return NULL;
-- 
2.49.0

[RFC PATCH RESEND 12/42] accel/split: Have thread routine ready to dispatch over HW/SW

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel-ops.c | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)

diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index 2c7945b6331..39495fdff14 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -19,12 +19,18 @@
 static void *split_cpu_thread_routine(void *arg)
 {
     AccelState *as = current_accel();
+    SplitAccelState *sas = SPLIT_ACCEL(as);
+    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
+    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
+    AccelOpsClass *hwops = hwc->ops;
+    AccelOpsClass *swops = swc->ops;
     void *sw_force_rcu;
     CPUState *cpu = arg;
     AccelCPUState *acs;
     int r;
 
-    /* TODO: check accel allowed */
+    assert(swc->allowed);
+    assert(hwc->allowed);
 
     rcu_register_thread();
     sw_force_rcu = mttcg_vcpu_register(cpu);
@@ -35,7 +41,8 @@ static void *split_cpu_thread_routine(void *arg)
     cpu->thread_id = qemu_get_thread_id();
     current_cpu = cpu;
 
-    /* TODO: init_vcpu_thread() */
+    hwops->init_vcpu_thread(cpu);
+    swops->init_vcpu_thread(cpu);
     cpu->accel = g_renew(AccelCPUState, cpu->accel, 1); /* XXX only with current TCG */
     acs = cpu->accel;
     acs->accel = as;
@@ -49,10 +56,12 @@ static void *split_cpu_thread_routine(void *arg)
     cpu->exit_request = 1;
 
     do {
-        r = 0;
-
         if (cpu_can_run(cpu)) {
-            r = 0; /* TODO: exec_vcpu_thread() */
+            if (acs->use_hw) {
+                r = hwops->exec_vcpu_thread(cpu);
+            } else {
+                r = swops->exec_vcpu_thread(cpu);
+            }
             switch (r) {
             case 0:
                 break;
@@ -83,7 +92,8 @@ static void *split_cpu_thread_routine(void *arg)
         qemu_wait_io_event(cpu);
     } while (!cpu->unplug || cpu_can_run(cpu));
 
-    /* TODO: destroy_vcpu_thread() */
+    hwops->destroy_vcpu_thread(cpu);
+    swops->destroy_vcpu_thread(cpu);
 
     cpu_thread_signal_destroyed(cpu);
     bql_unlock();
-- 
2.49.0

[RFC PATCH RESEND 13/42] accel/split: Implement cpu_reset_hold()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel-ops.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index 39495fdff14..d4fa07c2dec 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -10,6 +10,7 @@
 #include "qemu/main-loop.h"
 #include "qemu/guest-random.h"
 #include "exec/cpu-common.h"
+#include "hw/core/cpu.h"
 #include "system/accel-ops.h"
 #include "system/cpus.h"
 #include "split-accel.h"
@@ -121,7 +122,16 @@ static void split_cpu_common_unrealize(CPUState *cpu)
 
 static void split_cpu_reset_hold(CPUState *cpu)
 {
-    g_assert_not_reached();
+    SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
+    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
+    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
+
+    if (hwc->ops->cpu_reset_hold) {
+        hwc->ops->cpu_reset_hold(cpu);
+    }
+    if (swc->ops->cpu_reset_hold) {
+        swc->ops->cpu_reset_hold(cpu);
+    }
 }
 
 static void split_kick_vcpu_thread(CPUState *cpu)
-- 
2.49.0

[RFC PATCH RESEND 14/42] accel/split: Implement synchronize_post_init()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel-ops.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index d4fa07c2dec..6b3ddf21a17 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -151,7 +151,16 @@ static void split_synchronize_post_reset(CPUState *cpu)
 
 static void split_synchronize_post_init(CPUState *cpu)
 {
-    g_assert_not_reached();
+    SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
+    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
+    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
+
+    if (hwc->ops->synchronize_post_init) {
+        hwc->ops->synchronize_post_init(cpu);
+    }
+    if (swc->ops->synchronize_post_init) {
+        swc->ops->synchronize_post_init(cpu);
+    }
 }
 
 static void split_synchronize_state(CPUState *cpu)
-- 
2.49.0

[RFC PATCH RESEND 15/42] accel/split: Implement synchronize_pre_resume()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel-ops.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index 6b3ddf21a17..65feb929404 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -175,7 +175,17 @@ static void split_synchronize_pre_loadvm(CPUState *cpu)
 
 static void split_synchronize_pre_resume(bool step_pending)
 {
-    g_assert_not_reached();
+    AccelState *as = current_accel();
+    SplitAccelState *sas = SPLIT_ACCEL(as);
+    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
+    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
+
+    if (hwc->ops->synchronize_pre_resume) {
+        hwc->ops->synchronize_pre_resume(step_pending);
+    }
+    if (swc->ops->synchronize_pre_resume) {
+        swc->ops->synchronize_pre_resume(step_pending);
+    }
 }
 
 static void split_handle_interrupt(CPUState *cpu, int mask)
-- 
2.49.0

[RFC PATCH RESEND 16/42] accel/split: Implement synchronize_state()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel-ops.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index 65feb929404..80efbd82091 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -165,7 +165,16 @@ static void split_synchronize_post_init(CPUState *cpu)
 
 static void split_synchronize_state(CPUState *cpu)
 {
-    g_assert_not_reached();
+    SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
+    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
+    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
+
+    if (hwc->ops->synchronize_state) {
+        hwc->ops->synchronize_state(cpu);
+    }
+    if (swc->ops->synchronize_state) {
+        swc->ops->synchronize_state(cpu);
+    }
 }
 
 static void split_synchronize_pre_loadvm(CPUState *cpu)
-- 
2.49.0

[RFC PATCH RESEND 17/42] accel/split: Implement synchronize_post_reset()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel-ops.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index 80efbd82091..91ed0af2746 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -146,7 +146,16 @@ static bool split_cpu_thread_is_idle(CPUState *cpu)
 
 static void split_synchronize_post_reset(CPUState *cpu)
 {
-    g_assert_not_reached();
+    SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
+    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
+    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
+
+    if (hwc->ops->synchronize_post_reset) {
+        hwc->ops->synchronize_post_reset(cpu);
+    }
+    if (swc->ops->synchronize_post_reset) {
+        swc->ops->synchronize_post_reset(cpu);
+    }
 }
 
 static void split_synchronize_post_init(CPUState *cpu)
-- 
2.49.0

[RFC PATCH RESEND 18/42] accel/split: Implement synchronize_pre_loadvm()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel-ops.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index 91ed0af2746..746a6a2782d 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -188,7 +188,16 @@ static void split_synchronize_state(CPUState *cpu)
 
 static void split_synchronize_pre_loadvm(CPUState *cpu)
 {
-    g_assert_not_reached();
+    SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
+    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
+    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
+
+    if (hwc->ops->synchronize_pre_loadvm) {
+        hwc->ops->synchronize_pre_loadvm(cpu);
+    }
+    if (swc->ops->synchronize_pre_loadvm) {
+        swc->ops->synchronize_pre_loadvm(cpu);
+    }
 }
 
 static void split_synchronize_pre_resume(bool step_pending)
-- 
2.49.0

[RFC PATCH RESEND 19/42] accel/split: Implement kick_vcpu_thread()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel-ops.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index 746a6a2782d..63b6217f3fa 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -136,7 +136,13 @@ static void split_cpu_reset_hold(CPUState *cpu)
 
 static void split_kick_vcpu_thread(CPUState *cpu)
 {
-    g_assert_not_reached();
+    SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
+    AccelClass *ac = cpu->accel->use_hw ? ACCEL_GET_CLASS(sas->hw)
+                                        : ACCEL_GET_CLASS(sas->sw);
+
+    if (ac->ops->kick_vcpu_thread) {
+        ac->ops->kick_vcpu_thread(cpu);
+    }
 }
 
 static bool split_cpu_thread_is_idle(CPUState *cpu)
-- 
2.49.0

[RFC PATCH RESEND 20/42] accel/split: Implement cpu_common_realize()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel-ops.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index 63b6217f3fa..3278e01f18a 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -15,6 +15,7 @@
 #include "system/cpus.h"
 #include "split-accel.h"
 
+#include "accel/accel-internal.h"
 #include "accel/tcg/tcg-accel-ops-mttcg.h"
 
 static void *split_cpu_thread_routine(void *arg)
@@ -112,7 +113,14 @@ static void split_ops_init(AccelClass *ac)
 
 static bool split_cpu_common_realize(CPUState *cpu, Error **errp)
 {
-    g_assert_not_reached();
+    AccelState *as = current_accel();
+    SplitAccelState *sas = SPLIT_ACCEL(as);
+    bool success = false;
+
+    success |= accel_cpu_realize(sas->sw, cpu, errp);
+    success |= accel_cpu_realize(sas->hw, cpu, errp);
+
+    return success;
 }
 
 static void split_cpu_common_unrealize(CPUState *cpu)
-- 
2.49.0

[RFC PATCH RESEND 21/42] accel/split: Set use_hw in cpu_thread_routine() and switch over

[Philippe Mathieu-Daudé]

Introduce the EXCP_HWACCEL definition to switch to
hardware accelerator.

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel.h     |  1 +
 include/exec/cpu-common.h     |  1 +
 accel/split/split-accel-ops.c | 11 +++++++++++
 3 files changed, 13 insertions(+)

diff --git a/accel/split/split-accel.h b/accel/split/split-accel.h
index 87a08454ce6..8a2888507f3 100644
--- a/accel/split/split-accel.h
+++ b/accel/split/split-accel.h
@@ -42,6 +42,7 @@ struct AccelCPUState {
     char pad[128];
 
     AccelState *accel;
+
     bool use_hw;
 };
 
diff --git a/include/exec/cpu-common.h b/include/exec/cpu-common.h
index a68485547d5..6216be5f5e3 100644
--- a/include/exec/cpu-common.h
+++ b/include/exec/cpu-common.h
@@ -20,6 +20,7 @@
 #define EXCP_HALTED     0x10003 /* cpu is halted (waiting for external event) */
 #define EXCP_YIELD      0x10004 /* cpu wants to yield timeslice to another */
 #define EXCP_ATOMIC     0x10005 /* stop-the-world and emulate atomic */
+#define EXCP_HWACCEL    0x10006 /* use hardware accelerator */
 
 void cpu_exec_init_all(void);
 void cpu_exec_step_atomic(CPUState *cpu);
diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index 3278e01f18a..1766834d537 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -10,9 +10,12 @@
 #include "qemu/main-loop.h"
 #include "qemu/guest-random.h"
 #include "exec/cpu-common.h"
+#include "exec/cpu-interrupt.h"
 #include "hw/core/cpu.h"
 #include "system/accel-ops.h"
 #include "system/cpus.h"
+#include "system/hw_accel.h"
+#include "system/tcg.h"
 #include "split-accel.h"
 
 #include "accel/accel-internal.h"
@@ -66,10 +69,14 @@ static void *split_cpu_thread_routine(void *arg)
             }
             switch (r) {
             case 0:
+                if (acs->use_hw) {
+                    acs->use_hw = cpu_acceleratable(cpu);
+                }
                 break;
             case EXCP_INTERRUPT:
                 break;
             case EXCP_YIELD:
+                assert(!acs->use_hw);
                 break;
             case EXCP_DEBUG:
                 cpu_handle_guest_debug(cpu);
@@ -86,6 +93,10 @@ static void *split_cpu_thread_routine(void *arg)
                 cpu_exec_step_atomic(cpu);
                 bql_lock();
                 break;
+            case EXCP_HWACCEL:
+                assert(!acs->use_hw);
+                acs->use_hw = true;
+                break;
             default:
                 /* Ignore everything else? */
                 break;
-- 
2.49.0

[RFC PATCH RESEND 22/42] accel/split: Add few trace events in cpu_thread_routine handler

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 meson.build                   | 1 +
 accel/split/trace.h           | 2 ++
 accel/split/split-accel-ops.c | 7 +++++++
 accel/split/trace-events      | 9 +++++++++
 4 files changed, 19 insertions(+)
 create mode 100644 accel/split/trace.h
 create mode 100644 accel/split/trace-events

diff --git a/meson.build b/meson.build
index 5004678a26b..06c32e0e958 100644
--- a/meson.build
+++ b/meson.build
@@ -3635,6 +3635,7 @@ if have_system
   trace_events_subdirs += [
     'accel/hvf',
     'accel/kvm',
+    'accel/split',
     'audio',
     'backends',
     'backends/tpm',
diff --git a/accel/split/trace.h b/accel/split/trace.h
new file mode 100644
index 00000000000..46b45388930
--- /dev/null
+++ b/accel/split/trace.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+#include "trace/trace-accel_split.h"
diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index 1766834d537..399bf71a9ec 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -17,6 +17,7 @@
 #include "system/hw_accel.h"
 #include "system/tcg.h"
 #include "split-accel.h"
+#include "trace.h"
 
 #include "accel/accel-internal.h"
 #include "accel/tcg/tcg-accel-ops-mttcg.h"
@@ -64,13 +65,18 @@ static void *split_cpu_thread_routine(void *arg)
         if (cpu_can_run(cpu)) {
             if (acs->use_hw) {
                 r = hwops->exec_vcpu_thread(cpu);
+                trace_accel_split_exec_vcpu_thread_hw(r);
             } else {
                 r = swops->exec_vcpu_thread(cpu);
+                trace_accel_split_exec_vcpu_thread_sw(r);
             }
             switch (r) {
             case 0:
                 if (acs->use_hw) {
                     acs->use_hw = cpu_acceleratable(cpu);
+                    if (!acs->use_hw) {
+                        trace_accel_split_sw_emulate();
+                    }
                 }
                 break;
             case EXCP_INTERRUPT:
@@ -95,6 +101,7 @@ static void *split_cpu_thread_routine(void *arg)
                 break;
             case EXCP_HWACCEL:
                 assert(!acs->use_hw);
+                trace_accel_split_hw_accelerate();
                 acs->use_hw = true;
                 break;
             default:
diff --git a/accel/split/trace-events b/accel/split/trace-events
new file mode 100644
index 00000000000..c0b9fd97d1d
--- /dev/null
+++ b/accel/split/trace-events
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# See docs/devel/tracing.rst for syntax documentation.
+
+# split-accel-ops.c
+accel_split_exec_vcpu_thread_sw(int retval) "sw: rv:0x%x"
+accel_split_exec_vcpu_thread_hw(int retval) "hw: rv:0x%x"
+accel_split_hw_accelerate(void) ""
+accel_split_sw_emulate(void) ""
-- 
2.49.0

[RFC PATCH RESEND 23/42] target/arm: Implement SysemuCPUOps::can_accelerate() handler

[Philippe Mathieu-Daudé]

ARM hardware can only accelerate EL0 and EL1.

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 target/arm/cpu.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/target/arm/cpu.c b/target/arm/cpu.c
index ab5fbd9b40b..1a19e5cfb45 100644
--- a/target/arm/cpu.c
+++ b/target/arm/cpu.c
@@ -149,6 +149,13 @@ static bool arm_cpu_has_work(CPUState *cs)
          | CPU_INTERRUPT_VFIQ | CPU_INTERRUPT_VIRQ | CPU_INTERRUPT_VSERR
          | CPU_INTERRUPT_EXITTB);
 }
+
+static bool arm_cpu_can_accelerate(CPUState *cs)
+{
+    CPUARMState *env = cpu_env(cs);
+
+    return arm_current_el(env) < 2;
+}
 #endif /* !CONFIG_USER_ONLY */
 
 void arm_register_pre_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook,
@@ -2695,6 +2702,7 @@ static vaddr aarch64_untagged_addr(CPUState *cs, vaddr x)
 
 static const struct SysemuCPUOps arm_sysemu_ops = {
     .has_work = arm_cpu_has_work,
+    .can_accelerate = arm_cpu_can_accelerate,
     .get_phys_page_attrs_debug = arm_cpu_get_phys_page_attrs_debug,
     .asidx_from_attrs = arm_asidx_from_attrs,
     .write_elf32_note = arm_cpu_write_elf32_note,
-- 
2.49.0

[RFC PATCH RESEND 24/42] accel/split: Implement handle_interrupt()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel-ops.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index 399bf71a9ec..5a36e22f205 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -249,7 +249,15 @@ static void split_synchronize_pre_resume(bool step_pending)
 
 static void split_handle_interrupt(CPUState *cpu, int mask)
 {
-    g_assert_not_reached();
+    SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
+    AccelClass *ac = cpu->accel->use_hw ? ACCEL_GET_CLASS(sas->hw)
+                                        : ACCEL_GET_CLASS(sas->sw);
+
+    if (ac->ops->handle_interrupt) {
+        ac->ops->handle_interrupt(cpu, mask);
+    } else {
+        generic_handle_interrupt(cpu, mask);
+    }
 }
 
 static int64_t split_get_virtual_clock(void)
-- 
2.49.0

[RFC PATCH RESEND 25/42] accel/split: Empty ops_init()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel-ops.c | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index 5a36e22f205..375821f1064 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -124,11 +124,6 @@ static void *split_cpu_thread_routine(void *arg)
     return NULL;
 }
 
-static void split_ops_init(AccelClass *ac)
-{
-    g_assert_not_reached();
-}
-
 static bool split_cpu_common_realize(CPUState *cpu, Error **errp)
 {
     AccelState *as = current_accel();
@@ -306,7 +301,6 @@ static void split_accel_ops_class_init(ObjectClass *oc, const void *data)
 {
     AccelOpsClass *ops = ACCEL_OPS_CLASS(oc);
 
-    ops->ops_init = split_ops_init;
     ops->cpu_common_realize = split_cpu_common_realize;
     ops->cpu_common_unrealize = split_cpu_common_unrealize;
     ops->cpu_reset_hold = split_cpu_reset_hold;
-- 
2.49.0

[RFC PATCH RESEND 26/42] accel/split: Empty set/get_virtual_clock()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel-ops.c | 12 ------------
 1 file changed, 12 deletions(-)

diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index 375821f1064..d19b4641a2e 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -255,16 +255,6 @@ static void split_handle_interrupt(CPUState *cpu, int mask)
     }
 }
 
-static int64_t split_get_virtual_clock(void)
-{
-    g_assert_not_reached();
-}
-
-static void split_set_virtual_clock(int64_t time)
-{
-    g_assert_not_reached();
-}
-
 static int64_t split_get_elapsed_ticks(void)
 {
     g_assert_not_reached();
@@ -317,8 +307,6 @@ static void split_accel_ops_class_init(ObjectClass *oc, const void *data)
     ops->handle_interrupt = split_handle_interrupt;
     ops->get_vcpu_stats = split_get_vcpu_stats;
 
-    ops->get_virtual_clock = split_get_virtual_clock;
-    ops->set_virtual_clock = split_set_virtual_clock;
     ops->get_elapsed_ticks = split_get_elapsed_ticks;
 
     ops->update_guest_debug = split_update_guest_debug;
-- 
2.49.0

[RFC PATCH RESEND 27/42] accel/split: Empty get_elapsed_ticks()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel-ops.c | 7 -------
 1 file changed, 7 deletions(-)

diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index d19b4641a2e..034b5ebc96c 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -255,11 +255,6 @@ static void split_handle_interrupt(CPUState *cpu, int mask)
     }
 }
 
-static int64_t split_get_elapsed_ticks(void)
-{
-    g_assert_not_reached();
-}
-
 static int split_update_guest_debug(CPUState *cpu)
 {
     g_assert_not_reached();
@@ -307,8 +302,6 @@ static void split_accel_ops_class_init(ObjectClass *oc, const void *data)
     ops->handle_interrupt = split_handle_interrupt;
     ops->get_vcpu_stats = split_get_vcpu_stats;
 
-    ops->get_elapsed_ticks = split_get_elapsed_ticks;
-
     ops->update_guest_debug = split_update_guest_debug;
     ops->insert_breakpoint = split_insert_breakpoint;
     ops->remove_breakpoint = split_remove_breakpoint;
-- 
2.49.0

[RFC PATCH RESEND 28/42] accel/split: Empty cpu_thread_is_idle()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel-ops.c | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index 034b5ebc96c..e1d91ace2fa 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -166,11 +166,6 @@ static void split_kick_vcpu_thread(CPUState *cpu)
     }
 }
 
-static bool split_cpu_thread_is_idle(CPUState *cpu)
-{
-    g_assert_not_reached();
-}
-
 static void split_synchronize_post_reset(CPUState *cpu)
 {
     SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
@@ -291,7 +286,6 @@ static void split_accel_ops_class_init(ObjectClass *oc, const void *data)
     ops->cpu_reset_hold = split_cpu_reset_hold;
     ops->cpu_thread_routine = split_cpu_thread_routine;
     ops->kick_vcpu_thread = split_kick_vcpu_thread;
-    ops->cpu_thread_is_idle = split_cpu_thread_is_idle;
 
     ops->synchronize_post_reset = split_synchronize_post_reset;
     ops->synchronize_post_init = split_synchronize_post_init;
-- 
2.49.0

[RFC PATCH RESEND 29/42] accel/split: Kludge qemu_tcg_mttcg_enabled()

[Philippe Mathieu-Daudé]

../../accel/tcg/tcg-all.c:59:TCG_STATE: Object 0x600000c42740 is not an instance of type tcg-accel

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/tcg/tcg-all.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/accel/tcg/tcg-all.c b/accel/tcg/tcg-all.c
index f5920b5796e..96104915eef 100644
--- a/accel/tcg/tcg-all.c
+++ b/accel/tcg/tcg-all.c
@@ -61,6 +61,7 @@ DECLARE_INSTANCE_CHECKER(TCGState, TCG_STATE,
 #ifndef CONFIG_USER_ONLY
 bool qemu_tcg_mttcg_enabled(void)
 {
+    return true;
     TCGState *s = TCG_STATE(current_accel());
     return s->mttcg_enabled == ON_OFF_AUTO_ON;
 }
-- 
2.49.0

[RFC PATCH RESEND 30/42] accel/split: Implement remove_all_breakpoints()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel-ops.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index e1d91ace2fa..0f3d48fc68d 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -269,7 +269,16 @@ static int split_remove_breakpoint(CPUState *cpu, int type,
 
 static void split_remove_all_breakpoints(CPUState *cpu)
 {
-    g_assert_not_reached();
+    SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
+    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
+    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
+
+    if (hwc->ops->remove_all_breakpoints) {
+        hwc->ops->remove_all_breakpoints(cpu);
+    }
+    if (swc->ops->remove_all_breakpoints) {
+        swc->ops->remove_all_breakpoints(cpu);
+    }
 }
 
 static void split_get_vcpu_stats(CPUState *cpu, GString *buf)
-- 
2.49.0

[RFC PATCH RESEND 31/42] accel/split: Implement remove_breakpoint()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel-ops.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index 0f3d48fc68d..40cd39aea5c 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -264,7 +264,18 @@ static int split_insert_breakpoint(CPUState *cpu, int type,
 static int split_remove_breakpoint(CPUState *cpu, int type,
                                    vaddr addr, vaddr len)
 {
-    g_assert_not_reached();
+    SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
+    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
+    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
+    int err = 0;
+
+    if (hwc->ops->remove_breakpoint) {
+        err |= hwc->ops->remove_breakpoint(cpu, type, addr, len);
+    }
+    if (swc->ops->remove_breakpoint) {
+        err |= swc->ops->remove_breakpoint(cpu, type, addr, len);
+    }
+    return err;
 }
 
 static void split_remove_all_breakpoints(CPUState *cpu)
-- 
2.49.0

[RFC PATCH RESEND 32/42] accel/split: Implement insert_breakpoint()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel-ops.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index 40cd39aea5c..eb52d690c7c 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -258,7 +258,18 @@ static int split_update_guest_debug(CPUState *cpu)
 static int split_insert_breakpoint(CPUState *cpu, int type,
                                    vaddr addr, vaddr len)
 {
-    g_assert_not_reached();
+    SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
+    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
+    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
+    int err = 0;
+
+    if (hwc->ops->insert_breakpoint) {
+        err |= hwc->ops->insert_breakpoint(cpu, type, addr, len);
+    }
+    if (swc->ops->insert_breakpoint) {
+        err |= swc->ops->insert_breakpoint(cpu, type, addr, len);
+    }
+    return err;
 }
 
 static int split_remove_breakpoint(CPUState *cpu, int type,
-- 
2.49.0

[RFC PATCH RESEND 33/42] accel/split: Implement update_guest_debug()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel-ops.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index eb52d690c7c..d59e70e0d9b 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -252,7 +252,16 @@ static void split_handle_interrupt(CPUState *cpu, int mask)
 
 static int split_update_guest_debug(CPUState *cpu)
 {
-    g_assert_not_reached();
+    SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
+    AccelClass *ac = cpu->accel->use_hw ? ACCEL_GET_CLASS(sas->hw)
+                                        : ACCEL_GET_CLASS(sas->sw);
+    int ret = 0;
+
+    if (ac->ops->update_guest_debug) {
+        ret = ac->ops->update_guest_debug(cpu);
+    }
+
+    return ret;
 }
 
 static int split_insert_breakpoint(CPUState *cpu, int type,
-- 
2.49.0

[RFC PATCH RESEND 34/42] accel/split: Implement get_[vcpu]_stats()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel.h     |  5 +++++
 accel/split/split-accel-ops.c | 24 +++++++++++++++++++++++-
 accel/split/split-all.c       | 16 +++++++++++++++-
 3 files changed, 43 insertions(+), 2 deletions(-)

diff --git a/accel/split/split-accel.h b/accel/split/split-accel.h
index 8a2888507f3..80a19bbb1e5 100644
--- a/accel/split/split-accel.h
+++ b/accel/split/split-accel.h
@@ -36,6 +36,11 @@ struct SplitAccelState {
 
     char *hw_name;
     char *sw_name;
+
+    struct {
+        unsigned transitions; /* number of HW <-> SW transitions */
+        uint64_t exec_count[2]; /* SW:0 HW:1 */
+    } stats;
 };
 
 struct AccelCPUState {
diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index d59e70e0d9b..615faf1d96b 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -9,6 +9,7 @@
 #include "qemu/osdep.h"
 #include "qemu/main-loop.h"
 #include "qemu/guest-random.h"
+#include "qapi/type-helpers.h"
 #include "exec/cpu-common.h"
 #include "exec/cpu-interrupt.h"
 #include "hw/core/cpu.h"
@@ -16,6 +17,7 @@
 #include "system/cpus.h"
 #include "system/hw_accel.h"
 #include "system/tcg.h"
+#include "monitor/monitor.h"
 #include "split-accel.h"
 #include "trace.h"
 
@@ -70,6 +72,7 @@ static void *split_cpu_thread_routine(void *arg)
                 r = swops->exec_vcpu_thread(cpu);
                 trace_accel_split_exec_vcpu_thread_sw(r);
             }
+            sas->stats.exec_count[acs->use_hw]++;
             switch (r) {
             case 0:
                 if (acs->use_hw) {
@@ -103,6 +106,7 @@ static void *split_cpu_thread_routine(void *arg)
                 assert(!acs->use_hw);
                 trace_accel_split_hw_accelerate();
                 acs->use_hw = true;
+                sas->stats.transitions++;
                 break;
             default:
                 /* Ignore everything else? */
@@ -314,7 +318,25 @@ static void split_remove_all_breakpoints(CPUState *cpu)
 
 static void split_get_vcpu_stats(CPUState *cpu, GString *buf)
 {
-    g_assert_not_reached();
+    SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
+    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
+    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
+    g_autofree char *cpu_model =
+        cpu_model_from_type(object_get_typename(OBJECT(cpu)));
+
+    if (cpu->cluster_index != -1) {
+        g_string_append_printf(buf, "CPU#%d@%d: %s\n",
+                               cpu->cpu_index, cpu->cluster_index, cpu_model);
+    } else {
+        g_string_append_printf(buf, "CPU#%d: %s\n",
+                               cpu->cpu_index, cpu_model);
+    }
+    if (hwc->ops->get_vcpu_stats) {
+        hwc->ops->get_vcpu_stats(cpu, buf);
+    }
+    if (swc->ops->get_vcpu_stats) {
+        swc->ops->get_vcpu_stats(cpu, buf);
+    }
 }
 
 static void split_accel_ops_class_init(ObjectClass *oc, const void *data)
diff --git a/accel/split/split-all.c b/accel/split/split-all.c
index 413954af96c..9e0848db971 100644
--- a/accel/split/split-all.c
+++ b/accel/split/split-all.c
@@ -94,7 +94,21 @@ static int split_gdbstub_supported_sstep_flags(AccelState *as)
 
 static void split_get_stats(AccelState *as, GString *buf)
 {
-    g_assert_not_reached();
+    SplitAccelState *sas = SPLIT_ACCEL(as);
+    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
+    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
+
+    g_string_append_printf(buf,
+                           "%u transitions, %llu SW and %llu HW executions\n",
+                           sas->stats.transitions,
+                           sas->stats.exec_count[0],
+                           sas->stats.exec_count[1]);
+    if (hwc->get_stats) {
+        hwc->get_stats(as, buf);
+    }
+    if (swc->get_stats) {
+        swc->get_stats(as, buf);
+    }
 }
 
 static char *split_get_hw(Object *obj, Error **errp)
-- 
2.49.0

[RFC PATCH RESEND 35/42] target/arm: Emulate EL2 under TCG

[Philippe Mathieu-Daudé]

When using hardware acceleration and TCG is available,
expose EL2 and EL3 features as available (they will be
emulated).

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 target/arm/cpu64.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
index 200da1c489b..b5c2796f3a1 100644
--- a/target/arm/cpu64.c
+++ b/target/arm/cpu64.c
@@ -738,20 +738,23 @@ static void aarch64_a53_initfn(Object *obj)
 
 static void aarch64_host_initfn(Object *obj)
 {
-#if defined(CONFIG_KVM)
     ARMCPU *cpu = ARM_CPU(obj);
+#if defined(CONFIG_KVM)
     kvm_arm_set_cpu_features_from_host(cpu);
     if (arm_feature(&cpu->env, ARM_FEATURE_AARCH64)) {
         aarch64_add_sve_properties(obj);
         aarch64_add_pauth_properties(obj);
     }
 #elif defined(CONFIG_HVF)
-    ARMCPU *cpu = ARM_CPU(obj);
     hvf_arm_set_cpu_features_from_host(cpu);
     aarch64_add_pauth_properties(obj);
 #else
     g_assert_not_reached();
 #endif
+    if (tcg_enabled()) {
+        set_feature(&cpu->env, ARM_FEATURE_EL2);
+        set_feature(&cpu->env, ARM_FEATURE_EL3);
+    }
 }
 
 static void aarch64_max_initfn(Object *obj)
-- 
2.49.0

[RFC PATCH RESEND 36/42] target/arm: Have ERET switch to hw accel for EL0/EL1

[Philippe Mathieu-Daudé]

When hardware accelerator available, use it to
directly run EL0 and EL1.

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 target/arm/tcg/helper-a64.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/target/arm/tcg/helper-a64.c b/target/arm/tcg/helper-a64.c
index 4f618ae390e..5962cff4e7a 100644
--- a/target/arm/tcg/helper-a64.c
+++ b/target/arm/tcg/helper-a64.c
@@ -29,6 +29,7 @@
 #include "internals.h"
 #include "qemu/crc32c.h"
 #include "exec/cpu-common.h"
+#include "system/hw_accel.h"
 #include "accel/tcg/cpu-ldst.h"
 #include "accel/tcg/helper-retaddr.h"
 #include "accel/tcg/probe.h"
@@ -761,6 +762,14 @@ void HELPER(exception_return)(CPUARMState *env, uint64_t new_pc)
     arm_call_el_change_hook(cpu);
     bql_unlock();
 
+    if (new_el < 2 && hwaccel_enabled()) {
+        CPUState *cs = env_cpu(env);
+
+        cs->exception_index = EXCP_HWACCEL;
+        cs->exit_request = 1;
+        cpu_loop_exit(cs);
+        /* never returns */
+    }
     return;
 
 illegal_return:
-- 
2.49.0

[RFC PATCH RESEND 37/42] accel/hvf: Emulate HVC at EL2

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 target/arm/hvf/hvf.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/target/arm/hvf/hvf.c b/target/arm/hvf/hvf.c
index 3907ea8791e..a4f823f834d 100644
--- a/target/arm/hvf/hvf.c
+++ b/target/arm/hvf/hvf.c
@@ -17,6 +17,7 @@
 #include "system/hvf.h"
 #include "system/hvf_int.h"
 #include "system/hw_accel.h"
+#include "system/tcg.h"
 #include "hvf_arm.h"
 #include "cpregs.h"
 
@@ -1117,6 +1118,8 @@ static void hvf_raise_exception(CPUState *cpu, uint32_t excp,
     env->exception.syndrome = syndrome;
 
     arm_cpu_do_interrupt(cpu);
+    cpu->interrupt_request &= ~CPU_INTERRUPT_EXITTB;
+    cpu->exception_index = -1;
 }
 
 static void hvf_psci_cpu_off(ARMCPU *arm_cpu)
@@ -2090,6 +2093,9 @@ int hvf_vcpu_exec(CPUState *cpu)
                 /* SMCCC 1.3 section 5.2 says every unknown SMCCC call returns -1 */
                 env->xregs[0] = -1;
             }
+        } else if (tcg_enabled()) {
+            /* FIXME check we are called from EL1 (so EL0 -> UDEF) ? */
+            hvf_raise_exception(cpu, EXCP_HVC, syn_aa64_hvc(0), 2);
         } else {
             trace_hvf_unknown_hvc(env->pc, env->xregs[0]);
             hvf_raise_exception(cpu, EXCP_UDEF, syn_uncategorized(), 1);
-- 
2.49.0

[RFC PATCH RESEND 38/42] accel/tcg: Introduce TCGCPUOps::rebuild_tb_hflags handler

[Philippe Mathieu-Daudé]

In order to allow rebuilding target specific TB flags,
introduce tcg_rebuild_tb_flags() which dispatches to
a TCGCPUOps handler.

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/tcg/internal-common.h | 1 +
 include/accel/tcg/cpu-ops.h | 2 ++
 include/system/accel-ops.h  | 8 ++++++++
 accel/tcg/cpu-exec.c        | 9 +++++++++
 accel/tcg/tcg-accel-ops.c   | 1 +
 5 files changed, 21 insertions(+)

diff --git a/accel/tcg/internal-common.h b/accel/tcg/internal-common.h
index fb265d0cefa..a3828e34145 100644
--- a/accel/tcg/internal-common.h
+++ b/accel/tcg/internal-common.h
@@ -54,6 +54,7 @@ void tb_reset_jump(TranslationBlock *tb, int n);
 TranslationBlock *tb_link_page(TranslationBlock *tb);
 void cpu_restore_state_from_tb(CPUState *cpu, TranslationBlock *tb,
                                uintptr_t host_pc);
+void tcg_rebuild_tb_flags(CPUState *cpu);
 
 /**
  * tlb_init - initialize a CPU's TLB
diff --git a/include/accel/tcg/cpu-ops.h b/include/accel/tcg/cpu-ops.h
index dd8ea300168..bb047461973 100644
--- a/include/accel/tcg/cpu-ops.h
+++ b/include/accel/tcg/cpu-ops.h
@@ -67,6 +67,8 @@ struct TCGCPUOps {
      * Fill in all data required to select or compile a TranslationBlock.
      */
     TCGTBCPUState (*get_tb_cpu_state)(CPUState *cs);
+    /** @rebuild_tb_hflags: Callback to rebuild TB hflags.  */
+    void (*rebuild_tb_hflags)(CPUState *cpu);
     /**
      * @synchronize_from_tb: Synchronize state from a TCG #TranslationBlock
      *
diff --git a/include/system/accel-ops.h b/include/system/accel-ops.h
index 29ebcf45928..20999033c89 100644
--- a/include/system/accel-ops.h
+++ b/include/system/accel-ops.h
@@ -71,6 +71,14 @@ struct AccelOpsClass {
     void (*synchronize_pre_loadvm)(CPUState *cpu);
     void (*synchronize_pre_resume)(bool step_pending);
 
+    /**
+     * rebuild_tcg_tb_flags:
+     *
+     * Used to rebuild TCG TB flags when a hardware accelerator transitions
+     * to TCG, prior to calling TCG %exec_vcpu_thread() handler.
+     */
+    void (*rebuild_tcg_tb_flags)(CPUState *cpu);
+
     void (*handle_interrupt)(CPUState *cpu, int mask);
 
     void (*get_vcpu_stats)(CPUState *cpu, GString *buf);
diff --git a/accel/tcg/cpu-exec.c b/accel/tcg/cpu-exec.c
index 713bdb20564..7ded765889c 100644
--- a/accel/tcg/cpu-exec.c
+++ b/accel/tcg/cpu-exec.c
@@ -1028,6 +1028,15 @@ int cpu_exec(CPUState *cpu)
     return ret;
 }
 
+void tcg_rebuild_tb_flags(CPUState *cpu)
+{
+    const TCGCPUOps *tcg_ops = cpu->cc->tcg_ops;
+
+    if (tcg_ops->rebuild_tb_hflags) {
+        tcg_ops->rebuild_tb_hflags(cpu);
+    }
+}
+
 bool tcg_exec_realizefn(CPUState *cpu, Error **errp)
 {
     static bool tcg_target_initialized;
diff --git a/accel/tcg/tcg-accel-ops.c b/accel/tcg/tcg-accel-ops.c
index 0e4ef548f99..a141c4702e4 100644
--- a/accel/tcg/tcg-accel-ops.c
+++ b/accel/tcg/tcg-accel-ops.c
@@ -242,6 +242,7 @@ static void tcg_accel_ops_init(AccelClass *ac)
     ops->insert_breakpoint = tcg_insert_breakpoint;
     ops->remove_breakpoint = tcg_remove_breakpoint;
     ops->remove_all_breakpoints = tcg_remove_all_breakpoints;
+    ops->rebuild_tcg_tb_flags = tcg_rebuild_tb_flags;
 }
 
 static void tcg_accel_ops_class_init(ObjectClass *oc, const void *data)
-- 
2.49.0

[RFC PATCH RESEND 39/42] target/arm: Implement TCGCPUOps::rebuild_tb_hflags()

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 target/arm/cpu.h         | 2 ++
 target/arm/cpu.c         | 1 +
 target/arm/tcg/cpu-v7m.c | 1 +
 target/arm/tcg/hflags.c  | 5 +++++
 4 files changed, 9 insertions(+)

diff --git a/target/arm/cpu.h b/target/arm/cpu.h
index c31f69912b8..b703ec7edc9 100644
--- a/target/arm/cpu.h
+++ b/target/arm/cpu.h
@@ -3160,9 +3160,11 @@ void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, void
 
 /**
  * arm_rebuild_hflags:
+ * arm_cpu_rebuild_hflags:
  * Rebuild the cached TBFLAGS for arbitrary changed processor state.
  */
 void arm_rebuild_hflags(CPUARMState *env);
+void arm_cpu_rebuild_hflags(CPUState *cpu);
 
 /**
  * aa32_vfp_dreg:
diff --git a/target/arm/cpu.c b/target/arm/cpu.c
index 1a19e5cfb45..32a2e6c75e1 100644
--- a/target/arm/cpu.c
+++ b/target/arm/cpu.c
@@ -2743,6 +2743,7 @@ static const TCGCPUOps arm_tcg_ops = {
 
     .initialize = arm_translate_init,
     .translate_code = arm_translate_code,
+    .rebuild_tb_hflags = arm_cpu_rebuild_hflags,
     .get_tb_cpu_state = arm_get_tb_cpu_state,
     .synchronize_from_tb = arm_cpu_synchronize_from_tb,
     .debug_excp_handler = arm_debug_excp_handler,
diff --git a/target/arm/tcg/cpu-v7m.c b/target/arm/tcg/cpu-v7m.c
index 8e1a083b911..fa17029c65f 100644
--- a/target/arm/tcg/cpu-v7m.c
+++ b/target/arm/tcg/cpu-v7m.c
@@ -238,6 +238,7 @@ static const TCGCPUOps arm_v7m_tcg_ops = {
 
     .initialize = arm_translate_init,
     .translate_code = arm_translate_code,
+    .rebuild_tb_hflags = arm_cpu_rebuild_hflags,
     .get_tb_cpu_state = arm_get_tb_cpu_state,
     .synchronize_from_tb = arm_cpu_synchronize_from_tb,
     .debug_excp_handler = arm_debug_excp_handler,
diff --git a/target/arm/tcg/hflags.c b/target/arm/tcg/hflags.c
index 1ccec63bbd4..ea1174c661c 100644
--- a/target/arm/tcg/hflags.c
+++ b/target/arm/tcg/hflags.c
@@ -453,6 +453,11 @@ void arm_rebuild_hflags(CPUARMState *env)
     env->hflags = rebuild_hflags_internal(env);
 }
 
+void arm_cpu_rebuild_hflags(CPUState *cpu)
+{
+    arm_rebuild_hflags(cpu_env(cpu));
+}
+
 /*
  * If we have triggered a EL state change we can't rely on the
  * translator having passed it to us, we need to recompute.
-- 
2.49.0

[RFC PATCH RESEND 40/42] accel/split: Call TCGCPUOps::rebuild_tb_hflags()

[Philippe Mathieu-Daudé]

Call TCG rebuild_tb_hflags() when transitioning from
hardware accelerator to TCG.

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 accel/split/split-accel-ops.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
index 615faf1d96b..4b058034252 100644
--- a/accel/split/split-accel-ops.c
+++ b/accel/split/split-accel-ops.c
@@ -40,6 +40,8 @@ static void *split_cpu_thread_routine(void *arg)
     assert(swc->allowed);
     assert(hwc->allowed);
 
+    assert(swops->rebuild_tcg_tb_flags);
+
     rcu_register_thread();
     sw_force_rcu = mttcg_vcpu_register(cpu);
 
@@ -69,6 +71,7 @@ static void *split_cpu_thread_routine(void *arg)
                 r = hwops->exec_vcpu_thread(cpu);
                 trace_accel_split_exec_vcpu_thread_hw(r);
             } else {
+                swops->rebuild_tcg_tb_flags(cpu);
                 r = swops->exec_vcpu_thread(cpu);
                 trace_accel_split_exec_vcpu_thread_sw(r);
             }
-- 
2.49.0

[RFC PATCH RESEND 41/42] tests/functional: Add split_available() helper

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 python/qemu/utils/__init__.py          |  3 ++-
 python/qemu/utils/accel.py             | 10 ++++++++++
 tests/functional/qemu_test/testcase.py |  4 +++-
 3 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/python/qemu/utils/__init__.py b/python/qemu/utils/__init__.py
index d2fe5db223c..74218b9f2c6 100644
--- a/python/qemu/utils/__init__.py
+++ b/python/qemu/utils/__init__.py
@@ -23,7 +23,8 @@
 from typing import Optional
 
 # pylint: disable=import-error
-from .accel import hvf_available, kvm_available, list_accel, tcg_available
+from .accel import hvf_available, kvm_available, list_accel, tcg_available, \
+                   split_available
 
 
 __all__ = (
diff --git a/python/qemu/utils/accel.py b/python/qemu/utils/accel.py
index 376d1e30005..0f58998b2f7 100644
--- a/python/qemu/utils/accel.py
+++ b/python/qemu/utils/accel.py
@@ -90,3 +90,13 @@ def hvf_available(qemu_bin: str) -> bool:
     @param qemu_bin (str): path to the QEMU binary
     """
     return 'hvf' in list_accel(qemu_bin)
+
+def split_available(qemu_bin: str) -> bool:
+    """
+    Check if split accelerator is available.
+
+    @param qemu_bin (str): path to the QEMU binary
+    """
+    return 'split' in list_accel(qemu_bin) \
+           and tcg_available(qemu_bin) \
+           and hvf_available(qemu_bin)
diff --git a/tests/functional/qemu_test/testcase.py b/tests/functional/qemu_test/testcase.py
index 2082c6fce43..b6a534a29f0 100644
--- a/tests/functional/qemu_test/testcase.py
+++ b/tests/functional/qemu_test/testcase.py
@@ -23,7 +23,8 @@
 import uuid
 
 from qemu.machine import QEMUMachine
-from qemu.utils import hvf_available, kvm_available, tcg_available
+from qemu.utils import hvf_available, kvm_available, tcg_available, \
+                       split_available
 
 from .archive import archive_extract
 from .asset import Asset
@@ -319,6 +320,7 @@ def require_accelerator(self, accelerator):
         checker = {'tcg': tcg_available,
                    'kvm': kvm_available,
                    'hvf': hvf_available,
+                   'split': split_available,
                   }.get(accelerator)
         if checker is None:
             self.skipTest("Don't know how to check for the presence "
-- 
2.49.0

[RFC PATCH RESEND 42/42] tests/functional: Test Aarch64 virt machine with split-accelerator

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 tests/functional/meson.build                |  1 +
 tests/functional/test_aarch64_virt_split.py | 69 +++++++++++++++++++++
 2 files changed, 70 insertions(+)
 create mode 100644 tests/functional/test_aarch64_virt_split.py

diff --git a/tests/functional/meson.build b/tests/functional/meson.build
index e9f19d54a27..20d18eb8f5d 100644
--- a/tests/functional/meson.build
+++ b/tests/functional/meson.build
@@ -98,6 +98,7 @@ tests_aarch64_system_thorough = [
   'aarch64_tuxrun',
   'aarch64_virt',
   'aarch64_virt_gpu',
+  'aarch64_virt_split',
   'aarch64_xen',
   'aarch64_xlnx_versal',
   'multiprocess',
diff --git a/tests/functional/test_aarch64_virt_split.py b/tests/functional/test_aarch64_virt_split.py
new file mode 100644
index 00000000000..840ffe2f732
--- /dev/null
+++ b/tests/functional/test_aarch64_virt_split.py
@@ -0,0 +1,69 @@
+#!/usr/bin/env python3
+#
+# Copyright (c) 2024 Linaro Ltd.
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+import os
+import re
+
+from qemu_test import QemuSystemTest, Asset
+from qemu_test import exec_command, wait_for_console_pattern
+from qemu_test import exec_command_and_wait_for_pattern
+
+def test_nested_guest(test_instance):
+
+    # TODO:
+    # - kvm-unit-tests
+    # - boot KVM guest
+    pass
+
+def get_accel_transitions(vm):
+    hmp = vm.cmd('human-monitor-command', command_line='info accel')
+    for line in hmp.split('\r\n'):
+        match = re.search(r'Transitions: (\d+)', line)
+        if match is not None:
+            return int(match[1])
+    return 0
+
+
+class Aarch64VirtSplit(QemuSystemTest):
+
+    ASSET_KERNEL = Asset(
+        ('https://fileserver.linaro.org/s/Jao8KCct246QFy2/'
+         'download/Image-6.14.5+initramfs_with_kvm_unit_tests.xz'),
+         '9cc074434b41bce61f0534381dd6912ca6f33524077b02b5f4062a5e696d1b30')
+
+    def test_aarch64_virt_split(self):
+        self.set_machine('virt')
+        self.require_accelerator('split')
+
+        self.vm.set_console()
+
+        kernel = self.uncompress(self.ASSET_KERNEL)
+
+        self.vm.add_args('-accel', 'split')
+        self.vm.add_args('-cpu', 'host')
+        self.vm.add_args('-m', '2G')
+        self.vm.add_args('-M', 'virtualization=on,'
+                               'gic-version=max')
+        self.vm.add_args('-kernel', kernel)
+        self.vm.add_args('-netdev', 'user,id=net0')
+        self.vm.add_args('-append', 'console=ttyAMA0 '
+                                    'kvm-arm.mode=nvhe')
+
+        self.vm.launch()
+        wait_for_console_pattern(self, "CPU: All CPU(s) started at EL2")
+        wait_for_console_pattern(self, "kvm [1]: Hyp nVHE mode initialized")
+        wait_for_console_pattern(self, 'Welcome to Buildroot',
+                                 failure_message='Synchronous Exception at')
+        exec_command_and_wait_for_pattern(self, 'root', '#')
+
+        self.assertGreater(get_accel_transitions(self.vm), 13,
+                           "Not enough accel transitions")
+
+        test_nested_guest(self)
+
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
-- 
2.49.0

Re: [RFC PATCH RESEND 01/42] accel/split: Minimal stubs for split accelerator

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> From: Julian Armistead<julian.armistead@linaro.org>
> 
> Signed-off-by: Julian Armistead<julian.armistead@linaro.org>
> Signed-off-by: Philippe Mathieu-Daudé<philmd@linaro.org>
> ---
>   accel/split/split-accel-ops.c | 161 ++++++++++++++++++++++++++++++++++
>   accel/split/split-all.c       |  77 ++++++++++++++++
>   system/vl.c                   |   4 +
>   accel/Kconfig                 |   6 ++
>   accel/meson.build             |   1 +
>   accel/split/meson.build       |   9 ++
>   6 files changed, 258 insertions(+)
>   create mode 100644 accel/split/split-accel-ops.c
>   create mode 100644 accel/split/split-all.c
>   create mode 100644 accel/split/meson.build

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [RFC PATCH RESEND 02/42] accel/split: Define SplitAccelState

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé<philmd@linaro.org>
> ---
>   accel/split/split-accel.h | 38 ++++++++++++++++++++++++++++++++++++++
>   accel/split/split-all.c   |  5 ++++-
>   2 files changed, 42 insertions(+), 1 deletion(-)
>   create mode 100644 accel/split/split-accel.h

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [RFC PATCH RESEND 03/42] accel/split: Implement cpus_are_resettable()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-all.c | 13 ++++++++++++-
>   1 file changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/accel/split/split-all.c b/accel/split/split-all.c
> index 6bc95c7a7c8..c86d0e8583a 100644
> --- a/accel/split/split-all.c
> +++ b/accel/split/split-all.c
> @@ -30,7 +30,18 @@ static bool split_has_memory(MachineState *ms, AddressSpace *as,
>   
>   static bool split_cpus_are_resettable(AccelState *as)
>   {
> -    g_assert_not_reached();
> +    SplitAccelState *sas = SPLIT_ACCEL(as);
> +    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
> +    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
> +    bool rv = false;
> +
> +    if (hwc->cpus_are_resettable) {
> +        rv |= hwc->cpus_are_resettable(sas->hw);
> +    }
> +    if (swc->cpus_are_resettable) {
> +        rv |= swc->cpus_are_resettable(sas->sw);
> +    }
> +    return rv;
>   }
>   
>   static bool split_supports_guest_debug(AccelState *as)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [RFC PATCH RESEND 04/42] accel/split: Implement accel_init_machine()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Force to TCG + HVF for now.
> 
> Signed-off-by: Philippe Mathieu-Daudé<philmd@linaro.org>
> ---
>   accel/split/split-accel.h |  3 +++
>   accel/split/split-all.c   | 30 +++++++++++++++++++++++++++++-
>   2 files changed, 32 insertions(+), 1 deletion(-)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [RFC PATCH RESEND 04/42] accel/split: Implement accel_init_machine()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> +static void split_accel_instance_init(Object *obj)
> +{
> +    SplitAccelState *sas = SPLIT_ACCEL(obj);
> +
> +    sas->sw_name = g_strdup("tcg");
> +    sas->hw_name = g_strdup("hvf");

There's no real need to strdup.


r~

Re: [RFC PATCH RESEND 05/42] accel/split: Expose 'hw' and 'sw' properties

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> In preparation of other accelerator (or potential emulator),
> expose the "hw" and "sw" keys. Only HVF and TCG allowed so far.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-all.c | 46 +++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 46 insertions(+)
> 
> diff --git a/accel/split/split-all.c b/accel/split/split-all.c
> index 7cbe32ea768..28f626d0ff4 100644
> --- a/accel/split/split-all.c
> +++ b/accel/split/split-all.c
> @@ -8,6 +8,7 @@
>   
>   #include "qemu/osdep.h"
>   #include "qemu/accel.h"
> +#include "qapi/error.h"
>   #include "hw/boards.h"
>   #include "accel/accel-internal.h"
>   #include "split-accel.h"
> @@ -78,6 +79,42 @@ static void split_get_stats(AccelState *as, GString *buf)
>       g_assert_not_reached();
>   }
>   
> +static char *split_get_hw(Object *obj, Error **errp)
> +{
> +    SplitAccelState *s = SPLIT_ACCEL(obj);
> +
> +    return g_strdup(s->hw_name);
> +}
> +
> +static void split_set_hw(Object *obj, const char *value, Error **errp)
> +{
> +    SplitAccelState *s = SPLIT_ACCEL(obj);
> +
> +    if (strcmp(value, "hvf") == 0) {
> +        s->hw_name = g_strdup(value);

Since you've compared, you could assign the literal and not strdup.
I.e.

     if (strcmp(value, "hvf") == 0) {
         s->hw_name = "hvf";


> +    } else {
> +        error_setg(errp, "'%s' accelerator no supported", value);

not

> +static char *split_get_sw(Object *obj, Error **errp)
> +{
> +    SplitAccelState *s = SPLIT_ACCEL(obj);
> +
> +    return g_strdup(s->sw_name);
> +}
> +
> +static void split_set_sw(Object *obj, const char *value, Error **errp)
> +{
> +    SplitAccelState *s = SPLIT_ACCEL(obj);
> +
> +    if (strcmp(value, "tcg") == 0) {
> +        s->hw_name = g_strdup(value);
> +    } else {
> +        error_setg(errp, "'%s' emulator no supported", value);
> +    }
> +}

Maybe just skip this until such time as there really is a second sw emulator.


r~

Re: [RFC PATCH RESEND 06/42] accel/split: Empty setup_post()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-all.c | 6 ------
>   1 file changed, 6 deletions(-)
> 
> diff --git a/accel/split/split-all.c b/accel/split/split-all.c
> index 28f626d0ff4..8b9f8ff77a4 100644
> --- a/accel/split/split-all.c
> +++ b/accel/split/split-all.c
> @@ -37,11 +37,6 @@ static int split_accel_init_machine(MachineState *ms, AccelState *as)
>       return 0;
>   }
>   
> -static void split_setup_post(MachineState *ms, AccelState *accel)
> -{
> -    g_assert_not_reached();
> -}
> -
>   static bool split_has_memory(MachineState *ms, AddressSpace *as,
>                                hwaddr start_addr, hwaddr size)
>   {
> @@ -129,7 +124,6 @@ static void split_accel_class_init(ObjectClass *oc, const void *data)
>   
>       ac->name = "split";
>       ac->init_machine = split_accel_init_machine;
> -    ac->setup_post = split_setup_post;
>       ac->has_memory = split_has_memory;
>       ac->cpus_are_resettable = split_cpus_are_resettable;
>       ac->supports_guest_debug = split_supports_guest_debug;

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [RFC PATCH RESEND 07/42] accel/split: Implement supports_guest_debug()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-all.c | 13 ++++++++++++-
>   1 file changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/accel/split/split-all.c b/accel/split/split-all.c
> index 8b9f8ff77a4..cb910300931 100644
> --- a/accel/split/split-all.c
> +++ b/accel/split/split-all.c
> @@ -61,7 +61,18 @@ static bool split_cpus_are_resettable(AccelState *as)
>   
>   static bool split_supports_guest_debug(AccelState *as)
>   {
> -    g_assert_not_reached();
> +    SplitAccelState *sas = SPLIT_ACCEL(as);
> +    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
> +    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
> +    bool supported = false;
> +
> +    if (hwc->supports_guest_debug) {
> +        supported |= hwc->supports_guest_debug(sas->hw);
> +    }
> +    if (swc->supports_guest_debug) {
> +        supported |= swc->supports_guest_debug(sas->sw);
> +    }
> +    return supported;
>   }
>   
>   static int split_gdbstub_supported_sstep_flags(AccelState *as)

Require both to support debug?  Otherwise support depends on state.


r~

Re: [RFC PATCH RESEND 08/42] accel/split: Implement gdbstub_supported_sstep_flags()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-all.c | 14 +++++++++++++-
>   1 file changed, 13 insertions(+), 1 deletion(-)
> 
> diff --git a/accel/split/split-all.c b/accel/split/split-all.c
> index cb910300931..413954af96c 100644
> --- a/accel/split/split-all.c
> +++ b/accel/split/split-all.c
> @@ -77,7 +77,19 @@ static bool split_supports_guest_debug(AccelState *as)
>   
>   static int split_gdbstub_supported_sstep_flags(AccelState *as)
>   {
> -    g_assert_not_reached();
> +    SplitAccelState *sas = SPLIT_ACCEL(as);
> +    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
> +    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
> +    int flags = -1;

This only works if one of hw/sw implements the hook.


r~

> +
> +    if (hwc->gdbstub_supported_sstep_flags) {
> +        flags &= hwc->gdbstub_supported_sstep_flags(sas->hw);
> +    }
> +    if (swc->gdbstub_supported_sstep_flags) {
> +        flags &= swc->gdbstub_supported_sstep_flags(sas->sw);
> +    }
> +
> +    return flags;
>   }
>   
>   static void split_get_stats(AccelState *as, GString *buf)

Re: [RFC PATCH RESEND 09/42] accel/split: Add cpu_thread_routine() stub

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-accel-ops.c | 70 ++++++++++++++++++++++++++++++++++-
>   1 file changed, 69 insertions(+), 1 deletion(-)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

> 
> diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
> index e5c1d51d426..294ea79420e 100644
> --- a/accel/split/split-accel-ops.c
> +++ b/accel/split/split-accel-ops.c
> @@ -7,11 +7,79 @@
>    */
>   
>   #include "qemu/osdep.h"
> +#include "qemu/main-loop.h"
> +#include "qemu/guest-random.h"
> +#include "exec/cpu-common.h"
>   #include "system/accel-ops.h"
> +#include "system/cpus.h"
>   
>   static void *split_cpu_thread_routine(void *arg)
>   {
> -    g_assert_not_reached();
> +    CPUState *cpu = arg;
> +    int r;
> +
> +    /* TODO: check accel allowed */
> +
> +    rcu_register_thread();
> +
> +    bql_lock();
> +    qemu_thread_get_self(cpu->thread);
> +
> +    cpu->thread_id = qemu_get_thread_id();
> +    current_cpu = cpu;
> +
> +    /* TODO: init_vcpu_thread() */
> +    /* TODO: allocate AccelCPUState */
> +
> +    /* signal CPU creation */
> +    cpu_thread_signal_created(cpu);
> +    qemu_guest_random_seed_thread_part2(cpu->random_seed);
> +
> +    /* process any pending work */
> +    cpu->exit_request = 1;
> +
> +    do {
> +        r = 0;
> +
> +        if (cpu_can_run(cpu)) {
> +            r = 0; /* TODO: exec_vcpu_thread() */
> +            switch (r) {
> +            case 0:
> +                break;
> +            case EXCP_INTERRUPT:
> +                break;
> +            case EXCP_YIELD:
> +                break;
> +            case EXCP_DEBUG:
> +                cpu_handle_guest_debug(cpu);
> +                break;
> +            case EXCP_HALTED:
> +                /*
> +                 * Usually cpu->halted is set, but may have already been
> +                 * reset by another thread by the time we arrive here.
> +                 */
> +                break;
> +            case EXCP_ATOMIC:
> +                bql_unlock();
> +                cpu_exec_step_atomic(cpu);
> +                bql_lock();
> +                break;
> +            default:
> +                /* Ignore everything else? */
> +                break;
> +            }
> +        }
> +        qemu_wait_io_event(cpu);
> +    } while (!cpu->unplug || cpu_can_run(cpu));
> +
> +    /* TODO: destroy_vcpu_thread() */
> +
> +    cpu_thread_signal_destroyed(cpu);
> +    bql_unlock();
> +
> +    rcu_unregister_thread();
> +
> +    return NULL;
>   }
>   
>   static void split_ops_init(AccelClass *ac)

Re: [RFC PATCH RESEND 10/42] accel/split: Define and allocate AccelCPUState

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> AccelCPUState is where we store per-vCPU accelerator
> related information.
> 
> FIXME: structures might clash so add padding (ok with TCG+HVF)
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-accel.h     | 7 +++++++
>   accel/split/split-accel-ops.c | 9 ++++++++-
>   2 files changed, 15 insertions(+), 1 deletion(-)
> 
> diff --git a/accel/split/split-accel.h b/accel/split/split-accel.h
> index 19fb530f207..87a08454ce6 100644
> --- a/accel/split/split-accel.h
> +++ b/accel/split/split-accel.h
> @@ -38,4 +38,11 @@ struct SplitAccelState {
>       char *sw_name;
>   };
>   
> +struct AccelCPUState {
> +    char pad[128];

That's a pretty big fixme.
Is this a problem of name overloading?


r~

> +
> +    AccelState *accel;
> +    bool use_hw;
> +};
> +
>   #endif /* SPLIT_ACCEL_H */
> diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
> index 294ea79420e..efef0d60fde 100644
> --- a/accel/split/split-accel-ops.c
> +++ b/accel/split/split-accel-ops.c
> @@ -12,10 +12,13 @@
>   #include "exec/cpu-common.h"
>   #include "system/accel-ops.h"
>   #include "system/cpus.h"
> +#include "split-accel.h"
>   
>   static void *split_cpu_thread_routine(void *arg)
>   {
> +    AccelState *as = current_accel();
>       CPUState *cpu = arg;
> +    AccelCPUState *acs;
>       int r;
>   
>       /* TODO: check accel allowed */
> @@ -29,7 +32,10 @@ static void *split_cpu_thread_routine(void *arg)
>       current_cpu = cpu;
>   
>       /* TODO: init_vcpu_thread() */
> -    /* TODO: allocate AccelCPUState */
> +    cpu->accel = g_renew(AccelCPUState, cpu->accel, 1); /* XXX only with current TCG */
> +    acs = cpu->accel;
> +    acs->accel = as;
> +    acs->use_hw = false;
>   
>       /* signal CPU creation */
>       cpu_thread_signal_created(cpu);
> @@ -60,6 +66,7 @@ static void *split_cpu_thread_routine(void *arg)
>                    */
>                   break;
>               case EXCP_ATOMIC:
> +                assert(!acs->use_hw);
>                   bql_unlock();
>                   cpu_exec_step_atomic(cpu);
>                   bql_lock();

Re: [RFC PATCH RESEND 11/42] accel/split: Register MTTCG

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> FIXME: Use sw-hooks?

I'm OK with hard-coding tcg for sw, especially for now.

> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-accel-ops.c | 5 +++++
>   1 file changed, 5 insertions(+)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [RFC PATCH RESEND 12/42] accel/split: Have thread routine ready to dispatch over HW/SW

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> @@ -49,10 +56,12 @@ static void *split_cpu_thread_routine(void *arg)
>       cpu->exit_request = 1;
>   
>       do {
> -        r = 0;
> -
>           if (cpu_can_run(cpu)) {
> -            r = 0; /* TODO: exec_vcpu_thread() */
> +            if (acs->use_hw) {
> +                r = hwops->exec_vcpu_thread(cpu);
> +            } else {
> +                r = swops->exec_vcpu_thread(cpu);
> +            }

Maybe

   AccelOpsClass *curops = acs->use_hw ? hwops : swops;
   r = curops->exec_vcpu_thread(cpu);

Anyway,
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>


r~

Re: [RFC PATCH RESEND 13/42] accel/split: Implement cpu_reset_hold()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-accel-ops.c | 12 +++++++++++-
>   1 file changed, 11 insertions(+), 1 deletion(-)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

> 
> diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
> index 39495fdff14..d4fa07c2dec 100644
> --- a/accel/split/split-accel-ops.c
> +++ b/accel/split/split-accel-ops.c
> @@ -10,6 +10,7 @@
>   #include "qemu/main-loop.h"
>   #include "qemu/guest-random.h"
>   #include "exec/cpu-common.h"
> +#include "hw/core/cpu.h"
>   #include "system/accel-ops.h"
>   #include "system/cpus.h"
>   #include "split-accel.h"
> @@ -121,7 +122,16 @@ static void split_cpu_common_unrealize(CPUState *cpu)
>   
>   static void split_cpu_reset_hold(CPUState *cpu)
>   {
> -    g_assert_not_reached();
> +    SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
> +    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
> +    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
> +
> +    if (hwc->ops->cpu_reset_hold) {
> +        hwc->ops->cpu_reset_hold(cpu);
> +    }
> +    if (swc->ops->cpu_reset_hold) {
> +        swc->ops->cpu_reset_hold(cpu);
> +    }
>   }
>   
>   static void split_kick_vcpu_thread(CPUState *cpu)

Re: [RFC PATCH RESEND 14/42] accel/split: Implement synchronize_post_init()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-accel-ops.c | 11 ++++++++++-
>   1 file changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
> index d4fa07c2dec..6b3ddf21a17 100644
> --- a/accel/split/split-accel-ops.c
> +++ b/accel/split/split-accel-ops.c
> @@ -151,7 +151,16 @@ static void split_synchronize_post_reset(CPUState *cpu)
>   
>   static void split_synchronize_post_init(CPUState *cpu)
>   {
> -    g_assert_not_reached();
> +    SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
> +    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
> +    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
> +
> +    if (hwc->ops->synchronize_post_init) {
> +        hwc->ops->synchronize_post_init(cpu);
> +    }
> +    if (swc->ops->synchronize_post_init) {
> +        swc->ops->synchronize_post_init(cpu);
> +    }
>   }
>   
>   static void split_synchronize_state(CPUState *cpu)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [RFC PATCH RESEND 15/42] accel/split: Implement synchronize_pre_resume()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-accel-ops.c | 12 +++++++++++-
>   1 file changed, 11 insertions(+), 1 deletion(-)
> 
> diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
> index 6b3ddf21a17..65feb929404 100644
> --- a/accel/split/split-accel-ops.c
> +++ b/accel/split/split-accel-ops.c
> @@ -175,7 +175,17 @@ static void split_synchronize_pre_loadvm(CPUState *cpu)
>   
>   static void split_synchronize_pre_resume(bool step_pending)
>   {
> -    g_assert_not_reached();
> +    AccelState *as = current_accel();
> +    SplitAccelState *sas = SPLIT_ACCEL(as);
> +    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
> +    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
> +
> +    if (hwc->ops->synchronize_pre_resume) {
> +        hwc->ops->synchronize_pre_resume(step_pending);
> +    }
> +    if (swc->ops->synchronize_pre_resume) {
> +        swc->ops->synchronize_pre_resume(step_pending);
> +    }
>   }
>   
>   static void split_handle_interrupt(CPUState *cpu, int mask)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [RFC PATCH RESEND 16/42] accel/split: Implement synchronize_state()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-accel-ops.c | 11 ++++++++++-
>   1 file changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
> index 65feb929404..80efbd82091 100644
> --- a/accel/split/split-accel-ops.c
> +++ b/accel/split/split-accel-ops.c
> @@ -165,7 +165,16 @@ static void split_synchronize_post_init(CPUState *cpu)
>   
>   static void split_synchronize_state(CPUState *cpu)
>   {
> -    g_assert_not_reached();
> +    SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
> +    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
> +    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
> +
> +    if (hwc->ops->synchronize_state) {
> +        hwc->ops->synchronize_state(cpu);
> +    }
> +    if (swc->ops->synchronize_state) {
> +        swc->ops->synchronize_state(cpu);
> +    }
>   }

I suppose in the short-term this is ok, but it does more work than it should.

Let's make the simplifying assumption that even if there is another sw emulator, it'll 
still use CPUArchState.  So the "sync" to sw is pointless.

But syncing state to hw is only applicable when use_hw is true.
I'm not sure how to do the pointer chasing to find that from here.

r~

Re: [RFC PATCH RESEND 16/42] accel/split: Implement synchronize_state()

[Richard Henderson]

On 6/21/25 19:52, Richard Henderson wrote:
> On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
>> ---
>>   accel/split/split-accel-ops.c | 11 ++++++++++-
>>   1 file changed, 10 insertions(+), 1 deletion(-)
>>
>> diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
>> index 65feb929404..80efbd82091 100644
>> --- a/accel/split/split-accel-ops.c
>> +++ b/accel/split/split-accel-ops.c
>> @@ -165,7 +165,16 @@ static void split_synchronize_post_init(CPUState *cpu)
>>   static void split_synchronize_state(CPUState *cpu)
>>   {
>> -    g_assert_not_reached();
>> +    SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
>> +    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
>> +    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
>> +
>> +    if (hwc->ops->synchronize_state) {
>> +        hwc->ops->synchronize_state(cpu);
>> +    }
>> +    if (swc->ops->synchronize_state) {
>> +        swc->ops->synchronize_state(cpu);
>> +    }
>>   }
> 
> I suppose in the short-term this is ok, but it does more work than it should.
> 
> Let's make the simplifying assumption that even if there is another sw emulator, it'll 
> still use CPUArchState.  So the "sync" to sw is pointless.
> 
> But syncing state to hw is only applicable when use_hw is true.
> I'm not sure how to do the pointer chasing to find that from here.

The same applies to patches 14-18.

r~

Re: [RFC PATCH RESEND 19/42] accel/split: Implement kick_vcpu_thread()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-accel-ops.c | 8 +++++++-
>   1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
> index 746a6a2782d..63b6217f3fa 100644
> --- a/accel/split/split-accel-ops.c
> +++ b/accel/split/split-accel-ops.c
> @@ -136,7 +136,13 @@ static void split_cpu_reset_hold(CPUState *cpu)
>   
>   static void split_kick_vcpu_thread(CPUState *cpu)
>   {
> -    g_assert_not_reached();
> +    SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
> +    AccelClass *ac = cpu->accel->use_hw ? ACCEL_GET_CLASS(sas->hw)
> +                                        : ACCEL_GET_CLASS(sas->sw);

Move the ?: inside the ACCEL_GET_CLASS?

Anyway,
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>


r~

> +
> +    if (ac->ops->kick_vcpu_thread) {
> +        ac->ops->kick_vcpu_thread(cpu);
> +    }
>   }
>   
>   static bool split_cpu_thread_is_idle(CPUState *cpu)

Re: [RFC PATCH RESEND 20/42] accel/split: Implement cpu_common_realize()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-accel-ops.c | 10 +++++++++-
>   1 file changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
> index 63b6217f3fa..3278e01f18a 100644
> --- a/accel/split/split-accel-ops.c
> +++ b/accel/split/split-accel-ops.c
> @@ -15,6 +15,7 @@
>   #include "system/cpus.h"
>   #include "split-accel.h"
>   
> +#include "accel/accel-internal.h"
>   #include "accel/tcg/tcg-accel-ops-mttcg.h"
>   
>   static void *split_cpu_thread_routine(void *arg)
> @@ -112,7 +113,14 @@ static void split_ops_init(AccelClass *ac)
>   
>   static bool split_cpu_common_realize(CPUState *cpu, Error **errp)
>   {
> -    g_assert_not_reached();
> +    AccelState *as = current_accel();
> +    SplitAccelState *sas = SPLIT_ACCEL(as);
> +    bool success = false;
> +
> +    success |= accel_cpu_realize(sas->sw, cpu, errp);
> +    success |= accel_cpu_realize(sas->hw, cpu, errp);

&&

r~

Re: [RFC PATCH RESEND 21/42] accel/split: Set use_hw in cpu_thread_routine() and switch over

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Introduce the EXCP_HWACCEL definition to switch to
> hardware accelerator.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-accel.h     |  1 +
>   include/exec/cpu-common.h     |  1 +
>   accel/split/split-accel-ops.c | 11 +++++++++++
>   3 files changed, 13 insertions(+)
> 
> diff --git a/accel/split/split-accel.h b/accel/split/split-accel.h
> index 87a08454ce6..8a2888507f3 100644
> --- a/accel/split/split-accel.h
> +++ b/accel/split/split-accel.h
> @@ -42,6 +42,7 @@ struct AccelCPUState {
>       char pad[128];
>   
>       AccelState *accel;
> +
>       bool use_hw;

Stray.

> @@ -66,10 +69,14 @@ static void *split_cpu_thread_routine(void *arg)
>               }
>               switch (r) {
>               case 0:
> +                if (acs->use_hw) {
> +                    acs->use_hw = cpu_acceleratable(cpu);
> +                }
>                   break;

Why the conditional?  Why can't we enable use_hw here?

>               case EXCP_INTERRUPT:
>                   break;
>               case EXCP_YIELD:
> +                assert(!acs->use_hw);
>                   break;
>               case EXCP_DEBUG:
>                   cpu_handle_guest_debug(cpu);
> @@ -86,6 +93,10 @@ static void *split_cpu_thread_routine(void *arg)
>                   cpu_exec_step_atomic(cpu);
>                   bql_lock();
>                   break;
> +            case EXCP_HWACCEL:
> +                assert(!acs->use_hw);
> +                acs->use_hw = true;

Why the unconditional?  Why isn't it more correct to check cpu_acceleratable?

r~

Re: [RFC PATCH RESEND 23/42] target/arm: Implement SysemuCPUOps::can_accelerate() handler

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> ARM hardware can only accelerate EL0 and EL1.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   target/arm/cpu.c | 8 ++++++++
>   1 file changed, 8 insertions(+)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [RFC PATCH RESEND 24/42] accel/split: Implement handle_interrupt()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-accel-ops.c | 10 +++++++++-
>   1 file changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
> index 399bf71a9ec..5a36e22f205 100644
> --- a/accel/split/split-accel-ops.c
> +++ b/accel/split/split-accel-ops.c
> @@ -249,7 +249,15 @@ static void split_synchronize_pre_resume(bool step_pending)
>   
>   static void split_handle_interrupt(CPUState *cpu, int mask)
>   {
> -    g_assert_not_reached();
> +    SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
> +    AccelClass *ac = cpu->accel->use_hw ? ACCEL_GET_CLASS(sas->hw)
> +                                        : ACCEL_GET_CLASS(sas->sw);
> +
> +    if (ac->ops->handle_interrupt) {
> +        ac->ops->handle_interrupt(cpu, mask);
> +    } else {
> +        generic_handle_interrupt(cpu, mask);
> +    }

Better to have handle_interrupt always filled-in.

r~

Re: [RFC PATCH RESEND 25/42] accel/split: Empty ops_init()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-accel-ops.c | 6 ------
>   1 file changed, 6 deletions(-)
> 
> diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
> index 5a36e22f205..375821f1064 100644
> --- a/accel/split/split-accel-ops.c
> +++ b/accel/split/split-accel-ops.c
> @@ -124,11 +124,6 @@ static void *split_cpu_thread_routine(void *arg)
>       return NULL;
>   }
>   
> -static void split_ops_init(AccelClass *ac)
> -{
> -    g_assert_not_reached();
> -}
> -
>   static bool split_cpu_common_realize(CPUState *cpu, Error **errp)
>   {
>       AccelState *as = current_accel();
> @@ -306,7 +301,6 @@ static void split_accel_ops_class_init(ObjectClass *oc, const void *data)
>   {
>       AccelOpsClass *ops = ACCEL_OPS_CLASS(oc);
>   
> -    ops->ops_init = split_ops_init;
>       ops->cpu_common_realize = split_cpu_common_realize;
>       ops->cpu_common_unrealize = split_cpu_common_unrealize;
>       ops->cpu_reset_hold = split_cpu_reset_hold;

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [RFC PATCH RESEND 26/42] accel/split: Empty set/get_virtual_clock()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-accel-ops.c | 12 ------------
>   1 file changed, 12 deletions(-)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [RFC PATCH RESEND 27/42] accel/split: Empty get_elapsed_ticks()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-accel-ops.c | 7 -------
>   1 file changed, 7 deletions(-)
> 
> diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
> index d19b4641a2e..034b5ebc96c 100644
> --- a/accel/split/split-accel-ops.c
> +++ b/accel/split/split-accel-ops.c
> @@ -255,11 +255,6 @@ static void split_handle_interrupt(CPUState *cpu, int mask)
>       }
>   }
>   
> -static int64_t split_get_elapsed_ticks(void)
> -{
> -    g_assert_not_reached();
> -}
> -
>   static int split_update_guest_debug(CPUState *cpu)
>   {
>       g_assert_not_reached();
> @@ -307,8 +302,6 @@ static void split_accel_ops_class_init(ObjectClass *oc, const void *data)
>       ops->handle_interrupt = split_handle_interrupt;
>       ops->get_vcpu_stats = split_get_vcpu_stats;
>   
> -    ops->get_elapsed_ticks = split_get_elapsed_ticks;
> -
>       ops->update_guest_debug = split_update_guest_debug;
>       ops->insert_breakpoint = split_insert_breakpoint;
>       ops->remove_breakpoint = split_remove_breakpoint;

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [RFC PATCH RESEND 28/42] accel/split: Empty cpu_thread_is_idle()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-accel-ops.c | 6 ------
>   1 file changed, 6 deletions(-)

Why does this not defer to the current sub-accel?

r~

> 
> diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
> index 034b5ebc96c..e1d91ace2fa 100644
> --- a/accel/split/split-accel-ops.c
> +++ b/accel/split/split-accel-ops.c
> @@ -166,11 +166,6 @@ static void split_kick_vcpu_thread(CPUState *cpu)
>       }
>   }
>   
> -static bool split_cpu_thread_is_idle(CPUState *cpu)
> -{
> -    g_assert_not_reached();
> -}
> -
>   static void split_synchronize_post_reset(CPUState *cpu)
>   {
>       SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
> @@ -291,7 +286,6 @@ static void split_accel_ops_class_init(ObjectClass *oc, const void *data)
>       ops->cpu_reset_hold = split_cpu_reset_hold;
>       ops->cpu_thread_routine = split_cpu_thread_routine;
>       ops->kick_vcpu_thread = split_kick_vcpu_thread;
> -    ops->cpu_thread_is_idle = split_cpu_thread_is_idle;
>   
>       ops->synchronize_post_reset = split_synchronize_post_reset;
>       ops->synchronize_post_init = split_synchronize_post_init;

Re: [RFC PATCH RESEND 30/42] accel/split: Implement remove_all_breakpoints()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-accel-ops.c | 11 ++++++++++-
>   1 file changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
> index e1d91ace2fa..0f3d48fc68d 100644
> --- a/accel/split/split-accel-ops.c
> +++ b/accel/split/split-accel-ops.c
> @@ -269,7 +269,16 @@ static int split_remove_breakpoint(CPUState *cpu, int type,
>   
>   static void split_remove_all_breakpoints(CPUState *cpu)
>   {
> -    g_assert_not_reached();
> +    SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
> +    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
> +    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
> +
> +    if (hwc->ops->remove_all_breakpoints) {
> +        hwc->ops->remove_all_breakpoints(cpu);
> +    }
> +    if (swc->ops->remove_all_breakpoints) {
> +        swc->ops->remove_all_breakpoints(cpu);
> +    }
>   }
>   
>   static void split_get_vcpu_stats(CPUState *cpu, GString *buf)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [RFC PATCH RESEND 31/42] accel/split: Implement remove_breakpoint()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-accel-ops.c | 13 ++++++++++++-
>   1 file changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
> index 0f3d48fc68d..40cd39aea5c 100644
> --- a/accel/split/split-accel-ops.c
> +++ b/accel/split/split-accel-ops.c
> @@ -264,7 +264,18 @@ static int split_insert_breakpoint(CPUState *cpu, int type,
>   static int split_remove_breakpoint(CPUState *cpu, int type,
>                                      vaddr addr, vaddr len)
>   {
> -    g_assert_not_reached();
> +    SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
> +    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
> +    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
> +    int err = 0;
> +
> +    if (hwc->ops->remove_breakpoint) {
> +        err |= hwc->ops->remove_breakpoint(cpu, type, addr, len);
> +    }
> +    if (swc->ops->remove_breakpoint) {
> +        err |= swc->ops->remove_breakpoint(cpu, type, addr, len);
> +    }
> +    return err;
>   }
>   
>   static void split_remove_all_breakpoints(CPUState *cpu)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [RFC PATCH RESEND 32/42] accel/split: Implement insert_breakpoint()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-accel-ops.c | 13 ++++++++++++-
>   1 file changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
> index 40cd39aea5c..eb52d690c7c 100644
> --- a/accel/split/split-accel-ops.c
> +++ b/accel/split/split-accel-ops.c
> @@ -258,7 +258,18 @@ static int split_update_guest_debug(CPUState *cpu)
>   static int split_insert_breakpoint(CPUState *cpu, int type,
>                                      vaddr addr, vaddr len)
>   {
> -    g_assert_not_reached();
> +    SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
> +    AccelClass *hwc = ACCEL_GET_CLASS(sas->hw);
> +    AccelClass *swc = ACCEL_GET_CLASS(sas->sw);
> +    int err = 0;
> +
> +    if (hwc->ops->insert_breakpoint) {
> +        err |= hwc->ops->insert_breakpoint(cpu, type, addr, len);
> +    }
> +    if (swc->ops->insert_breakpoint) {
> +        err |= swc->ops->insert_breakpoint(cpu, type, addr, len);
> +    }
> +    return err;
>   }
>   
>   static int split_remove_breakpoint(CPUState *cpu, int type,

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [RFC PATCH RESEND 33/42] accel/split: Implement update_guest_debug()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-accel-ops.c | 11 ++++++++++-
>   1 file changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
> index eb52d690c7c..d59e70e0d9b 100644
> --- a/accel/split/split-accel-ops.c
> +++ b/accel/split/split-accel-ops.c
> @@ -252,7 +252,16 @@ static void split_handle_interrupt(CPUState *cpu, int mask)
>   
>   static int split_update_guest_debug(CPUState *cpu)
>   {
> -    g_assert_not_reached();
> +    SplitAccelState *sas = SPLIT_ACCEL(cpu->accel->accel);
> +    AccelClass *ac = cpu->accel->use_hw ? ACCEL_GET_CLASS(sas->hw)
> +                                        : ACCEL_GET_CLASS(sas->sw);
> +    int ret = 0;
> +
> +    if (ac->ops->update_guest_debug) {
> +        ret = ac->ops->update_guest_debug(cpu);
> +    }
> +
> +    return ret;
>   }
>   
>   static int split_insert_breakpoint(CPUState *cpu, int type,

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [RFC PATCH RESEND 34/42] accel/split: Implement get_[vcpu]_stats()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-accel.h     |  5 +++++
>   accel/split/split-accel-ops.c | 24 +++++++++++++++++++++++-
>   accel/split/split-all.c       | 16 +++++++++++++++-
>   3 files changed, 43 insertions(+), 2 deletions(-)
> 
> diff --git a/accel/split/split-accel.h b/accel/split/split-accel.h
> index 8a2888507f3..80a19bbb1e5 100644
> --- a/accel/split/split-accel.h
> +++ b/accel/split/split-accel.h
> @@ -36,6 +36,11 @@ struct SplitAccelState {
>   
>       char *hw_name;
>       char *sw_name;
> +
> +    struct {
> +        unsigned transitions; /* number of HW <-> SW transitions */
> +        uint64_t exec_count[2]; /* SW:0 HW:1 */
> +    } stats;
>   };
>   
>   struct AccelCPUState {
> diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
> index d59e70e0d9b..615faf1d96b 100644
> --- a/accel/split/split-accel-ops.c
> +++ b/accel/split/split-accel-ops.c
> @@ -9,6 +9,7 @@
>   #include "qemu/osdep.h"
>   #include "qemu/main-loop.h"
>   #include "qemu/guest-random.h"
> +#include "qapi/type-helpers.h"
>   #include "exec/cpu-common.h"
>   #include "exec/cpu-interrupt.h"
>   #include "hw/core/cpu.h"
> @@ -16,6 +17,7 @@
>   #include "system/cpus.h"
>   #include "system/hw_accel.h"
>   #include "system/tcg.h"
> +#include "monitor/monitor.h"
>   #include "split-accel.h"
>   #include "trace.h"
>   
> @@ -70,6 +72,7 @@ static void *split_cpu_thread_routine(void *arg)
>                   r = swops->exec_vcpu_thread(cpu);
>                   trace_accel_split_exec_vcpu_thread_sw(r);
>               }
> +            sas->stats.exec_count[acs->use_hw]++;
>               switch (r) {
>               case 0:
>                   if (acs->use_hw) {
> @@ -103,6 +106,7 @@ static void *split_cpu_thread_routine(void *arg)
>                   assert(!acs->use_hw);
>                   trace_accel_split_hw_accelerate();
>                   acs->use_hw = true;
> +                sas->stats.transitions++;

There's also the transitions in case 0.
Otherwise,
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>



r~

Re: [RFC PATCH RESEND 35/42] target/arm: Emulate EL2 under TCG

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> When using hardware acceleration and TCG is available,
> expose EL2 and EL3 features as available (they will be
> emulated).
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   target/arm/cpu64.c | 7 +++++--
>   1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
> index 200da1c489b..b5c2796f3a1 100644
> --- a/target/arm/cpu64.c
> +++ b/target/arm/cpu64.c
> @@ -738,20 +738,23 @@ static void aarch64_a53_initfn(Object *obj)
>   
>   static void aarch64_host_initfn(Object *obj)
>   {
> -#if defined(CONFIG_KVM)
>       ARMCPU *cpu = ARM_CPU(obj);
> +#if defined(CONFIG_KVM)
>       kvm_arm_set_cpu_features_from_host(cpu);
>       if (arm_feature(&cpu->env, ARM_FEATURE_AARCH64)) {
>           aarch64_add_sve_properties(obj);
>           aarch64_add_pauth_properties(obj);
>       }
>   #elif defined(CONFIG_HVF)
> -    ARMCPU *cpu = ARM_CPU(obj);
>       hvf_arm_set_cpu_features_from_host(cpu);
>       aarch64_add_pauth_properties(obj);
>   #else
>       g_assert_not_reached();
>   #endif
> +    if (tcg_enabled()) {
> +        set_feature(&cpu->env, ARM_FEATURE_EL2);
> +        set_feature(&cpu->env, ARM_FEATURE_EL3);
> +    }
>   }
>   
>   static void aarch64_max_initfn(Object *obj)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [RFC PATCH RESEND 36/42] target/arm: Have ERET switch to hw accel for EL0/EL1

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> When hardware accelerator available, use it to
> directly run EL0 and EL1.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   target/arm/tcg/helper-a64.c | 9 +++++++++
>   1 file changed, 9 insertions(+)
> 
> diff --git a/target/arm/tcg/helper-a64.c b/target/arm/tcg/helper-a64.c
> index 4f618ae390e..5962cff4e7a 100644
> --- a/target/arm/tcg/helper-a64.c
> +++ b/target/arm/tcg/helper-a64.c
> @@ -29,6 +29,7 @@
>   #include "internals.h"
>   #include "qemu/crc32c.h"
>   #include "exec/cpu-common.h"
> +#include "system/hw_accel.h"
>   #include "accel/tcg/cpu-ldst.h"
>   #include "accel/tcg/helper-retaddr.h"
>   #include "accel/tcg/probe.h"
> @@ -761,6 +762,14 @@ void HELPER(exception_return)(CPUARMState *env, uint64_t new_pc)
>       arm_call_el_change_hook(cpu);
>       bql_unlock();
>   
> +    if (new_el < 2 && hwaccel_enabled()) {

Because you're raising EXCP_HWACCEL, I really think this should be

   if (split_enabled() && arm_cpu_can_accelerate(cs))


r~

> +        CPUState *cs = env_cpu(env);
> +
> +        cs->exception_index = EXCP_HWACCEL;
> +        cs->exit_request = 1;
> +        cpu_loop_exit(cs);
> +        /* never returns */
> +    }
>       return;
>   
>   illegal_return:

Re: [RFC PATCH RESEND 37/42] accel/hvf: Emulate HVC at EL2

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   target/arm/hvf/hvf.c | 6 ++++++
>   1 file changed, 6 insertions(+)
> 
> diff --git a/target/arm/hvf/hvf.c b/target/arm/hvf/hvf.c
> index 3907ea8791e..a4f823f834d 100644
> --- a/target/arm/hvf/hvf.c
> +++ b/target/arm/hvf/hvf.c
> @@ -17,6 +17,7 @@
>   #include "system/hvf.h"
>   #include "system/hvf_int.h"
>   #include "system/hw_accel.h"
> +#include "system/tcg.h"
>   #include "hvf_arm.h"
>   #include "cpregs.h"
>   
> @@ -1117,6 +1118,8 @@ static void hvf_raise_exception(CPUState *cpu, uint32_t excp,
>       env->exception.syndrome = syndrome;
>   
>       arm_cpu_do_interrupt(cpu);
> +    cpu->interrupt_request &= ~CPU_INTERRUPT_EXITTB;

Why is this needed?

r~

> +    cpu->exception_index = -1;
>   }
>   
>   static void hvf_psci_cpu_off(ARMCPU *arm_cpu)
> @@ -2090,6 +2093,9 @@ int hvf_vcpu_exec(CPUState *cpu)
>                   /* SMCCC 1.3 section 5.2 says every unknown SMCCC call returns -1 */
>                   env->xregs[0] = -1;
>               }
> +        } else if (tcg_enabled()) {
> +            /* FIXME check we are called from EL1 (so EL0 -> UDEF) ? */
> +            hvf_raise_exception(cpu, EXCP_HVC, syn_aa64_hvc(0), 2);
>           } else {
>               trace_hvf_unknown_hvc(env->pc, env->xregs[0]);
>               hvf_raise_exception(cpu, EXCP_UDEF, syn_uncategorized(), 1);

Re: [RFC PATCH RESEND 38/42] accel/tcg: Introduce TCGCPUOps::rebuild_tb_hflags handler

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> index 29ebcf45928..20999033c89 100644
> --- a/include/system/accel-ops.h
> +++ b/include/system/accel-ops.h
> @@ -71,6 +71,14 @@ struct AccelOpsClass {
>       void (*synchronize_pre_loadvm)(CPUState *cpu);
>       void (*synchronize_pre_resume)(bool step_pending);
>   
> +    /**
> +     * rebuild_tcg_tb_flags:
> +     *
> +     * Used to rebuild TCG TB flags when a hardware accelerator transitions
> +     * to TCG, prior to calling TCG %exec_vcpu_thread() handler.
> +     */
> +    void (*rebuild_tcg_tb_flags)(CPUState *cpu);

I think this isn't right.  This should be part of synchronize between hw/sw.


r~

Re: [RFC PATCH RESEND 40/42] accel/split: Call TCGCPUOps::rebuild_tb_hflags()

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> Call TCG rebuild_tb_hflags() when transitioning from
> hardware accelerator to TCG.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   accel/split/split-accel-ops.c | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/accel/split/split-accel-ops.c b/accel/split/split-accel-ops.c
> index 615faf1d96b..4b058034252 100644
> --- a/accel/split/split-accel-ops.c
> +++ b/accel/split/split-accel-ops.c
> @@ -40,6 +40,8 @@ static void *split_cpu_thread_routine(void *arg)
>       assert(swc->allowed);
>       assert(hwc->allowed);
>   
> +    assert(swops->rebuild_tcg_tb_flags);
> +
>       rcu_register_thread();
>       sw_force_rcu = mttcg_vcpu_register(cpu);
>   
> @@ -69,6 +71,7 @@ static void *split_cpu_thread_routine(void *arg)
>                   r = hwops->exec_vcpu_thread(cpu);
>                   trace_accel_split_exec_vcpu_thread_hw(r);
>               } else {
> +                swops->rebuild_tcg_tb_flags(cpu);
>                   r = swops->exec_vcpu_thread(cpu)

You should not need to do this before every exec, only on transitions.


r~

Re: [RFC PATCH RESEND 41/42] tests/functional: Add split_available() helper

[Richard Henderson]

On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
> +    return 'split' in list_accel(qemu_bin) \
> +           and tcg_available(qemu_bin) \
> +           and hvf_available(qemu_bin)

Do you really need 3 checks?  Kconfig gates on this already.

r~

Re: [RFC PATCH RESEND 37/42] accel/hvf: Emulate HVC at EL2

[Philippe Mathieu-Daudé]

On 22/6/25 05:28, Richard Henderson wrote:
> On 6/20/25 10:27, Philippe Mathieu-Daudé wrote:
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
>> ---
>>   target/arm/hvf/hvf.c | 6 ++++++
>>   1 file changed, 6 insertions(+)
>>
>> diff --git a/target/arm/hvf/hvf.c b/target/arm/hvf/hvf.c
>> index 3907ea8791e..a4f823f834d 100644
>> --- a/target/arm/hvf/hvf.c
>> +++ b/target/arm/hvf/hvf.c
>> @@ -17,6 +17,7 @@
>>   #include "system/hvf.h"
>>   #include "system/hvf_int.h"
>>   #include "system/hw_accel.h"
>> +#include "system/tcg.h"
>>   #include "hvf_arm.h"
>>   #include "cpregs.h"
>> @@ -1117,6 +1118,8 @@ static void hvf_raise_exception(CPUState *cpu, 
>> uint32_t excp,
>>       env->exception.syndrome = syndrome;
>>       arm_cpu_do_interrupt(cpu);
>> +    cpu->interrupt_request &= ~CPU_INTERRUPT_EXITTB;
> 
> Why is this needed?

Left-over from previous intents, indeed unneeded.

> 
> r~
> 
>> +    cpu->exception_index = -1;
>>   }
>>   static void hvf_psci_cpu_off(ARMCPU *arm_cpu)
>> @@ -2090,6 +2093,9 @@ int hvf_vcpu_exec(CPUState *cpu)
>>                   /* SMCCC 1.3 section 5.2 says every unknown SMCCC 
>> call returns -1 */
>>                   env->xregs[0] = -1;
>>               }
>> +        } else if (tcg_enabled()) {

As you noted you last night during an epic debugging session, since HVF
doesn't execute the instructions but we get here being trapped, if we
are going to resume under TCG we need to skip the HVC call (otherwise
we keep looping over it) with:

                 env->pc += 4;

(Same applies to "Emulate SMC at EL3" patch).

>> +            /* FIXME check we are called from EL1 (so EL0 -> UDEF) ? */
>> +            hvf_raise_exception(cpu, EXCP_HVC, syn_aa64_hvc(0), 2);
>>           } else {
>>               trace_hvf_unknown_hvc(env->pc, env->xregs[0]);
>>               hvf_raise_exception(cpu, EXCP_UDEF, syn_uncategorized(), 
>> 1);
> 

Re: [RFC PATCH RESEND 00/42] accel/split/arm: Run EL2 using TCG and EL1/EL0 in hardware with HVF

[Philippe Mathieu-Daudé]

Cc'ing Salil

On 20/6/25 19:27, Philippe Mathieu-Daudé wrote:
> Hi,
> 
> This RFC is a proof-of-concept we can have QEMU run both software
> emulator (TCG) and hardware accelerator (here HVF). Unfortunately
> I'm exhausted so I'll let Alex explain what this is about,
> otherwise I'll post the real cover letter next Monday after
> getting some rest.
> 
> Full work available in the split-accel-rfc-v5 tag:
> https://gitlab.com/philmd/qemu/-/commits/split-accel-rfc-v5
> 
> Regards,
> 
> Phil.
> 
> Julian Armistead (1):
>    accel/split: Minimal stubs for split accelerator
> 
> Philippe Mathieu-Daudé (41):
>    accel/split: Define SplitAccelState
>    accel/split: Implement cpus_are_resettable()
>    accel/split: Implement accel_init_machine()
>    accel/split: Expose 'hw' and 'sw' properties
>    accel/split: Empty setup_post()
>    accel/split: Implement supports_guest_debug()
>    accel/split: Implement gdbstub_supported_sstep_flags()
>    accel/split: Add cpu_thread_routine() stub
>    accel/split: Define and allocate AccelCPUState
>    accel/split: Register MTTCG
>    accel/split: Have thread routine ready to dispatch over HW/SW
>    accel/split: Implement cpu_reset_hold()
>    accel/split: Implement synchronize_post_init()
>    accel/split: Implement synchronize_pre_resume()
>    accel/split: Implement synchronize_state()
>    accel/split: Implement synchronize_post_reset()
>    accel/split: Implement synchronize_pre_loadvm()
>    accel/split: Implement kick_vcpu_thread()
>    accel/split: Implement cpu_common_realize()
>    accel/split: Set use_hw in cpu_thread_routine() and switch over
>    accel/split: Add few trace events in cpu_thread_routine handler
>    target/arm: Implement SysemuCPUOps::can_accelerate() handler
>    accel/split: Implement handle_interrupt()
>    accel/split: Empty ops_init()
>    accel/split: Empty set/get_virtual_clock()
>    accel/split: Empty get_elapsed_ticks()
>    accel/split: Empty cpu_thread_is_idle()
>    accel/split: Kludge qemu_tcg_mttcg_enabled()
>    accel/split: Implement remove_all_breakpoints()
>    accel/split: Implement remove_breakpoint()
>    accel/split: Implement insert_breakpoint()
>    accel/split: Implement update_guest_debug()
>    accel/split: Implement get_[vcpu]_stats()
>    target/arm: Emulate EL2 under TCG
>    target/arm: Have ERET switch to hw accel for EL0/EL1
>    accel/hvf: Emulate HVC at EL2
>    accel/tcg: Introduce TCGCPUOps::rebuild_tb_hflags handler
>    target/arm: Implement TCGCPUOps::rebuild_tb_hflags()
>    accel/split: Call TCGCPUOps::rebuild_tb_hflags()
>    tests/functional: Add split_available() helper
>    tests/functional: Test Aarch64 virt machine with split-accelerator
> 
>   meson.build                                 |   1 +
>   accel/split/split-accel.h                   |  54 +++
>   accel/split/trace.h                         |   2 +
>   accel/tcg/internal-common.h                 |   1 +
>   include/accel/tcg/cpu-ops.h                 |   2 +
>   include/exec/cpu-common.h                   |   1 +
>   include/system/accel-ops.h                  |   8 +
>   target/arm/cpu.h                            |   2 +
>   accel/split/split-accel-ops.c               | 381 ++++++++++++++++++++
>   accel/split/split-all.c                     | 196 ++++++++++
>   accel/tcg/cpu-exec.c                        |   9 +
>   accel/tcg/tcg-accel-ops.c                   |   1 +
>   accel/tcg/tcg-all.c                         |   1 +
>   system/vl.c                                 |   4 +
>   target/arm/cpu.c                            |   9 +
>   target/arm/cpu64.c                          |   7 +-
>   target/arm/hvf/hvf.c                        |   6 +
>   target/arm/tcg/cpu-v7m.c                    |   1 +
>   target/arm/tcg/helper-a64.c                 |   9 +
>   target/arm/tcg/hflags.c                     |   5 +
>   accel/Kconfig                               |   6 +
>   accel/meson.build                           |   1 +
>   accel/split/meson.build                     |   9 +
>   accel/split/trace-events                    |   9 +
>   python/qemu/utils/__init__.py               |   3 +-
>   python/qemu/utils/accel.py                  |  10 +
>   tests/functional/meson.build                |   1 +
>   tests/functional/qemu_test/testcase.py      |   4 +-
>   tests/functional/test_aarch64_virt_split.py |  69 ++++
>   29 files changed, 808 insertions(+), 4 deletions(-)
>   create mode 100644 accel/split/split-accel.h
>   create mode 100644 accel/split/trace.h
>   create mode 100644 accel/split/split-accel-ops.c
>   create mode 100644 accel/split/split-all.c
>   create mode 100644 accel/split/meson.build
>   create mode 100644 accel/split/trace-events
>   create mode 100644 tests/functional/test_aarch64_virt_split.py
>