* [PATCH v3 0/7] accel/tcg: cut per-block dispatch overhead
@ 2026-08-22 19:08 Matt Turner
2026-08-22 19:08 ` [PATCH v3 1/7] accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags Matt Turner
` (16 more replies)
0 siblings, 17 replies; 47+ messages in thread
From: Matt Turner @ 2026-08-22 19:08 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
For guests running large amounts of code, most of what TCG executes is not
translated guest work but the fixed overhead around it. Blocks are short and
there are a great many of them, so the constant cost at each end of a block
(the interrupt poll and the can_do_io stores on entry, the dispatch on exit)
ends up dominating everything else.
The workload throughout is qemu-alpha running an emulated alpha gcc 16.2.0
compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64
host, in a --static --enable-lto --target-list=alpha-linux-user build. It
executes 34.2 billion TBs at 6.04 guest instructions each, and 24.6% of its
TB exits cannot use goto_tb. That is a representative shape for any guest
whose text is much larger than a page: indirect calls and returns
everywhere, plus direct branches that merely crossed a page boundary.
The first three patches are ordinary cleanups that stand on their own; patch
3 picked up review tags in v2. The remaining four are marked RFC individually
and are where the interesting questions are.
1 accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags
curr_cflags() recomputed three unlikely tests on every one of the run's
8.4 billion dispatches, from state that changes only when gdb enables
single-step, when one-insn-per-tb is toggled, or when the log mask
moves. Fold each into tcg_cflags where it changes. -5.15%
2 accel/tcg: enlarge the TB jump cache to 64K entries
4096 entries is too small for a guest running a large program;
tb_htable_lookup() is 6.10% of samples. 16 bits is the knee of the
sizing curve, at 1 MiB per vCPU. -5.92%, -8.71% wall
3 accel/tcg: skip the can_do_io stores in user-only builds
Two stores per TB that nothing in a user-only build reads: 68 billion
of them over the run. -4.57%, -4.53% wall
4 RFC: tcg: probe the TB jump cache inline instead of calling a helper
95.8% of those 8.4 billion helper_lookup_tb_ptr() calls hit the jump
cache. Emit the probe inline (hash, four guarded loads, goto_ptr) and
call the helper only on a miss. -34.67%, -25.94% wall
5 RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds
translator_use_goto_tb() refuses to chain across a page. In user-only
builds the invalidation path already covers what that was protecting
against: every mmap/mprotect/munmap reaches page_set_flags(), which
invalidates and unlinks. Lift it there for runs that can never acquire
a breakpoint, keep it for system mode. -2.75%, -4.84% wall
6 RFC: accel/tcg: poison the jump cache instead of polling for indirect
exits
A block only needs the icount_decr poll if it can leave by goto_tb;
every other exit already passes through a dispatch. Give the inline
probe its own jump cache base pointer and point it at zeroes when an
exit is requested, so every dispatch misses into the helper, which
returns the epilogue. Emit the poll only in blocks that emitted a
goto_tb. -2.52%, -1.96% wall
7 RFC: tcg: fold a guest displacement into the host addressing mode
tcg_gen_qemu_ld/st cannot express a based access, so a target with a
displacement in its encodings materializes the address with an lea
that the host addressing mode would have done for free. Fold a
preceding constant add into a new argument on the op, opt-in per
backend, wired up for x86_64 user-only. -5.70%, -3.20% wall
Each percentage is against the patch before it. Every stage was measured in
one session on the same host, so end to end, from an unmodified LTO build of
the same base to the full series:
instructions retired: 1,646,994,254,249 -> 819,262,147,022 -50.26%
wall clock: 133.19s -> 77.30s -41.96%
The two figures do not track each other, and that is the interesting part:
what the series removes is cheap, well-predicted, highly pipelined work, so
it retires far more instructions than it saves time. Patch 4 also cuts L1
icache load misses by 39.0%, because a dispatch no longer jumps into qemu's
.text and evicts translated code; qemu's own .text falls from 38.8% to 5.3%
of profile samples.
Every revision was built and measured separately, so the series bisects, and
the emulated compiler produces byte-identical assembly output at every step,
which is the correctness check these patches most need. Three new alpha
tests cover the hazards the series creates: tests/tcg/alpha/test-xpage-chain.c
and tests/tcg/alpha/gdbstub/xpage-bp.py (patch 5) and test-indirect-irq.c
(patch 6). Each fails or hangs if the mechanism it covers is removed, which
is what makes them tests of the new behavior rather than of the old.
Changes since v2
================
The biggest change is that "RFC: accel/tcg: only poll for interrupts in
blocks that can close a cycle" is dropped. Richard pointed out that it let a
straight-line run of arbitrary length go unchecked, since a block with no
backward edge polled nowhere. Patch 6 now keeps the poll where a block can
leave by goto_tb and relies on the dispatch everywhere else, which holds the
one-block bound without any analysis of the guest's control flow graph. All
of v2's measurements for that patch were taken with the dropped patch
underneath and have been replaced by a fresh measurement of the series as it
now stands.
1 Rewritten as folding the dynamic bits into CPUState::tcg_cflags where
they change, rather than caching curr_cflags() in a second field.
Monitor-side updates ('one-insn-per-tb on', 'log nochain') are queued
with async_safe_run_on_cpu() so each CPU writes its own cflags.
2 Sizing curve re-measured on top of the new patch 1, now with wall clock
alongside instructions retired.
3 Back to #ifndef CONFIG_USER_ONLY. QEMU's IS_ENABLED() is IS_EMPTY(),
true only for a symbol Meson defines empty, and CONFIG_USER_ONLY is
defined as 1, so v2's test was always false and the stores were emitted
after all. Review tags carried over; the numbers are from the working
form.
4 Folded into tcg_gen_lookup_and_goto_ptr() instead of adding a second
entry point beside it (Richard), which changed all 38 call sites.
cs_base is compared too, which is what lets the probe be enabled
generically rather than per target. Audited which targets may pass a
real PC: five do, the rest pass NULL and keep the helper call. The
breakpoint poison now happens in cpu_breakpoint_insert() rather than
only from the poisoned CPU's own main loop, since a thread dispatching
indirectly need never reach that loop.
5 Only take the shortcut when no gdbstub was requested: the same-page rule
also forces a breakpoint check on entry to every page, and without that
a chain established earlier runs past a breakpoint set later. Reported
by Richard. Changed translator_use_goto_tb() rather than
translator_is_same_page(), which i386, riscv and s390x use for something
else and which v2 perturbed as a side effect. Added the gdbstub half of
the test.
6 Rebased onto the removal described above, with the deferred-emission
machinery moved here from the dropped patch, and re-measured.
7 Unchanged in substance, re-measured on the new baseline.
What I would most like reviewed
===============================
- Patch 5 reverses a deliberate decision made in d3a2a1d803 on the
strength of an argument about the user-only invalidation paths, plus a
gate on whether gdb can ever attach.
- Patch 6's un-poison in the main loop races a concurrent poison from
another thread. I believe the existing barrier around
icount_decr.u16.high covers it, but my testing was single-threaded user
mode.
- Patch 4 treats cpu flags, cflags and cs_base as translation-time
constants in its guards, reads a jump cache entry without qatomic_read(),
and leaves one_insn_per_tb and -d nochain toggles visible only at the
next non-inline exit.
- Patch 7 only examines the immediately preceding op, refuses any access
with a slow path (so user-only, and no alignment check), and leaves the
i128 pairs alone.
- Patch 2's 1 MiB per vCPU is easy to justify for a single-vCPU
linux-user process and less obvious for system emulation with many
vCPUs. It may want to be sized per target or made tunable rather than
raised unconditionally.
Patches 4 and 7 are wired up for alpha and x86_64 respectively; everything
else is target-independent, and no other backend changes behavior or needs
touching.
v2: https://lore.kernel.org/qemu-devel/20260817190038.580257-1-mattst88@gmail.com/
Matt Turner (7):
accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags
accel/tcg: enlarge the TB jump cache to 64K entries
accel/tcg: skip the can_do_io stores in user-only builds
RFC: tcg: probe the TB jump cache inline instead of calling a helper
RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds
RFC: accel/tcg: poison the jump cache instead of polling for indirect
exits
RFC: tcg: fold a guest displacement into the host addressing mode
accel/tcg/cpu-exec-common.c | 33 +++-
accel/tcg/cpu-exec.c | 122 +++++++++++++++
accel/tcg/internal-common.h | 13 +-
accel/tcg/tb-jmp-cache.h | 2 +-
accel/tcg/tcg-accel-ops.c | 1 +
accel/tcg/tcg-all.c | 1 +
accel/tcg/translator.c | 94 +++++++++++-
cpu-common.c | 11 ++
cpu-target.c | 3 +
gdbstub/user.c | 14 ++
include/gdbstub/user.h | 11 ++
include/hw/core/cpu.h | 11 ++
include/system/tcg.h | 21 +++
include/tcg/tcg-op-common.h | 16 +-
include/tcg/tcg-op.h | 12 ++
include/tcg/tcg-opc.h | 9 +-
include/tcg/tcg.h | 2 +
stubs/meson.build | 1 +
stubs/tcg-cflags.c | 20 +++
target/alpha/translate.c | 4 +-
target/arm/tcg/translate-a64.c | 4 +-
target/arm/tcg/translate.c | 10 +-
target/avr/translate.c | 4 +-
target/hexagon/translate.c | 4 +-
target/hppa/translate.c | 6 +-
target/i386/tcg/translate.c | 2 +-
.../tcg/insn_trans/trans_branch.c.inc | 2 +-
target/loongarch/tcg/translate.c | 4 +-
target/m68k/translate.c | 2 +-
target/microblaze/translate.c | 4 +-
target/mips/tcg/nanomips_translate.c.inc | 2 +-
target/mips/tcg/translate.c | 6 +-
target/or1k/translate.c | 4 +-
target/ppc/translate.c | 4 +-
target/riscv/tcg/insn_trans/trans_rvzce.c.inc | 4 +-
target/riscv/tcg/translate.c | 2 +-
target/rx/translate.c | 4 +-
target/s390x/tcg/translate.c | 5 +-
target/sh4/translate.c | 4 +-
target/sparc/translate.c | 4 +-
target/tricore/translate.c | 4 +-
tcg/tcg-op-ldst.c | 3 +-
tcg/tcg-op.c | 101 +++++++++++-
tcg/tcg.c | 86 ++++++++++-
tcg/x86_64/tcg-target.c.inc | 61 ++++++++
tcg/x86_64/tcg-target.h | 3 +
tests/tcg/alpha/Makefile.target | 18 ++-
tests/tcg/alpha/gdbstub/xpage-bp.py | 34 +++++
tests/tcg/alpha/test-indirect-irq.c | 55 +++++++
tests/tcg/alpha/test-xpage-chain.c | 144 ++++++++++++++++++
util/log.c | 4 +
51 files changed, 932 insertions(+), 63 deletions(-)
create mode 100644 stubs/tcg-cflags.c
create mode 100644 tests/tcg/alpha/gdbstub/xpage-bp.py
create mode 100644 tests/tcg/alpha/test-indirect-irq.c
create mode 100644 tests/tcg/alpha/test-xpage-chain.c
--
2.54.0
^ permalink raw reply [flat|nested] 47+ messages in thread
* [PATCH v3 1/7] accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags
2026-08-22 19:08 [PATCH v3 0/7] accel/tcg: cut per-block dispatch overhead Matt Turner
@ 2026-08-22 19:08 ` Matt Turner
2026-08-25 21:47 ` Richard Henderson
2026-08-26 7:46 ` Alex Bennée
2026-08-22 19:08 ` [PATCH v3 2/7] accel/tcg: enlarge the TB jump cache to 64K entries Matt Turner
` (15 subsequent siblings)
16 siblings, 2 replies; 47+ messages in thread
From: Matt Turner @ 2026-08-22 19:08 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
curr_cflags() is called once per TB dispatch, from helper_lookup_tb_ptr()
and from the cpu_exec() loop. It recomputes the same value every time:
uint32_t cflags = cpu->tcg_cflags;
if (unlikely(cpu_single_stepping(cpu))) { ... }
else if (qatomic_read(&one_insn_per_tb)) { ... }
else if (qemu_loglevel_mask(CPU_LOG_TB_NOCHAIN)) { ... }
That is three loads and three branches on the hottest path in the
interpreter, for state that changes only when gdb enables single-step,
when one-insn-per-tb is toggled, or when the log mask changes.
None of the three has to be sampled at dispatch time. Fold each into
CPUState::tcg_cflags where it changes and curr_cflags() becomes a single
load of a field that TB lookup has to read anyway.
The derived bits -- CF_COUNT_MASK, CF_NO_GOTO_TB, CF_NO_GOTO_PTR and
CF_SINGLE_STEP -- are never set by tcg_cflags_set(), so tcg_update_cflags()
can recompute them in place without disturbing the rest, and conversely
tcg_cflags_set() ORs in its bits without disturbing them.
There are three places to call it:
- tcg_exec_realizefn(), so that a CPU created after the command line has
been parsed starts out with the right value. This covers user-only,
where tcg_cpu_init_cflags() is not reached. linux-user's cpu_copy()
copies tcg_cflags wholesale, so a cloned thread inherits it.
- cpu_single_step(), which changes one CPU and runs either on that CPU's
thread or with it stopped.
- tcg_set_one_insn_per_tb() and qemu_set_log_internal(), which change
every CPU. Both can be reached from the monitor while the vCPUs are
running -- 'one-insn-per-tb on' and 'log nochain' -- so the update is
queued with async_safe_run_on_cpu() and each CPU writes its own cflags
with the others halted.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling
the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, in a
build configured with --enable-lto:
before: 1,646,994,254,249 instructions
after: 1,562,204,796,597 instructions -5.15%
That workload issues 8.4 billion dispatches, so the per-call saving is
small but the aggregate is not. The emulated compiler produces
byte-identical output before and after.
Wall clock does not move: 133.19s to 132.58s, a 0.46% difference against a
run-to-run spread larger than that. The removed work is a few predictable
loads and branches that the host executes largely in parallel with the
surrounding dispatch, so this patch is worth taking for the instruction
count and for what it enables, not for a time saving that can be measured
on its own.
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
accel/tcg/cpu-exec-common.c | 33 ++++++++++++++++++++++++++++++---
accel/tcg/cpu-exec.c | 3 +++
accel/tcg/internal-common.h | 11 +++++++++--
accel/tcg/tcg-all.c | 1 +
cpu-target.c | 3 +++
include/system/tcg.h | 12 ++++++++++++
stubs/meson.build | 1 +
stubs/tcg-cflags.c | 16 ++++++++++++++++
util/log.c | 4 ++++
9 files changed, 79 insertions(+), 5 deletions(-)
create mode 100644 stubs/tcg-cflags.c
diff --git ./accel/tcg/cpu-exec-common.c ./accel/tcg/cpu-exec-common.c
index 44e84344f3..dd2be475e2 100644
--- ./accel/tcg/cpu-exec-common.c
+++ ./accel/tcg/cpu-exec-common.c
@@ -36,9 +36,16 @@ void tcg_cflags_set(CPUState *cpu, uint32_t flags)
cpu->tcg_cflags |= flags;
}
-uint32_t curr_cflags(CPUState *cpu)
+/*
+ * The bits of CPUState::tcg_cflags that tcg_cflags_set() never sets, because
+ * they are derived from gdb single-step, one-insn-per-tb and -d nochain.
+ */
+#define CF_DERIVED (CF_COUNT_MASK | CF_NO_GOTO_TB | CF_NO_GOTO_PTR | \
+ CF_SINGLE_STEP)
+
+void tcg_update_cflags(CPUState *cpu)
{
- uint32_t cflags = cpu->tcg_cflags;
+ uint32_t cflags = cpu->tcg_cflags & ~CF_DERIVED;
/*
* Record gdb single-step. We should be exiting the TB by raising
@@ -55,7 +62,27 @@ uint32_t curr_cflags(CPUState *cpu)
cflags |= CF_NO_GOTO_TB;
}
- return cflags;
+ cpu->tcg_cflags = cflags;
+}
+
+static void tcg_update_cflags_work(CPUState *cpu, run_on_cpu_data data)
+{
+ tcg_update_cflags(cpu);
+}
+
+void tcg_update_all_cflags(void)
+{
+ CPUState *cpu;
+
+ /*
+ * one-insn-per-tb and -d nochain can both be changed from the monitor
+ * while the vCPUs are running. Have each CPU update its own cflags
+ * with the others halted, so that no dispatch can read a value that
+ * another thread is in the middle of writing.
+ */
+ CPU_FOREACH(cpu) {
+ async_safe_run_on_cpu(cpu, tcg_update_cflags_work, RUN_ON_CPU_NULL);
+ }
}
/* exit the current TB, but without causing any exception to be raised */
diff --git ./accel/tcg/cpu-exec.c ./accel/tcg/cpu-exec.c
index 257211235d..148e0f583e 100644
--- ./accel/tcg/cpu-exec.c
+++ ./accel/tcg/cpu-exec.c
@@ -1068,6 +1068,9 @@ bool tcg_exec_realizefn(CPUState *cpu, Error **errp)
tcg_target_initialized = true;
}
+ /* Pick up one-insn-per-tb and -d nochain from the command line. */
+ tcg_update_cflags(cpu);
+
cpu->tb_jmp_cache = g_new0(CPUJumpCache, 1);
tlb_init(cpu);
#ifndef CONFIG_USER_ONLY
diff --git ./accel/tcg/internal-common.h ./accel/tcg/internal-common.h
index 9e7be2d78d..853d1b51ee 100644
--- ./accel/tcg/internal-common.h
+++ ./accel/tcg/internal-common.h
@@ -69,8 +69,15 @@ void tlb_destroy(CPUState *cpu);
bool tcg_exec_realizefn(CPUState *cpu, Error **errp);
void tcg_exec_unrealizefn(CPUState *cpu);
-/* current cflags for hashing/comparison */
-uint32_t curr_cflags(CPUState *cpu);
+/*
+ * Current cflags for hashing/comparison. Everything that feeds into the
+ * value is folded into CPUState::tcg_cflags when it changes, by
+ * tcg_update_cflags(), so that TB dispatch only has to load it.
+ */
+static inline uint32_t curr_cflags(CPUState *cpu)
+{
+ return cpu->tcg_cflags;
+}
void tb_check_watchpoint(CPUState *cpu, uintptr_t retaddr);
diff --git ./accel/tcg/tcg-all.c ./accel/tcg/tcg-all.c
index 7186c10cf0..c9874a286a 100644
--- ./accel/tcg/tcg-all.c
+++ ./accel/tcg/tcg-all.c
@@ -254,6 +254,7 @@ static void tcg_set_one_insn_per_tb(Object *obj, bool value, Error **errp)
s->one_insn_per_tb = value;
/* Set the global also: this changes the behaviour */
qatomic_set(&one_insn_per_tb, value);
+ tcg_update_all_cflags();
}
static void tcg_accel_class_init(ObjectClass *oc, const void *data)
diff --git ./cpu-target.c ./cpu-target.c
index 4783845c9b..50be591acf 100644
--- ./cpu-target.c
+++ ./cpu-target.c
@@ -24,6 +24,7 @@
#include "exec/replay-core.h"
#include "exec/log.h"
#include "hw/core/cpu.h"
+#include "system/tcg.h"
#include "trace/trace-root.h"
/* enable or disable single step mode. EXCP_DEBUG is returned by the
@@ -35,6 +36,8 @@ void cpu_single_step(CPUState *cpu, unsigned flags)
cpu->singlestep_flags, flags);
cpu->singlestep_flags = flags;
+ tcg_update_cflags(cpu);
+
#if !defined(CONFIG_USER_ONLY)
const AccelOpsClass *ops = cpus_get_accel();
if (ops->update_guest_debug) {
diff --git ./include/system/tcg.h ./include/system/tcg.h
index 7622dcea30..2c2dbc753b 100644
--- ./include/system/tcg.h
+++ ./include/system/tcg.h
@@ -17,6 +17,18 @@ extern bool tcg_allowed;
#define tcg_enabled() 0
#endif
+/*
+ * Recompute the parts of CPUState::tcg_cflags that TB dispatch consumes but
+ * tcg_cflags_set() does not provide: gdb single-step, one-insn-per-tb and
+ * the CPU_LOG_TB_NOCHAIN log flag. Call whenever one of those changes.
+ *
+ * tcg_update_cflags() updates one CPU and must be called from that CPU's
+ * thread, or with it stopped. tcg_update_all_cflags() updates every CPU
+ * and is safe to call from the monitor while the vCPUs run.
+ */
+void tcg_update_cflags(CPUState *cpu);
+void tcg_update_all_cflags(void);
+
/**
* qemu_tcg_mttcg_enabled:
* Check whether we are running MultiThread TCG or not.
diff --git ./stubs/meson.build ./stubs/meson.build
index 3b2f2680b1..0025e79226 100644
--- ./stubs/meson.build
+++ ./stubs/meson.build
@@ -3,6 +3,7 @@
# below, so that it is clear who needs the stubbed functionality.
stub_ss.add(files('cpu-get-clock.c'))
+stub_ss.add(files('tcg-cflags.c'))
stub_ss.add(files('fdset.c'))
stub_ss.add(files('iothread-lock.c'))
stub_ss.add(files('is-daemonized.c'))
diff --git ./stubs/tcg-cflags.c ./stubs/tcg-cflags.c
new file mode 100644
index 0000000000..cb278e94aa
--- /dev/null
+++ ./stubs/tcg-cflags.c
@@ -0,0 +1,16 @@
+/*
+ * Stub for tcg_update_all_cflags(), for binaries that link util/log.c
+ * or cpu-target.c but not TCG.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+#include "qemu/osdep.h"
+#include "system/tcg.h"
+
+void tcg_update_cflags(CPUState *cpu)
+{
+}
+
+void tcg_update_all_cflags(void)
+{
+}
diff --git ./util/log.c ./util/log.c
index 7cffbc1bf8..3fa46a67fa 100644
--- ./util/log.c
+++ ./util/log.c
@@ -27,6 +27,7 @@
#include "qemu/thread.h"
#include "qemu/lockable.h"
#include "qemu/rcu.h"
+#include "system/tcg.h"
#ifdef CONFIG_LINUX
#include <sys/syscall.h>
#endif
@@ -301,6 +302,9 @@ static bool qemu_set_log_internal(const char *filename, bool changed_name,
#endif
qemu_loglevel = log_flags;
+ /* CPU_LOG_TB_NOCHAIN feeds into the per-CPU cflags. */
+ tcg_update_all_cflags();
+
daemonized = is_daemonized();
need_to_open_file = false;
if (!daemonized) {
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH v3 2/7] accel/tcg: enlarge the TB jump cache to 64K entries
2026-08-22 19:08 [PATCH v3 0/7] accel/tcg: cut per-block dispatch overhead Matt Turner
2026-08-22 19:08 ` [PATCH v3 1/7] accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags Matt Turner
@ 2026-08-22 19:08 ` Matt Turner
2026-08-25 21:50 ` Richard Henderson
2026-08-22 19:08 ` [PATCH v3 3/7] accel/tcg: skip the can_do_io stores in user-only builds Matt Turner
` (14 subsequent siblings)
16 siblings, 1 reply; 47+ messages in thread
From: Matt Turner @ 2026-08-22 19:08 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
The per-CPU TB jump cache has held 4096 entries since it was introduced.
That is too small for guests running large programs: an emulated compiler
misses often enough that the fallback qht lookup shows up prominently in
a profile.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling
the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host. The
compile performs 34.2 billion TB executions, of which 8.4 billion take
the indirect dispatch path.
Sizing curve, on top of the preceding patch, instructions retired and
wall clock:
12 bits ( 64 KiB): 1,562,204,796,597 132.58s
14 bits ( 256 KiB): 1,493,318,515,396 -4.41% 124.67s -5.97%
16 bits ( 1 MiB): 1,469,772,951,575 -5.92% 121.04s -8.71%
18 bits ( 4 MiB): 1,462,309,832,762 -6.39% 119.82s -9.62%
16 bits is the knee. 18 buys another 0.47% of instructions for four times
the memory. It does show a further 1.01% of wall clock, which is outside
the 0.70% run-to-run spread at 16 bits, so the effect is probably real --
but paying four times the memory for it is a poor trade, and instructions
retired does not account for the data cache pressure of a 4 MiB table.
In a perf profile the mechanism is visible directly: tb_htable_lookup(),
which is where qht_lookup_custom() lands once it is inlined in an LTO
build, falls from 6.10% of samples to 1.66%.
The cost is memory: the cache grows from 64 KiB to 1 MiB per vCPU. That
is easy to justify for a single-vCPU linux-user process and less obvious
for system emulation with many vCPUs, so this may want to be sized by
target or made tunable rather than raised unconditionally.
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
accel/tcg/tb-jmp-cache.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git ./accel/tcg/tb-jmp-cache.h ./accel/tcg/tb-jmp-cache.h
index c3a505e394..268dacd7ba 100644
--- ./accel/tcg/tb-jmp-cache.h
+++ ./accel/tcg/tb-jmp-cache.h
@@ -12,7 +12,7 @@
#include "qemu/rcu.h"
#include "exec/cpu-common.h"
-#define TB_JMP_CACHE_BITS 12
+#define TB_JMP_CACHE_BITS 16
#define TB_JMP_CACHE_SIZE (1 << TB_JMP_CACHE_BITS)
/*
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH v3 3/7] accel/tcg: skip the can_do_io stores in user-only builds
2026-08-22 19:08 [PATCH v3 0/7] accel/tcg: cut per-block dispatch overhead Matt Turner
2026-08-22 19:08 ` [PATCH v3 1/7] accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags Matt Turner
2026-08-22 19:08 ` [PATCH v3 2/7] accel/tcg: enlarge the TB jump cache to 64K entries Matt Turner
@ 2026-08-22 19:08 ` Matt Turner
2026-08-22 19:08 ` [PATCH v3 4/7] RFC: tcg: probe the TB jump cache inline instead of calling a helper Matt Turner
` (13 subsequent siblings)
16 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-08-22 19:08 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
Every translation block stores to cpu->neg.can_do_io twice: false before
the first instruction, true before the last one. Nothing reads it in a
user-only build. There is no memory-mapped I/O in linux-user, and every
reader is in system_ss: cputlb.c, watchpoint.c, icount-common.c and
tcg-accel-ops-icount.c.
Two stores per TB is not much on its own, but TBs are short. An emulated
alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines,
-O2) executes 34.2 billion TBs at 6.04 guest instructions each, so this is
68 billion stores for nothing.
Measured on an x86-64 host, LTO build, on top of the preceding two
patches:
before: 1,469,772,951,575 instructions
after: 1,402,667,803,616 instructions -4.57%
before: 121.04s wall clock
after: 115.56s wall clock -4.53%
The emulated compiler produces byte-identical output.
v3: Use #ifndef CONFIG_USER_ONLY again rather than
if (IS_ENABLED(CONFIG_USER_ONLY)). QEMU's IS_ENABLED() is IS_EMPTY(),
which is only true for a symbol Meson defines empty; CONFIG_USER_ONLY
is defined as 1, so the test was always false and v2 emitted the two
stores after all. The measurements above are from the working form.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
accel/tcg/translator.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git ./accel/tcg/translator.c ./accel/tcg/translator.c
index 57daded60f..6c8fcd7a20 100644
--- ./accel/tcg/translator.c
+++ ./accel/tcg/translator.c
@@ -21,12 +21,14 @@
#include "disas/disas.h"
#include "tb-internal.h"
+#ifndef CONFIG_USER_ONLY
static void set_can_do_io(DisasContextBase *db, bool val)
{
QEMU_BUILD_BUG_ON(sizeof_field(CPUState, neg.can_do_io) != 1);
tcg_gen_st8_i32(tcg_constant_i32(val), tcg_env,
offsetof(CPUState, neg.can_do_io) - sizeof(CPUState));
}
+#endif
bool translator_io_start(DisasContextBase *db)
{
@@ -210,17 +212,25 @@ void translator_loop(CPUState *cpu, TranslationBlock *tb, int *max_insns,
/*
* Manage can_do_io for the translation block: set to false before
* the first insn and set to true before the last insn.
+ *
+ * Nothing reads can_do_io in user-only builds. There is no MMIO
+ * there, and every reader (cputlb.c, watchpoint.c, icount) is in
+ * system_ss, so skip the two stores per TB entirely.
*/
if (db->num_insns == 1) {
tcg_debug_assert(first_insn_start == db->insn_start);
} else {
tcg_debug_assert(first_insn_start != db->insn_start);
+#ifndef CONFIG_USER_ONLY
tcg_ctx->emit_before_op = first_insn_start;
set_can_do_io(db, false);
+#endif
}
+#ifndef CONFIG_USER_ONLY
tcg_ctx->emit_before_op = db->insn_start;
set_can_do_io(db, true);
tcg_ctx->emit_before_op = NULL;
+#endif
/* May be used by disas_log or plugin callbacks. */
tb->size = db->pc_next - db->pc_first;
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH v3 4/7] RFC: tcg: probe the TB jump cache inline instead of calling a helper
2026-08-22 19:08 [PATCH v3 0/7] accel/tcg: cut per-block dispatch overhead Matt Turner
` (2 preceding siblings ...)
2026-08-22 19:08 ` [PATCH v3 3/7] accel/tcg: skip the can_do_io stores in user-only builds Matt Turner
@ 2026-08-22 19:08 ` Matt Turner
2026-08-25 22:28 ` Richard Henderson
2026-08-22 19:08 ` [PATCH v3 5/7] RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds Matt Turner
` (12 subsequent siblings)
16 siblings, 1 reply; 47+ messages in thread
From: Matt Turner @ 2026-08-22 19:08 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
Every indirect branch that cannot use goto_tb ends in
tcg_gen_lookup_and_goto_ptr(), which calls helper_lookup_tb_ptr(). For an
emulated compiler that is 8.4 billion helper calls in a single translation
unit: 24.6% of all TB exits take this path, because jsr/ret/jmp have a
register destination and because goto_tb is restricted to same-page
targets.
The helper itself is already tight, but each call pays for a call frame,
the can_do_io store, the get_tb_cpu_state() indirect call through
TCGCPUOps, curr_cflags(), and a breakpoint check, before it gets to the
jump cache probe that almost always hits (95.8% for this workload).
Emit the probe inline instead. The destination PC is already in a TCG
temp, and the flags, cflags and cs_base the destination must match are
constants at translation time, so the fast path is a hash, four guarded
loads and a goto_ptr. Only a miss calls the helper, which still owns
filling the cache.
tcg_gen_lookup_and_goto_ptr() therefore takes the destination PC and the
TB being generated, and decides for itself whether to emit the probe or
the old helper call; there is no second entry point for targets that opt
in. A target that cannot name its destination in a single temp passes
NULL and gets the helper. Since the probe hashes and compares the PC as
one 64-bit value, a 32-bit guest PC also falls back.
The PC a target passes must be exactly what get_tb_cpu_state() reports for
the destination, which is the whole of the contract. alpha, loongarch,
mips, ppc and s390x pass their PC register, whose value is that pc by
construction. The rest pass NULL for now: avr's TB pc is the word address
doubled, i386's is eip before segmentation, riscv masks it to 32 bits when
xl is MXL_RV32, hppa derives it from the IAQ, hexagon adjusts it inside a
hardware loop, and sparc puts npc in cs_base so the guard could not hit
anyway. Each of those is a one-line change for whoever wants to measure it.
Two details matter for the generated code. The flags and cflags guards are
folded into a single aligned 64-bit load and compare, since the fields are
adjacent. And each path emits its own goto_ptr rather than branching to a
shared one: a temp live across the label is spilled and reloaded on every
dispatch, which cost 6.3% on its own.
The probe cannot check everything the helper checks, and the one that
matters is breakpoints. check_for_breakpoints() raises EXCP_DEBUG on an
exact pc match and selects CF_BP_PAGE cflags for the rest of the page, and
setting a breakpoint deliberately invalidates no TB, so a block translated
before the breakpoint was set is still sitting in the jump cache. Rather
than pay for a breakpoint test on the fast path, give the probe its own
base pointer, tb_jmp_cache_probe, that nothing else reads, and point it at
a page of zeroes while any breakpoint is set. Every entry the probe finds
then has a NULL tb, so every dispatch misses into the helper and the old
behaviour is restored exactly. cpu_breakpoint_insert() poisons the pointer,
so the poison takes effect at the next dispatch rather than whenever that
vCPU next reaches its main loop, which matters because a vCPU chaining
indirectly need never reach it. The main loop puts the pointer back once the
last breakpoint is gone; that is a load and a compare per block dispatched
from the main loop, and nothing at all in generated code.
The flags and cflags constants are safe against the other things that can
change them. CF_PARALLEL is only ever set by begin_parallel_context(),
which flushes first, so no block predating it survives to dispatch. gdb
single-step is only turned on with the CPU stopped, and a block translated
without CF_SINGLE_STEP can only be re-entered through tb_lookup(), which
from then on demands the new cflags -- so a stale-cflags block is never the
one running. What is left is one_insn_per_tb and -d nochain, which the
monitor can toggle under a running vCPU without a flush; see below.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling
the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, LTO
build, on top of the preceding three patches:
before: 1,402,667,803,616 instructions
after: 916,415,123,244 instructions -34.67%
before: 115.56s wall clock
after: 85.59s wall clock -25.94%
The gap between the two is the point at which this stops being a
straight-line win: the helper call was highly predictable work that the
host pipelined well, so removing it retires far fewer instructions than it
saves time. IPC falls from 2.48 to 2.17 across this patch for that reason.
Despite emitting more code, this also reduces instruction cache pressure,
because a dispatch no longer jumps into qemu's .text and evicts translated
code:
before: 11,735,141,703 L1-icache-load-misses
after: 7,154,863,292 L1-icache-load-misses -39.0%
The mechanism is visible directly in a profile: helper_lookup_tb_ptr()
falls from 31.01% of samples to 0.35%, and qemu's own .text falls from
38.8% to 5.3%, with the balance moving into generated code.
Combined with the three preceding patches, against an unmodified LTO
build, 1,646,994,254,249 instructions fall to 916,415,123,244, or -44.36%.
The emulated compiler produces byte-identical output throughout.
Open issues, hence RFC:
- one_insn_per_tb and CPU_LOG_TB_NOCHAIN can be toggled from the monitor
while a vCPU is inside a block that was translated without them. The
block keeps dispatching inline with the old cflags until it exits for
some other reason. Poisoning the probe from
tcg_update_all_curr_cflags() would close it.
- The jump cache entry is read without qatomic_read(); entries are
invalidated concurrently by setting tb to NULL.
- Only alpha has been measured. The other four targets that pass a PC are
built and boot-tested only.
v3: Fold the fast path into tcg_gen_lookup_and_goto_ptr() instead of
adding tcg_gen_lookup_and_goto_ptr_inline() beside it (Richard). It
now takes the destination PC and the TB unconditionally, from all 38
call sites, and picks the probe or the helper itself. Translators
built for both values of TARGET_LONG_BITS -- arm, s390x, microblaze --
cannot include tcg-op.h, so the common entry point takes a TCGTemp and
reads the width from it, and tcg-op.h wraps that for everyone else;
this is the same split as tcg_gen_qemu_ld_*_chk().
Compare cs_base too. v2 listed this as an open issue, and closing it
is what lets the choice be made generically rather than per target: a
target that uses cs_base would otherwise have been enabled silently by
a decision keyed on PC width alone. It costs a load and a compare on
the fast path, and the numbers above were measured with it in place.
Audited which targets may pass a real PC, the contract being that it
is exactly what get_tb_cpu_state() reports for the destination. Five
do; the rest pass NULL and keep the helper call, sparc among them
because it puts npc in cs_base and so could essentially never hit.
Poison the probe from cpu_breakpoint_insert() rather than only from
the poisoned CPU's own main loop. gdb inserts a breakpoint into every
CPU (tcg_insert_gdbstub_breakpoint()), and a thread already inside
generated code, dispatching indirectly, need never return to the main
loop -- so it would keep dispatching inline and run past a breakpoint
another thread had just set. Upstream has no such window: its
helper_lookup_tb_ptr() sees the new breakpoint at the next indirect
branch. The un-poison in tcg_cpu_sync_jmp_cache() now re-checks after
its store, with a barrier, so that it loses the race with a concurrent
insert in the safe direction.
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
accel/tcg/cpu-exec.c | 102 ++++++++++++++++++
accel/tcg/internal-common.h | 2 +
cpu-common.c | 11 ++
include/hw/core/cpu.h | 9 ++
include/system/tcg.h | 9 ++
include/tcg/tcg-op-common.h | 16 ++-
include/tcg/tcg-op.h | 12 +++
stubs/tcg-cflags.c | 8 +-
target/alpha/translate.c | 4 +-
target/arm/tcg/translate-a64.c | 4 +-
target/arm/tcg/translate.c | 10 +-
target/avr/translate.c | 4 +-
target/hexagon/translate.c | 4 +-
target/hppa/translate.c | 6 +-
target/i386/tcg/translate.c | 2 +-
.../tcg/insn_trans/trans_branch.c.inc | 2 +-
target/loongarch/tcg/translate.c | 4 +-
target/m68k/translate.c | 2 +-
target/microblaze/translate.c | 4 +-
target/mips/tcg/nanomips_translate.c.inc | 2 +-
target/mips/tcg/translate.c | 6 +-
target/or1k/translate.c | 4 +-
target/ppc/translate.c | 4 +-
target/riscv/tcg/insn_trans/trans_rvzce.c.inc | 4 +-
target/riscv/tcg/translate.c | 2 +-
target/rx/translate.c | 4 +-
target/s390x/tcg/translate.c | 5 +-
target/sh4/translate.c | 4 +-
target/sparc/translate.c | 4 +-
target/tricore/translate.c | 4 +-
tcg/tcg-op.c | 92 +++++++++++++++-
31 files changed, 300 insertions(+), 50 deletions(-)
diff --git ./accel/tcg/cpu-exec.c ./accel/tcg/cpu-exec.c
index 148e0f583e..e546f717e8 100644
--- ./accel/tcg/cpu-exec.c
+++ ./accel/tcg/cpu-exec.c
@@ -752,6 +752,99 @@ static inline bool cpu_handle_exception(CPUState *cpu, int *ret)
return false;
}
+/*
+ * The inline jump cache probe reads cpu->tb_jmp_cache_probe and takes the
+ * slow path when the entry it finds has a NULL tb. Pointing the probe at a
+ * region that is all zeroes therefore forces every indirect dispatch into
+ * helper_lookup_tb_ptr(), which does the full lookup the inline probe only
+ * approximates. The real jump cache is untouched, so no contents are lost
+ * and recovery is a single store.
+ *
+ * Only ever read from, and only the tb field of one entry per dispatch, so
+ * one shared zero-filled cache is enough for every CPU.
+ */
+static const CPUJumpCache *tb_jmp_cache_poison(void)
+{
+ static CPUJumpCache *poison;
+
+ if (unlikely(poison == NULL)) {
+ /* Raced allocations are harmless: both are all zeroes. */
+ qatomic_cmpxchg(&poison, NULL, g_new0(CPUJumpCache, 1));
+ }
+ return poison;
+}
+
+/*
+ * Whether the generated code may dispatch to the next block by itself.
+ *
+ * The inline probe matches on the destination pc and on the flags and
+ * cflags the dispatching block was translated with. It does not consult
+ * cpu->breakpoints, so it must not run while one is set: setting a
+ * breakpoint deliberately invalidates nothing, and check_for_breakpoints()
+ * both raises EXCP_DEBUG on an exact match and picks CF_BP_PAGE cflags for
+ * the rest of the page. A block translated before the breakpoint was set is
+ * therefore still in the jump cache, and dispatching to it inline would step
+ * straight over the breakpoint.
+ */
+static bool tcg_cpu_may_dispatch(CPUState *cpu)
+{
+ return QTAILQ_EMPTY(&cpu->breakpoints);
+}
+
+/*
+ * Poison @cpu's probe, from any thread. Called when a breakpoint is
+ * inserted, which is what makes the poison take effect at the dispatch
+ * after the insert rather than whenever @cpu next reaches its main loop:
+ * a vCPU chaining indirectly need never reach it, and would run past a
+ * breakpoint another thread had just set.
+ *
+ * A plain store is enough. The value only ever costs a slow path that is
+ * correct on its own, and the generated code re-reads the base on every
+ * dispatch. Un-poisoning is tcg_cpu_sync_jmp_cache()'s job.
+ */
+void tcg_cpu_poison_jmp_cache(CPUState *cpu)
+{
+ if (qatomic_read(&cpu->tb_jmp_cache_probe) != NULL) {
+ qatomic_set(&cpu->tb_jmp_cache_probe,
+ (CPUJumpCache *)tb_jmp_cache_poison());
+ }
+}
+
+/*
+ * Called from the main loop, which is the only context that can establish
+ * that no reason to be poisoned is left. Cheap enough to call every time
+ * round: the common case is a load, a compare and no store at all.
+ */
+void tcg_cpu_sync_jmp_cache(CPUState *cpu)
+{
+ CPUJumpCache *want;
+
+ if (qatomic_read(&cpu->tb_jmp_cache_probe) == NULL) {
+ return; /* not realized, or already unrealized */
+ }
+
+ want = tcg_cpu_may_dispatch(cpu)
+ ? cpu->tb_jmp_cache
+ : (CPUJumpCache *)tb_jmp_cache_poison();
+
+ if (qatomic_read(&cpu->tb_jmp_cache_probe) != want) {
+ qatomic_set(&cpu->tb_jmp_cache_probe, want);
+
+ /*
+ * Un-poisoning races a concurrent tcg_cpu_poison_jmp_cache(): the
+ * reason may have appeared after tcg_cpu_may_dispatch() read it and
+ * the poison may have landed before the store above. Order the
+ * store against a re-read, and lose the race in the safe direction.
+ */
+ if (want == cpu->tb_jmp_cache) {
+ smp_mb();
+ if (!tcg_cpu_may_dispatch(cpu)) {
+ tcg_cpu_poison_jmp_cache(cpu);
+ }
+ }
+ }
+}
+
void tcg_kick_vcpu_thread(CPUState *cpu)
{
/*
@@ -964,6 +1057,13 @@ cpu_exec_loop(CPUState *cpu, SyncClocks *sc)
break;
}
+ /*
+ * Reaching here means the main loop has just re-evaluated
+ * everything the inline probe assumes, so this is where the
+ * probe is allowed to come back after a poison.
+ */
+ tcg_cpu_sync_jmp_cache(cpu);
+
tb = tb_lookup(cpu, s);
if (tb == NULL) {
CPUJumpCache *jc;
@@ -1072,6 +1172,7 @@ bool tcg_exec_realizefn(CPUState *cpu, Error **errp)
tcg_update_cflags(cpu);
cpu->tb_jmp_cache = g_new0(CPUJumpCache, 1);
+ qatomic_set(&cpu->tb_jmp_cache_probe, cpu->tb_jmp_cache);
tlb_init(cpu);
#ifndef CONFIG_USER_ONLY
tcg_iommu_init_notifier_list(cpu);
@@ -1089,5 +1190,6 @@ void tcg_exec_unrealizefn(CPUState *cpu)
#endif /* !CONFIG_USER_ONLY */
tlb_destroy(cpu);
+ qatomic_set(&cpu->tb_jmp_cache_probe, NULL);
g_free_rcu(cpu->tb_jmp_cache, rcu);
}
diff --git ./accel/tcg/internal-common.h ./accel/tcg/internal-common.h
index 853d1b51ee..9d1f6712d6 100644
--- ./accel/tcg/internal-common.h
+++ ./accel/tcg/internal-common.h
@@ -144,6 +144,8 @@ void page_table_config_init(void);
G_NORETURN void cpu_io_recompile(CPUState *cpu, uintptr_t retaddr);
#endif /* CONFIG_USER_ONLY */
+void tcg_cpu_sync_jmp_cache(CPUState *cpu);
+
void tb_phys_invalidate(TranslationBlock *tb, tb_page_addr_t page_addr);
void tb_set_jmp_target(TranslationBlock *tb, int n, uintptr_t addr);
diff --git ./cpu-common.c ./cpu-common.c
index adb76b3a78..3aed0156e6 100644
--- ./cpu-common.c
+++ ./cpu-common.c
@@ -22,6 +22,7 @@
#include "exec/cpu-common.h"
#include "hw/core/cpu.h"
#include "qemu/lockable.h"
+#include "system/tcg.h"
#include "trace/trace-root.h"
QemuMutex qemu_cpu_list_lock;
@@ -429,6 +430,16 @@ int cpu_breakpoint_insert(CPUState *cpu, vaddr pc, int flags,
*breakpoint = bp;
}
+ /*
+ * Nothing is invalidated here, so blocks translated before this point
+ * are still live and still dispatch to each other without consulting
+ * cpu->breakpoints. Stop the ones that can: a TCG vCPU dispatching
+ * inline reads a base pointer that this poisons, so the next dispatch
+ * takes the slow path and sees the new breakpoint. @cpu may be another
+ * thread, and may be running.
+ */
+ tcg_cpu_poison_jmp_cache(cpu);
+
trace_breakpoint_insert(cpu->cpu_index, pc, flags);
return 0;
}
diff --git ./include/hw/core/cpu.h ./include/hw/core/cpu.h
index 81af7b9ee1..bd2cdd2a0b 100644
--- ./include/hw/core/cpu.h
+++ ./include/hw/core/cpu.h
@@ -519,6 +519,15 @@ struct CPUState {
MemoryRegion *memory;
struct CPUJumpCache *tb_jmp_cache;
+ /*
+ * @tb_jmp_cache_probe: base the inline jump cache probe reads.
+ *
+ * Normally @tb_jmp_cache. Pointed at a shared page of zeroes to force
+ * every inline dispatch to miss and fall back to helper_lookup_tb_ptr();
+ * see tcg_cpu_sync_jmp_cache(). NULL before tcg_exec_realizefn() and
+ * after tcg_exec_unrealizefn().
+ */
+ struct CPUJumpCache *tb_jmp_cache_probe;
GArray *gdb_regs;
int gdb_num_regs;
diff --git ./include/system/tcg.h ./include/system/tcg.h
index 2c2dbc753b..bf05db1329 100644
--- ./include/system/tcg.h
+++ ./include/system/tcg.h
@@ -29,6 +29,15 @@ extern bool tcg_allowed;
void tcg_update_cflags(CPUState *cpu);
void tcg_update_all_cflags(void);
+/*
+ * Force @cpu's generated code back into the slow dispatch path, which
+ * re-checks everything the inline jump cache probe assumes. Safe to call
+ * from any thread, and a no-op for a CPU that is not running TCG. Call
+ * whenever something the probe cannot see changes under a running vCPU;
+ * the main loop undoes it once the reason is gone.
+ */
+void tcg_cpu_poison_jmp_cache(CPUState *cpu);
+
/**
* qemu_tcg_mttcg_enabled:
* Check whether we are running MultiThread TCG or not.
diff --git ./include/tcg/tcg-op-common.h ./include/tcg/tcg-op-common.h
index 9b321f959c..ba580c7fb7 100644
--- ./include/tcg/tcg-op-common.h
+++ ./include/tcg/tcg-op-common.h
@@ -75,15 +75,25 @@ void tcg_gen_exit_tb(const TranslationBlock *tb, unsigned idx);
void tcg_gen_goto_tb(unsigned idx);
/**
- * tcg_gen_lookup_and_goto_ptr() - look up the current TB, jump to it if valid
- * @addr: Guest address of the target TB
+ * tcg_gen_lookup_and_goto_ptr() - look up the destination TB, jump to it
+ * @pc: temp holding the destination guest PC, or NULL
+ * @tb: the translation block being generated
*
* If the TB is not valid, jump to the epilogue.
*
+ * The lookup is normally a call to helper_lookup_tb_ptr(). If @pc is
+ * non-NULL and the destination can be keyed on it directly, the jump cache
+ * is probed inline instead and only a miss reaches the helper. @pc must
+ * then hold exactly the value get_tb_cpu_state() reports as the pc for the
+ * destination; a target whose pc is derived (avr's word address, i386's
+ * eip before segmentation) must pass NULL. The destination is required to
+ * match @tb's flags, cflags and cs_base, which is what makes them
+ * constants in the probe.
+ *
* This operation is optional. If the TCG backend does not implement goto_ptr,
* this op is equivalent to calling tcg_gen_exit_tb() with 0 as the argument.
*/
-void tcg_gen_lookup_and_goto_ptr(void);
+void tcg_gen_lookup_and_goto_ptr_tmp(TCGTemp *pc, const TranslationBlock *tb);
void tcg_gen_plugin_cb(unsigned from);
void tcg_gen_plugin_mem_cb(TCGv_i64 addr, unsigned meminfo);
diff --git ./include/tcg/tcg-op.h ./include/tcg/tcg-op.h
index 3721164236..b6c7c6fea2 100644
--- ./include/tcg/tcg-op.h
+++ ./include/tcg/tcg-op.h
@@ -49,6 +49,18 @@ typedef TCGv_i64 TCGv;
#error Unhandled TARGET_LONG_BITS value
#endif
+/*
+ * See tcg_gen_lookup_and_goto_ptr_tmp(). @pc may be NULL, for a target
+ * whose guest PC is not directly the key the jump cache is indexed by.
+ * A translator that is built for more than one value of TARGET_LONG_BITS,
+ * and so cannot include this header, calls the _tmp() form directly.
+ */
+static inline void
+tcg_gen_lookup_and_goto_ptr(TCGv pc, const TranslationBlock *tb)
+{
+ tcg_gen_lookup_and_goto_ptr_tmp(pc ? tcgv_tl_temp(pc) : NULL, tb);
+}
+
#if TARGET_LONG_BITS == 64
#define tcg_gen_movi_tl tcg_gen_movi_i64
#define tcg_gen_mov_tl tcg_gen_mov_i64
diff --git ./stubs/tcg-cflags.c ./stubs/tcg-cflags.c
index cb278e94aa..4ac8a82e92 100644
--- ./stubs/tcg-cflags.c
+++ ./stubs/tcg-cflags.c
@@ -1,6 +1,6 @@
/*
- * Stub for tcg_update_all_cflags(), for binaries that link util/log.c
- * or cpu-target.c but not TCG.
+ * Stubs for the TCG entry points in system/tcg.h, for binaries that link
+ * util/log.c, cpu-target.c or cpu-common.c but not TCG.
*
* SPDX-License-Identifier: GPL-2.0-or-later
*/
@@ -14,3 +14,7 @@ void tcg_update_cflags(CPUState *cpu)
void tcg_update_all_cflags(void)
{
}
+
+void tcg_cpu_poison_jmp_cache(CPUState *cpu)
+{
+}
diff --git ./target/alpha/translate.c ./target/alpha/translate.c
index c66e3f9c14..822f5cc120 100644
--- ./target/alpha/translate.c
+++ ./target/alpha/translate.c
@@ -449,7 +449,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_idx, int32_t disp)
tcg_gen_exit_tb(ctx->base.tb, tb_slot_idx);
} else {
gen_pc_disp(ctx, cpu_pc, disp);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_pc, ctx->base.tb);
}
}
@@ -2917,7 +2917,7 @@ static void alpha_tr_tb_stop(DisasContextBase *dcbase, CPUState *cpu)
gen_pc_disp(ctx, cpu_pc, 0);
/* FALLTHRU */
case DISAS_PC_UPDATED:
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_pc, ctx->base.tb);
break;
case DISAS_PC_UPDATED_NOCHAIN:
tcg_gen_exit_tb(NULL, 0);
diff --git ./target/arm/tcg/translate-a64.c ./target/arm/tcg/translate-a64.c
index 4f9a93950b..d1dd33a1af 100644
--- ./target/arm/tcg/translate-a64.c
+++ ./target/arm/tcg/translate-a64.c
@@ -562,7 +562,7 @@ static void gen_goto_tb(DisasContext *s, unsigned tb_slot_idx, int64_t diff)
if (s->ss_active) {
gen_step_complete_exception(s);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, s->base.tb);
s->base.is_jmp = DISAS_NORETURN;
}
}
@@ -11250,7 +11250,7 @@ static void aarch64_tr_tb_stop(DisasContextBase *dcbase, CPUState *cpu)
gen_a64_update_pc(dc, 4);
/* fall through */
case DISAS_JUMP:
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, dc->base.tb);
break;
case DISAS_NORETURN:
case DISAS_SWI:
diff --git ./target/arm/tcg/translate.c ./target/arm/tcg/translate.c
index c866148383..ca701b9cbc 100644
--- ./target/arm/tcg/translate.c
+++ ./target/arm/tcg/translate.c
@@ -1306,9 +1306,9 @@ void write_neon_element64(TCGv_i64 src, int reg, int ele, MemOp memop)
}
}
-static void gen_goto_ptr(void)
+static void gen_goto_ptr(DisasContext *s)
{
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr_tmp(NULL, s->base.tb);
}
/* This will end the TB but doesn't guarantee we'll return to
@@ -1336,7 +1336,7 @@ static void gen_goto_tb(DisasContext *s, unsigned tb_slot_idx, int64_t diff)
tcg_gen_exit_tb(s->base.tb, tb_slot_idx);
} else {
gen_update_pc(s, diff);
- gen_goto_ptr();
+ gen_goto_ptr(s);
}
s->base.is_jmp = DISAS_NORETURN;
}
@@ -1373,7 +1373,7 @@ static void gen_jmp_tb(DisasContext *s, int64_t diff, int tbno)
* and don't chain to another TB.
*/
gen_update_pc(s, diff);
- gen_goto_ptr();
+ gen_goto_ptr(s);
s->base.is_jmp = DISAS_NORETURN;
break;
default:
@@ -6858,7 +6858,7 @@ static void arm_tr_tb_stop(DisasContextBase *dcbase, CPUState *cpu)
gen_update_pc(dc, curr_insn_len(dc));
/* fall through */
case DISAS_JUMP:
- gen_goto_ptr();
+ gen_goto_ptr(dc);
break;
case DISAS_UPDATE_EXIT:
gen_update_pc(dc, curr_insn_len(dc));
diff --git ./target/avr/translate.c ./target/avr/translate.c
index 3c57606097..8f2e0baa67 100644
--- ./target/avr/translate.c
+++ ./target/avr/translate.c
@@ -992,7 +992,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_idx,
tcg_gen_exit_tb(tb, tb_slot_idx);
} else {
tcg_gen_movi_i32(cpu_pc, dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
ctx->base.is_jmp = DISAS_NORETURN;
}
@@ -2778,7 +2778,7 @@ static void avr_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
/* fall through */
case DISAS_LOOKUP:
if (!force_exit) {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
break;
}
/* fall through */
diff --git ./target/hexagon/translate.c ./target/hexagon/translate.c
index 06a8159d28..cc230b08d1 100644
--- ./target/hexagon/translate.c
+++ ./target/hexagon/translate.c
@@ -181,7 +181,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_idx,
if (move_to_pc) {
tcg_gen_movi_tl(hex_gpr[HEX_REG_PC], dest);
}
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
}
@@ -218,7 +218,7 @@ static void gen_end_tb(DisasContext *ctx)
gen_set_label(skip);
gen_goto_tb(ctx, 1, ctx->next_PC, false);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
ctx->base.is_jmp = DISAS_NORETURN;
diff --git ./target/hppa/translate.c ./target/hppa/translate.c
index 002189ddfb..cf8f1a2c13 100644
--- ./target/hppa/translate.c
+++ ./target/hppa/translate.c
@@ -816,7 +816,7 @@ static void gen_goto_tb(DisasContext *ctx, int which,
tcg_gen_goto_tb(which);
tcg_gen_exit_tb(ctx->base.tb, which);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
}
@@ -2027,7 +2027,7 @@ static bool do_ibranch(DisasContext *ctx, unsigned link,
store_psw_xb(ctx, PSW_B);
}
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
ctx->base.is_jmp = DISAS_NORETURN;
return nullify_end(ctx);
}
@@ -4838,7 +4838,7 @@ static void hppa_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
}
/* FALLTHRU */
case DISAS_IAQ_N_UPDATED:
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
break;
case DISAS_EXIT:
tcg_gen_exit_tb(NULL, 0);
diff --git ./target/i386/tcg/translate.c ./target/i386/tcg/translate.c
index 2115c5cd24..66a0ee3cdf 100644
--- ./target/i386/tcg/translate.c
+++ ./target/i386/tcg/translate.c
@@ -2005,7 +2005,7 @@ gen_eob(DisasContext *s, int mode)
} else if (mode == DISAS_JUMP &&
/* give irqs a chance to happen */
!inhibit_reset) {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, s->base.tb);
} else {
tcg_gen_exit_tb(NULL, 0);
}
diff --git ./target/loongarch/tcg/insn_trans/trans_branch.c.inc ./target/loongarch/tcg/insn_trans/trans_branch.c.inc
index da07778658..57d9d47353 100644
--- ./target/loongarch/tcg/insn_trans/trans_branch.c.inc
+++ ./target/loongarch/tcg/insn_trans/trans_branch.c.inc
@@ -27,7 +27,7 @@ static bool trans_jirl(DisasContext *ctx, arg_jirl *a)
tcg_gen_mov_tl(cpu_pc, addr);
tcg_gen_movi_tl(dest, make_address_pc(ctx, ctx->base.pc_next + 4));
gen_set_gpr(a->rd, dest, EXT_NONE);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_pc, ctx->base.tb);
ctx->base.is_jmp = DISAS_NORETURN;
return true;
}
diff --git ./target/loongarch/tcg/translate.c ./target/loongarch/tcg/translate.c
index 124dce6269..a45a51852a 100644
--- ./target/loongarch/tcg/translate.c
+++ ./target/loongarch/tcg/translate.c
@@ -111,7 +111,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_idx, vaddr dest)
tcg_gen_exit_tb(ctx->base.tb, tb_slot_idx);
} else {
tcg_gen_movi_tl(cpu_pc, dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_pc, ctx->base.tb);
}
}
@@ -311,7 +311,7 @@ static void loongarch_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
switch (ctx->base.is_jmp) {
case DISAS_STOP:
tcg_gen_movi_tl(cpu_pc, ctx->base.pc_next);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_pc, ctx->base.tb);
break;
case DISAS_TOO_MANY:
gen_goto_tb(ctx, 0, ctx->base.pc_next);
diff --git ./target/m68k/translate.c ./target/m68k/translate.c
index 138c89d3e5..73691bc0d1 100644
--- ./target/m68k/translate.c
+++ ./target/m68k/translate.c
@@ -6095,7 +6095,7 @@ static void m68k_tr_tb_stop(DisasContextBase *dcbase, CPUState *cpu)
if (dc->ss_active) {
gen_raise_exception_format2(dc, EXCP_TRACE, dc->pc_prev);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, dc->base.tb);
}
break;
case DISAS_EXIT:
diff --git ./target/microblaze/translate.c ./target/microblaze/translate.c
index 8b219afb5d..851b372f8f 100644
--- ./target/microblaze/translate.c
+++ ./target/microblaze/translate.c
@@ -127,7 +127,7 @@ static void gen_goto_tb(DisasContext *dc, unsigned tb_slot_idx, vaddr dest)
tcg_gen_exit_tb(dc->base.tb, tb_slot_idx);
} else {
tcg_gen_movi_i32(cpu_pc, dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr_tmp(NULL, dc->base.tb);
}
dc->base.is_jmp = DISAS_NORETURN;
}
@@ -1764,7 +1764,7 @@ static void mb_tr_tb_stop(DisasContextBase *dcb, CPUState *cs)
/* Indirect jump (or direct jump w/ goto_tb disabled) */
tcg_gen_mov_i32(cpu_pc, cpu_btarget);
tcg_gen_discard_i32(cpu_btarget);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr_tmp(NULL, dc->base.tb);
return;
default:
diff --git ./target/mips/tcg/nanomips_translate.c.inc ./target/mips/tcg/nanomips_translate.c.inc
index 4b0b01ba37..007e29f9ac 100644
--- ./target/mips/tcg/nanomips_translate.c.inc
+++ ./target/mips/tcg/nanomips_translate.c.inc
@@ -2406,7 +2406,7 @@ static void gen_compute_nanomips_pbalrsc_branch(DisasContext *ctx, int rs,
/* unconditional branch to register */
tcg_gen_mov_tl(cpu_PC, btarget);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_PC, ctx->base.tb);
}
/* nanoMIPS Branches */
diff --git ./target/mips/tcg/translate.c ./target/mips/tcg/translate.c
index e3467d1525..73abfbb5d4 100644
--- ./target/mips/tcg/translate.c
+++ ./target/mips/tcg/translate.c
@@ -4374,7 +4374,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_idx,
tcg_gen_exit_tb(ctx->base.tb, tb_slot_idx);
} else {
gen_save_pc(dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_PC, ctx->base.tb);
}
}
@@ -11014,7 +11014,7 @@ static void gen_branch(DisasContext *ctx, int insn_bytes)
} else {
tcg_gen_mov_tl(cpu_PC, btarget);
}
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_PC, ctx->base.tb);
break;
default:
LOG_DISAS("unknown branch 0x%x\n", proc_hflags);
@@ -15244,7 +15244,7 @@ static void mips_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
switch (ctx->base.is_jmp) {
case DISAS_STOP:
gen_save_pc(ctx->base.pc_next);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_PC, ctx->base.tb);
break;
case DISAS_NEXT:
case DISAS_TOO_MANY:
diff --git ./target/or1k/translate.c ./target/or1k/translate.c
index eb4485312f..4907284a6d 100644
--- ./target/or1k/translate.c
+++ ./target/or1k/translate.c
@@ -1605,7 +1605,7 @@ static void openrisc_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
/* The jump destination is indirect/computed; use jmp_pc. */
tcg_gen_mov_i32(cpu_pc, jmp_pc);
tcg_gen_discard_i32(jmp_pc);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, dc->base.tb);
break;
}
/* The jump destination is direct; use jmp_pc_imm.
@@ -1622,7 +1622,7 @@ static void openrisc_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
break;
}
tcg_gen_movi_i32(cpu_pc, jmp_dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, dc->base.tb);
break;
case DISAS_EXIT:
diff --git ./target/ppc/translate.c ./target/ppc/translate.c
index 06ed2adf10..42924281b0 100644
--- ./target/ppc/translate.c
+++ ./target/ppc/translate.c
@@ -3664,7 +3664,7 @@ static void gen_lookup_and_goto_ptr(DisasContext *ctx)
pmu_count_insns(ctx);
}
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_nip, ctx->base.tb);
}
}
@@ -6690,7 +6690,7 @@ static void ppc_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
pmu_count_insns(ctx);
}
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_nip, ctx->base.tb);
break;
case DISAS_EXIT_UPDATE:
diff --git ./target/riscv/tcg/insn_trans/trans_rvzce.c.inc ./target/riscv/tcg/insn_trans/trans_rvzce.c.inc
index 71b4ca5473..3f1e7c039e 100644
--- ./target/riscv/tcg/insn_trans/trans_rvzce.c.inc
+++ ./target/riscv/tcg/insn_trans/trans_rvzce.c.inc
@@ -213,7 +213,7 @@ static bool gen_pop(DisasContext *ctx, arg_cmpp *a, bool ret, bool ret_val)
}
#endif
tcg_gen_mov_tl(cpu_pc, ret_addr);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
ctx->base.is_jmp = DISAS_NORETURN;
}
@@ -334,7 +334,7 @@ static bool trans_cm_jalt(DisasContext *ctx, arg_cm_jalt *a)
tcg_gen_mov_tl(cpu_pc, addr);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
ctx->base.is_jmp = DISAS_NORETURN;
return true;
}
diff --git ./target/riscv/tcg/translate.c ./target/riscv/tcg/translate.c
index 9684dbe752..8475ab43b4 100644
--- ./target/riscv/tcg/translate.c
+++ ./target/riscv/tcg/translate.c
@@ -287,7 +287,7 @@ static void lookup_and_goto_ptr(DisasContext *ctx)
gen_helper_itrigger_match(tcg_env);
}
#endif
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
static void exit_tb(DisasContext *ctx)
diff --git ./target/rx/translate.c ./target/rx/translate.c
index 132d495710..e5a9783d84 100644
--- ./target/rx/translate.c
+++ ./target/rx/translate.c
@@ -161,7 +161,7 @@ static void gen_goto_tb(DisasContext *dc, unsigned tb_slot_idx, vaddr dest)
tcg_gen_exit_tb(dc->base.tb, tb_slot_idx);
} else {
tcg_gen_movi_i32(cpu_pc, dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, dc->base.tb);
}
dc->base.is_jmp = DISAS_NORETURN;
}
@@ -2242,7 +2242,7 @@ static void rx_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
gen_goto_tb(ctx, 0, dcbase->pc_next);
break;
case DISAS_JUMP:
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
break;
case DISAS_UPDATE:
tcg_gen_movi_i32(cpu_pc, ctx->base.pc_next);
diff --git ./target/s390x/tcg/translate.c ./target/s390x/tcg/translate.c
index 1b6023168b..607c039419 100644
--- ./target/s390x/tcg/translate.c
+++ ./target/s390x/tcg/translate.c
@@ -1162,7 +1162,7 @@ static DisasJumpType help_branch(DisasContext *s, DisasCompare *c,
tcg_gen_goto_tb(0);
tcg_gen_exit_tb(s->base.tb, 0);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr_tmp(tcgv_i64_temp(psw_addr), s->base.tb);
}
gen_set_label(lab);
@@ -6477,7 +6477,8 @@ static void s390x_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
if (dc->exit_to_mainloop) {
tcg_gen_exit_tb(NULL, 0);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr_tmp(tcgv_i64_temp(psw_addr),
+ dc->base.tb);
}
break;
default:
diff --git ./target/sh4/translate.c ./target/sh4/translate.c
index 373950fd66..a4be456bd9 100644
--- ./target/sh4/translate.c
+++ ./target/sh4/translate.c
@@ -242,7 +242,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_idx, vaddr dest)
if (use_exit_tb(ctx)) {
tcg_gen_exit_tb(NULL, 0);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
}
ctx->base.is_jmp = DISAS_NORETURN;
@@ -258,7 +258,7 @@ static void gen_jump(DisasContext * ctx)
if (use_exit_tb(ctx)) {
tcg_gen_exit_tb(NULL, 0);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
ctx->base.is_jmp = DISAS_NORETURN;
} else {
diff --git ./target/sparc/translate.c ./target/sparc/translate.c
index 3156be6a94..2ae0a02c44 100644
--- ./target/sparc/translate.c
+++ ./target/sparc/translate.c
@@ -376,7 +376,7 @@ static void gen_goto_tb(DisasContext *s, unsigned tb_slot_idx,
/* jump to another page: we can use an indirect jump */
tcg_gen_movi_tl(cpu_pc, pc);
tcg_gen_movi_tl(cpu_npc, npc);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, s->base.tb);
}
}
@@ -5807,7 +5807,7 @@ static void sparc_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
tcg_gen_movi_tl(cpu_npc, dc->npc);
}
if (may_lookup) {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, dc->base.tb);
} else {
tcg_gen_exit_tb(NULL, 0);
}
diff --git ./target/tricore/translate.c ./target/tricore/translate.c
index 8cd6b58f66..1d7f54f6df 100644
--- ./target/tricore/translate.c
+++ ./target/tricore/translate.c
@@ -2857,7 +2857,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_index, vaddr dest)
tcg_gen_exit_tb(ctx->base.tb, tb_slot_index);
} else {
gen_save_pc(dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
ctx->base.is_jmp = DISAS_NORETURN;
}
@@ -8478,7 +8478,7 @@ static void tricore_tr_tb_stop(DisasContextBase *dcbase, CPUState *cpu)
tcg_gen_exit_tb(NULL, 0);
break;
case DISAS_JUMP:
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
break;
case DISAS_NORETURN:
break;
diff --git ./tcg/tcg-op.c ./tcg/tcg-op.c
index 28d3b2a847..ce77541eab 100644
--- ./tcg/tcg-op.c
+++ ./tcg/tcg-op.c
@@ -28,6 +28,8 @@
#include "tcg/tcg-op-common.h"
#include "exec/translation-block.h"
#include "exec/plugin-gen.h"
+#include "hw/core/cpu.h"
+#include "../accel/tcg/tb-jmp-cache.h"
#include "tcg-internal.h"
#include "tcg-has.h"
@@ -2715,7 +2717,81 @@ void tcg_gen_goto_tb(unsigned idx)
tcg_gen_op1i(INDEX_op_goto_tb, 0, idx);
}
-void tcg_gen_lookup_and_goto_ptr(void)
+static void gen_jmp_cache_probe(TCGv_i64 pc, const TranslationBlock *tb)
+{
+ TCGv_ptr jc, ent, tbp, ptr;
+ TCGv_i64 h, tmp;
+ TCGLabel *slow;
+ uint64_t fpair;
+
+ QEMU_BUILD_BUG_ON(sizeof(((CPUJumpCache *)0)->array[0]) != 16);
+ QEMU_BUILD_BUG_ON(offsetof(TranslationBlock, cflags) !=
+ offsetof(TranslationBlock, flags) + 4);
+
+ jc = tcg_temp_ebb_new_ptr();
+ ent = tcg_temp_ebb_new_ptr();
+ tbp = tcg_temp_ebb_new_ptr();
+ ptr = tcg_temp_ebb_new_ptr();
+ h = tcg_temp_ebb_new_i64();
+ tmp = tcg_temp_ebb_new_i64();
+ slow = gen_new_label();
+
+ /* h = tb_jmp_cache_hash_func(pc) * sizeof(array[0]) */
+ tcg_gen_shri_i64(h, pc, TB_JMP_CACHE_BITS);
+ tcg_gen_xor_i64(h, h, pc);
+ tcg_gen_andi_i64(h, h, TB_JMP_CACHE_SIZE - 1);
+ tcg_gen_shli_i64(h, h, 4);
+
+ /*
+ * Not cpu->tb_jmp_cache: the probe reads its own base so that the main
+ * loop can poison it, which is how conditions the probe cannot test for
+ * itself force every dispatch back into the helper. See
+ * tcg_cpu_sync_jmp_cache().
+ */
+ tcg_gen_ld_ptr(jc, tcg_env,
+ offsetof(CPUState, tb_jmp_cache_probe) - sizeof(CPUState));
+ tcg_gen_trunc_i64_ptr(ent, h);
+ tcg_gen_add_ptr(ent, jc, ent);
+
+ tcg_gen_ld_ptr(tbp, ent, offsetof(CPUJumpCache, array[0].tb));
+ tcg_gen_brcondi_ptr(TCG_COND_EQ, tbp, 0, slow);
+
+ tcg_gen_ld_i64(tmp, ent, offsetof(CPUJumpCache, array[0].pc));
+ tcg_gen_brcond_i64(TCG_COND_NE, tmp, pc, slow);
+
+ /*
+ * flags and cflags are adjacent uint32_t, so one aligned 64-bit load
+ * and compare covers both.
+ */
+#if HOST_BIG_ENDIAN
+ fpair = ((uint64_t)tb->flags << 32) | tb->cflags;
+#else
+ fpair = ((uint64_t)tb->cflags << 32) | tb->flags;
+#endif
+ tcg_gen_ld_i64(tmp, tbp, offsetof(TranslationBlock, flags));
+ tcg_gen_brcondi_i64(TCG_COND_NE, tmp, fpair, slow);
+
+ /*
+ * The destination must have been translated with the same cs_base, which
+ * the pc alone does not imply on a target that uses it.
+ */
+ tcg_gen_ld_i64(tmp, tbp, offsetof(TranslationBlock, cs_base));
+ tcg_gen_brcondi_i64(TCG_COND_NE, tmp, tb->cs_base, slow);
+
+ tcg_gen_ld_ptr(ptr, tbp, offsetof(TranslationBlock, tc.ptr));
+ tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr));
+
+ /*
+ * Emit a second goto_ptr rather than branching to a shared one: a temp
+ * live across the label would be spilled and reloaded on every dispatch.
+ */
+ gen_set_label(slow);
+ ptr = tcg_temp_ebb_new_ptr();
+ gen_helper_lookup_tb_ptr(ptr, tcg_env);
+ tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr));
+}
+
+void tcg_gen_lookup_and_goto_ptr_tmp(TCGTemp *pc, const TranslationBlock *tb)
{
TCGv_ptr ptr;
@@ -2724,7 +2800,21 @@ void tcg_gen_lookup_and_goto_ptr(void)
return;
}
+ /*
+ * No icount_decr poll is needed for this exit: the helper is called on
+ * every dispatch and returns to the main loop while an exit is pending.
+ */
plugin_gen_disable_mem_helpers();
+
+ /*
+ * The inline probe hashes and compares the pc as a single 64-bit value.
+ * A target with a 32-bit guest PC keeps the helper call.
+ */
+ if (pc && pc->type == TCG_TYPE_I64) {
+ gen_jmp_cache_probe(temp_tcgv_i64(pc), tb);
+ return;
+ }
+
ptr = tcg_temp_ebb_new_ptr();
gen_helper_lookup_tb_ptr(ptr, tcg_env);
tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr));
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH v3 5/7] RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds
2026-08-22 19:08 [PATCH v3 0/7] accel/tcg: cut per-block dispatch overhead Matt Turner
` (3 preceding siblings ...)
2026-08-22 19:08 ` [PATCH v3 4/7] RFC: tcg: probe the TB jump cache inline instead of calling a helper Matt Turner
@ 2026-08-22 19:08 ` Matt Turner
2026-08-26 7:51 ` Alex Bennée
2026-08-22 19:08 ` [PATCH v3 6/7] RFC: accel/tcg: poison the jump cache instead of polling for indirect exits Matt Turner
` (11 subsequent siblings)
16 siblings, 1 reply; 47+ messages in thread
From: Matt Turner @ 2026-08-22 19:08 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
translator_use_goto_tb() refuses to chain unless the destination is on the
same page as the start of the TB. For guests whose text is much larger than
a page this is expensive: an emulated alpha gcc compiling a 255k line
translation unit takes the indirect dispatch path for 8.4 billion of its
34.2 billion TB exits, and a large share of those are ordinary direct
branches that simply crossed an 8 KiB page boundary.
The restriction was made unconditional by d3a2a1d803 ("accel/tcg:
Introduce translator_use_goto_tb"), whose rationale was:
Various targets avoid the page crossing test for CONFIG_USER_ONLY,
but that is wrong: mmap and mprotect can change page permissions.
That is true, but in user-only builds the invalidation path already covers
it. There are no page tables: every mmap, mprotect and munmap reaches
page_set_flags(), which calls tb_invalidate_phys_range() whenever the flags
actually change, and tb_phys_invalidate() calls tb_jmp_unlink() to reset
incoming jumps. A chained cross-page jump is therefore broken whenever the
destination page's permissions change. This is not true in system mode,
where TBs are keyed by physical address and a page table change invalidates
nothing, so the restriction is kept there.
The rule protects one more thing, which the original rationale does not
mention: it guarantees that execution cannot enter a page without a TB
lookup, and so without check_for_breakpoints(). That is what makes a
breakpoint set after a block was translated take effect, since insertion
deliberately invalidates nothing. A link established before the breakpoint
was set would jump straight over it.
So the chaining is only enabled for a run that can never acquire a
breakpoint. In user-only mode every breakpoint comes from gdb -- BP_CPU is
g_assert_not_reached() there, and the guest cannot ask for one -- and gdb
has to be requested with -g before the first block is translated, even
though with suspend=n it may connect later. gdb_may_set_breakpoints()
reports whether it was, and is fixed for the lifetime of the process.
Add tests/tcg/alpha/test-xpage-chain.c to cover both hazards directly. It
places a direct branch near the end of one page targeting the next page,
runs it 200000 times so the chain is established, then checks that
mprotect(PROT_NONE) makes the next call fault, and that remapping the page
with different code runs the new code rather than a stale translation.
The test detects the hazard it is meant to detect: with the
tb_invalidate_phys_range() call in page_set_flags() commented out, it fails
both phases, executing page B after PROT_NONE and returning the stale
result.
Run with -b, the same binary stops once the chain is established and lets
tests/tcg/alpha/gdbstub/xpage-bp.py set a breakpoint on the far side of it,
which the next call has to stop on. With gdb_may_set_breakpoints() forced to
false so that the chaining stays on under gdb, that breakpoint is missed and
the test fails, which is what makes it a test of the gate rather than of
gdb.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the
SQLite 3.45.1 amalgamation on an x86-64 host, LTO build, on top of the
preceding patches:
before: 916,415,123,244 instructions
after: 891,254,240,071 instructions -2.75%
before: 85.59s wall clock
after: 81.45s wall clock -4.84%
Note that this is worth more in time than in instructions, the reverse of
the preceding patch: a chained jump replaces a cache probe whose loads can
miss, so the instructions it removes are more expensive than average.
Measured before the inline jump cache probe, when a missed chain cost a
helper call rather than an inline probe, the same change was worth -7.9%.
RFC because this reverses a deliberate decision and the reasoning above
wants review from someone who knows the invalidation paths better than I
do.
v3: Only take the shortcut when no gdbstub was requested. The same-page
rule also forces a lookup, and so a breakpoint check, on entry to every
page; without that, a chain established before a breakpoint was set runs
past it. Reported by Richard Henderson.
v3: Change translator_use_goto_tb() rather than translator_is_same_page().
i386, riscv and s390x call translator_is_same_page() for something else
-- enforcing that only a single-insn TB may cross a page -- and v2
changed their TB boundaries in user-only mode as a side effect. alpha
does not call it, so the numbers above are unaffected.
v3: Add the gdbstub half of the test.
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
accel/tcg/translator.c | 33 ++++++-
gdbstub/user.c | 14 +++
include/gdbstub/user.h | 11 +++
tests/tcg/alpha/Makefile.target | 17 +++-
tests/tcg/alpha/gdbstub/xpage-bp.py | 34 +++++++
tests/tcg/alpha/test-xpage-chain.c | 144 ++++++++++++++++++++++++++++
6 files changed, 251 insertions(+), 2 deletions(-)
create mode 100644 tests/tcg/alpha/gdbstub/xpage-bp.py
create mode 100644 tests/tcg/alpha/test-xpage-chain.c
diff --git ./accel/tcg/translator.c ./accel/tcg/translator.c
index 6c8fcd7a20..8879cd626f 100644
--- ./accel/tcg/translator.c
+++ ./accel/tcg/translator.c
@@ -15,6 +15,9 @@
#include "accel/tcg/cpu-mmu-index.h"
#include "exec/target_page.h"
#include "exec/translator.h"
+#ifdef CONFIG_USER_ONLY
+#include "gdbstub/user.h"
+#endif
#include "exec/plugin-gen.h"
#include "tcg/tcg-op-common.h"
#include "internal-common.h"
@@ -110,6 +113,34 @@ bool translator_is_same_page(const DisasContextBase *db, vaddr addr)
return ((addr ^ db->pc_first) & TARGET_PAGE_MASK) == 0;
}
+/*
+ * Whether a direct jump may be chained to a destination outside the page
+ * the TB started in.
+ *
+ * In user-only mode there are no page tables. Every mmap, mprotect and
+ * munmap goes through page_set_flags(), which calls tb_invalidate_phys_range()
+ * whenever the flags actually change, and tb_phys_invalidate() unlinks
+ * incoming jumps. A cross-page link is therefore broken whenever the
+ * destination page's permissions change.
+ *
+ * What the same-page rule also provides is that execution cannot enter a page
+ * without a TB lookup, and so without check_for_breakpoints(), which is what
+ * makes a breakpoint set after a block was translated take effect. Nothing
+ * invalidates on breakpoint insertion, so a link established beforehand would
+ * jump straight over it. In user-only mode breakpoints only ever come from
+ * gdb -- BP_CPU is g_assert_not_reached() there and the guest has no way to
+ * ask for one -- and gdb has to be requested with -g before the first block
+ * is translated, so a run that has no gdbstub can never acquire a breakpoint.
+ */
+static bool use_cross_page_goto_tb(void)
+{
+#ifdef CONFIG_USER_ONLY
+ return !gdb_may_set_breakpoints();
+#else
+ return false;
+#endif
+}
+
bool translator_use_goto_tb(DisasContextBase *db, vaddr dest)
{
/* Suppress goto_tb if requested. */
@@ -118,7 +149,7 @@ bool translator_use_goto_tb(DisasContextBase *db, vaddr dest)
}
/* Check for the dest on the same page as the start of the TB. */
- return translator_is_same_page(db, dest);
+ return use_cross_page_goto_tb() || translator_is_same_page(db, dest);
}
void translator_loop(CPUState *cpu, TranslationBlock *tb, int *max_insns,
diff --git ./gdbstub/user.c ./gdbstub/user.c
index 9e6f9a6f37..d810f0f38c 100644
--- ./gdbstub/user.c
+++ ./gdbstub/user.c
@@ -470,6 +470,18 @@ static void *gdbserver_accept_thread(void *arg)
#define USAGE "\nUsage: -g {port|path}[,suspend={y|n}]"
+/*
+ * Set before the guest runs and never cleared, so that code translated at
+ * any point can rely on it: with suspend=n gdb may connect long after
+ * startup, and once connected it can insert a breakpoint at any time.
+ */
+static bool gdbserver_requested;
+
+bool gdb_may_set_breakpoints(void)
+{
+ return gdbserver_requested;
+}
+
bool gdbserver_start(const char *args, Error **errp)
{
g_auto(GStrv) argv = g_strsplit(args, ",", 0);
@@ -513,6 +525,8 @@ bool gdbserver_start(const char *args, Error **errp)
return false;
}
+ gdbserver_requested = true;
+
if (suspend) {
if (gdbserver_accept(port, gdb_fd, port_or_path)) {
gdb_handlesig(first_cpu, 0, NULL, NULL, 0);
diff --git ./include/gdbstub/user.h ./include/gdbstub/user.h
index 654986d483..c091cd9758 100644
--- ./include/gdbstub/user.h
+++ ./include/gdbstub/user.h
@@ -11,6 +11,17 @@
#define MAX_SIGINFO_LENGTH 128
+/**
+ * gdb_may_set_breakpoints() - whether a breakpoint can ever be inserted
+ *
+ * In user-only mode every breakpoint comes from gdb, and gdb is only ever
+ * reachable if -g was given at startup, before the guest ran a single
+ * instruction. A run that has no gdbstub can therefore never acquire a
+ * breakpoint, which lets translation take shortcuts that a breakpoint
+ * would invalidate. Stays true once true, even if gdb detaches.
+ */
+bool gdb_may_set_breakpoints(void);
+
/**
* gdb_handlesig() - yield control to gdb
* @cpu: CPU
diff --git ./tests/tcg/alpha/Makefile.target ./tests/tcg/alpha/Makefile.target
index 36d8ed1eae..1a3f541bec 100644
--- ./tests/tcg/alpha/Makefile.target
+++ ./tests/tcg/alpha/Makefile.target
@@ -5,7 +5,7 @@
ALPHA_SRC=$(SRC_PATH)/tests/tcg/alpha
VPATH+=$(ALPHA_SRC)
-ALPHA_TESTS=hello-alpha test-cond test-cmov test-ovf test-cvttq
+ALPHA_TESTS=hello-alpha test-cond test-cmov test-ovf test-cvttq test-xpage-chain
TESTS+=$(ALPHA_TESTS)
test-cmov: EXTRA_CFLAGS=-DTEST_CMOV
@@ -16,3 +16,18 @@ test-cmov: test-cond.c
test-plugin-mem-access: CFLAGS+=-mbwx
run-test-cmov: test-cmov
+
+ifneq ($(GDB),)
+GDB_SCRIPT=$(SRC_PATH)/tests/guest-debug/run-test.py
+
+# The chaining this exercises is only enabled when no gdbstub was requested,
+# so what is under test here is that requesting one turns it back off.
+run-gdbstub-xpage-bp: test-xpage-chain
+ $(call run-test, $@, $(GDB_SCRIPT) \
+ --gdb $(GDB) \
+ --qemu $(QEMU) --qargs "$(QEMU_OPTS)" \
+ --bin "$< -b" --test $(ALPHA_SRC)/gdbstub/xpage-bp.py, \
+ breakpoint behind an established cross-page chain)
+
+EXTRA_RUNS += run-gdbstub-xpage-bp
+endif
diff --git ./tests/tcg/alpha/gdbstub/xpage-bp.py ./tests/tcg/alpha/gdbstub/xpage-bp.py
new file mode 100644
index 0000000000..f0ec14cdec
--- /dev/null
+++ ./tests/tcg/alpha/gdbstub/xpage-bp.py
@@ -0,0 +1,34 @@
+"""Test that a breakpoint set after a cross-page chain is established is hit.
+
+translator_use_goto_tb() lets a direct branch chain to another page in
+user-only builds, which is only safe because a run with no gdbstub can never
+acquire a breakpoint. This runs with one, so the chaining must be off and
+the breakpoint must still be reached.
+
+This runs as a sourced script (via -x, via run-test.py).
+
+SPDX-License-Identifier: GPL-2.0-or-later
+"""
+from test_gdbstub import main, report
+
+
+def run_test():
+ """Run through the tests one by one"""
+ gdb.Breakpoint("break_here")
+ gdb.execute("continue")
+
+ # The chain exists by now; put a breakpoint on the far side of it.
+ target = int(gdb.parse_and_eval("(unsigned long)page_b_entry"))
+ gdb.execute("break *{}".format(target))
+ gdb.execute("continue")
+
+ pc = int(gdb.parse_and_eval("(unsigned long)$pc"))
+ report(pc == target, "stopped at {:#x}, expected {:#x}".format(pc, target))
+
+ gdb.execute("delete")
+ gdb.execute("continue")
+ exitcode = int(gdb.parse_and_eval("$_exitcode"))
+ report(exitcode == 0, "{} == 0".format(exitcode))
+
+
+main(run_test)
diff --git ./tests/tcg/alpha/test-xpage-chain.c ./tests/tcg/alpha/test-xpage-chain.c
new file mode 100644
index 0000000000..23b916ffe1
--- /dev/null
+++ ./tests/tcg/alpha/test-xpage-chain.c
@@ -0,0 +1,144 @@
+/*
+ * Cross-page TB chaining hazard test.
+ *
+ * Phase 1: a direct branch (br) near the end of page A targets page B.
+ * Run it enough times that QEMU chains TB_A -> TB_B.
+ * Phase 2: mprotect page B away. Re-running must fault.
+ * Phase 3: remap page B with different code. Re-running must execute the
+ * NEW code, not a stale chained translation of the old code.
+ *
+ * With -b, phases 2 and 3 are replaced by a stop at break_here(), where the
+ * gdbstub test sets a breakpoint on page B -- after the chain exists -- and
+ * checks that re-running the chain still stops on it. See
+ * tests/tcg/alpha/gdbstub/xpage-bp.py.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+#include <stdbool.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <setjmp.h>
+#include <signal.h>
+#include <sys/mman.h>
+#include <unistd.h>
+
+#define PS 8192
+
+static sigjmp_buf jb;
+/*
+ * Written by the SIGSEGV handler and read by main(), so it must not be
+ * cached in a register across the faulting call.
+ */
+static volatile sig_atomic_t caught;
+
+/* Where the branch lands, for the gdbstub test to set a breakpoint on. */
+unsigned int *page_b_entry;
+
+/* Somewhere for the gdbstub test to stop once the chain is established. */
+void __attribute__((noinline)) break_here(void)
+{
+ asm volatile ("");
+}
+
+static void segv(int sig)
+{
+ caught = 1;
+ siglongjmp(jb, 1);
+}
+
+/* lda $0, imm($31) -> v0 = imm */
+static unsigned int lda_v0(int imm)
+{
+ return 0x201F0000u | (unsigned short)imm;
+}
+
+int main(int argc, char **argv)
+{
+ bool bp_mode = argc > 1 && strcmp(argv[1], "-b") == 0;
+ struct sigaction sa;
+ int rc = 0;
+ unsigned char *m = mmap(NULL, 2 * PS, PROT_READ | PROT_WRITE | PROT_EXEC,
+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ if (m == MAP_FAILED) {
+ perror("mmap");
+ return 2;
+ }
+
+ unsigned char *pa = m, *pb = m + PS;
+ unsigned int *entry = (unsigned int *)(pa + PS - 64);
+ unsigned int *tgt = (unsigned int *)(pb + 16);
+
+ page_b_entry = tgt;
+
+ entry[0] = lda_v0(1);
+ long disp = ((long)tgt - ((long)&entry[1] + 4)) / 4;
+ entry[1] = 0xC3E00000u | (unsigned int)(disp & 0x1FFFFF); /* br $31,tgt */
+ tgt[0] = 0x6BFA8001u; /* ret */
+ __builtin___clear_cache((char *)m, (char *)m + 2 * PS);
+
+ long (*fn)(void) = (long (*)(void))entry;
+
+ for (int i = 0; i < 200000; i++) {
+ if (fn() != 1) {
+ printf("FAIL: phase 1 wrong result\n");
+ return 1;
+ }
+ }
+ printf("phase 1 ok (chained)\n");
+
+ if (bp_mode) {
+ /*
+ * The chain from page A to page B now exists. gdb puts a breakpoint
+ * on page_b_entry here; the call below has to stop on it rather than
+ * jump over it.
+ */
+ break_here();
+ if (fn() != 1) {
+ printf("FAIL: bp phase wrong result\n");
+ return 1;
+ }
+ printf("bp phase ok\n");
+ return 0;
+ }
+
+ memset(&sa, 0, sizeof(sa));
+ sa.sa_handler = segv;
+ sigemptyset(&sa.sa_mask);
+ if (sigaction(SIGSEGV, &sa, NULL) != 0) {
+ perror("sigaction");
+ return 2;
+ }
+ if (mprotect(pb, PS, PROT_NONE) != 0) {
+ perror("mprotect");
+ return 2;
+ }
+ if (sigsetjmp(jb, 1) == 0) {
+ fn();
+ printf("FAIL: phase 2 executed page B after mprotect(PROT_NONE)\n");
+ rc = 1;
+ } else if (!caught) {
+ printf("FAIL: phase 2 longjmp without entering the handler\n");
+ rc = 1;
+ } else {
+ printf("phase 2 ok (faulted)\n");
+ }
+
+ /* Phase 3: remap with different code, expect the new code to run. */
+ if (mprotect(pb, PS, PROT_READ | PROT_WRITE | PROT_EXEC) != 0) {
+ perror("mprotect back");
+ return 2;
+ }
+ tgt[0] = lda_v0(2);
+ tgt[1] = 0x6BFA8001u;
+ __builtin___clear_cache((char *)pb, (char *)pb + PS);
+
+ long r = fn();
+ if (r != 2) {
+ printf("FAIL: phase 3 returned %ld, expected 2 (stale chain)\n", r);
+ rc = 1;
+ } else {
+ printf("phase 3 ok (new code ran)\n");
+ }
+ return rc;
+}
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH v3 6/7] RFC: accel/tcg: poison the jump cache instead of polling for indirect exits
2026-08-22 19:08 [PATCH v3 0/7] accel/tcg: cut per-block dispatch overhead Matt Turner
` (4 preceding siblings ...)
2026-08-22 19:08 ` [PATCH v3 5/7] RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds Matt Turner
@ 2026-08-22 19:08 ` Matt Turner
2026-08-22 19:08 ` [PATCH v3 7/7] RFC: tcg: fold a guest displacement into the host addressing mode Matt Turner
` (10 subsequent siblings)
16 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-08-22 19:08 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
Every translation block begins by loading cpu->neg.icount_decr.u32, testing
it and branching to the exit path. That is three host instructions at the top
of every TB, and blocks are short: an emulated alpha gcc 16.2.0 compiling a
255k line translation unit executes 34.2 billion of them at 6.04 guest
instructions each.
A block does not need to poll if every way out of it already reaches a check.
A goto_tb does not: it chains straight into its destination, with nothing in
between that looks at icount_decr, so the destination has to poll on entry.
An indirect exit does. The out-of-line path calls helper_lookup_tb_ptr()
every time, so it only needs the helper to return the epilogue while an exit
is pending. The inline probe is already covered by the machinery the
breakpoint patch added: it reads its base pointer from
cpu->tb_jmp_cache_probe and takes the slow path when the entry it finds has a
NULL tb, so pointing that base at a page of zeroes turns every indirect
dispatch into a miss, and a miss lands in the same helper.
So a pending exit becomes one more reason for tcg_cpu_may_dispatch() to say
no. The two places that set icount_decr.u16.high poison the probe; the main
loop puts it back once the flag is clear, on the same pass that already
re-evaluates the breakpoint state. The real tb_jmp_cache is untouched
throughout, so no cache contents are lost, and the fast path pays nothing:
the base was a load from CPUState either way.
The poll is therefore emitted only in blocks that emit a goto_tb. Whether a
block does is not known until its last exit has been generated, so the
decision is deferred and the load and branch are emitted retroactively at the
head of the block in gen_tb_end(), using the same emit_before_op mechanism
the can_do_io stores use. icount opts out and keeps the counter
unconditionally.
Interrupt latency is bounded at one block, as before. It does not depend on
the shape of the guest's control flow graph: a block either polls on entry or
is checked on the way out, and no run of blocks can avoid both. What changes
is where the check sits, not how often one happens.
tests/tcg/alpha/test-indirect-irq.c is added for this: a loop whose only back
edge is an indirect branch, under alarm(1). That loop's block emits no
goto_tb, so it no longer polls, and the test passes only because the dispatch
notices instead -- it hangs if the poison is removed, which is what makes it a
test of the new mechanism rather than of the old poll. The other alpha tests
still pass and the emulated compiler still produces byte-identical output.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the
SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, LTO build, on
top of the preceding patches:
before: 891,254,240,071 instructions
after: 868,811,832,620 instructions -2.52%
before: 81.45s wall clock
after: 79.85s wall clock -1.96%
The emulated compiler produces byte-identical output.
RFC because:
- The un-poison in the main loop races a concurrent poison from another
thread. The existing barrier around icount_decr.u16.high covers it -- a
poison that lands after the sync also re-set the flag, and exit_request was
stored before it -- but this deserves more eyes than the single-threaded
user-mode testing I have given it.
- Only the inline probe needs the poison, and only alpha uses the inline
probe today. Targets on the out-of-line path are covered by the helper
check alone, but that has not been measured.
- The shared zero-filled CPUJumpCache is a 1MB allocation that is never
written. A read-only mapping would express that better.
v3: Rebased onto the removal of "only poll for interrupts in blocks that can
close a cycle", which v2 sat on top of and which is dropped: it let a
straight-line run of arbitrary length go unchecked, since a block with no
backward edge polled nowhere (Richard).
The rule is now that a block polls iff it emits a goto_tb, rather than
iff it can close a control flow cycle. That keeps the bound at one block
without any analysis of the guest's control flow graph, so the objection
to the dropped patch does not carry over. The deferred-emission machinery
it needs moves here from that patch; DisasContextBase::needs_exit_check
and the hook in translator_use_goto_tb() are gone with it, and the flag
is now set by tcg_gen_goto_tb() rather than by goto_ptr emission.
All of v2's measurements were dropped: they were taken with the
cycle-analysis patch underneath, which changes both the baseline and
what is left to remove, so none of them described this patch. The
numbers above are a fresh measurement of the series as it now stands.
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
accel/tcg/cpu-exec.c | 23 ++++++++++--
accel/tcg/tcg-accel-ops.c | 1 +
accel/tcg/translator.c | 51 ++++++++++++++++++++++++--
include/hw/core/cpu.h | 8 +++--
include/tcg/tcg.h | 2 ++
tcg/tcg-op.c | 13 +++++--
tests/tcg/alpha/Makefile.target | 3 +-
tests/tcg/alpha/test-indirect-irq.c | 55 +++++++++++++++++++++++++++++
8 files changed, 144 insertions(+), 12 deletions(-)
create mode 100644 tests/tcg/alpha/test-indirect-irq.c
diff --git ./accel/tcg/cpu-exec.c ./accel/tcg/cpu-exec.c
index e546f717e8..62f6984f7a 100644
--- ./accel/tcg/cpu-exec.c
+++ ./accel/tcg/cpu-exec.c
@@ -388,6 +388,16 @@ const void *HELPER(lookup_tb_ptr)(CPUArchState *env)
*/
cpu->neg.can_do_io = true;
+ /*
+ * A block that dispatches indirectly does not emit the icount_decr poll,
+ * so this is where a pending exit is noticed for that path: either the
+ * probe was poisoned and every dispatch arrives here, or the target uses
+ * the out-of-line lookup and always did.
+ */
+ if (unlikely(cpu_loop_exit_requested(cpu))) {
+ return tcg_code_gen_epilogue;
+ }
+
TCGTBCPUState s = cpu->cc->tcg_ops->get_tb_cpu_state(cpu);
s.cflags = curr_cflags(cpu);
@@ -757,8 +767,9 @@ static inline bool cpu_handle_exception(CPUState *cpu, int *ret)
* slow path when the entry it finds has a NULL tb. Pointing the probe at a
* region that is all zeroes therefore forces every indirect dispatch into
* helper_lookup_tb_ptr(), which does the full lookup the inline probe only
- * approximates. The real jump cache is untouched, so no contents are lost
- * and recovery is a single store.
+ * approximates and returns to the main loop while an exit is pending. The
+ * real jump cache is untouched, so no contents are lost and recovery is a
+ * single store.
*
* Only ever read from, and only the tb field of one entry per dispatch, so
* one shared zero-filled cache is enough for every CPU.
@@ -785,10 +796,13 @@ static const CPUJumpCache *tb_jmp_cache_poison(void)
* the rest of the page. A block translated before the breakpoint was set is
* therefore still in the jump cache, and dispatching to it inline would step
* straight over the breakpoint.
+ *
+ * A block that dispatches indirectly also does not emit the icount_decr
+ * poll, so the dispatch is where a pending exit has to be noticed.
*/
static bool tcg_cpu_may_dispatch(CPUState *cpu)
{
- return QTAILQ_EMPTY(&cpu->breakpoints);
+ return QTAILQ_EMPTY(&cpu->breakpoints) && !cpu_loop_exit_requested(cpu);
}
/*
@@ -857,6 +871,9 @@ void tcg_kick_vcpu_thread(CPUState *cpu)
/* Ensure cpu_exec will see the exit request after TCG has exited. */
qatomic_store_release(&cpu->neg.icount_decr.u16.high, -1);
+
+ /* Blocks that only dispatch indirectly do not poll; stop them chaining. */
+ tcg_cpu_poison_jmp_cache(cpu);
}
static inline bool icount_exit_request(CPUState *cpu)
diff --git ./accel/tcg/tcg-accel-ops.c ./accel/tcg/tcg-accel-ops.c
index 560fe2554b..9eb9e861ac 100644
--- ./accel/tcg/tcg-accel-ops.c
+++ ./accel/tcg/tcg-accel-ops.c
@@ -106,6 +106,7 @@ void tcg_handle_interrupt(CPUState *cpu, int mask)
qemu_cpu_kick(cpu);
} else {
qatomic_set(&cpu->neg.icount_decr.u16.high, -1);
+ tcg_cpu_poison_jmp_cache(cpu);
}
}
diff --git ./accel/tcg/translator.c ./accel/tcg/translator.c
index 8879cd626f..89d255bd04 100644
--- ./accel/tcg/translator.c
+++ ./accel/tcg/translator.c
@@ -45,12 +45,35 @@ bool translator_io_start(DisasContextBase *db)
return true;
}
+/*
+ * A block that ends in a goto_tb chains straight to its destination: nothing
+ * between the two looks at icount_decr, so the destination has to poll on
+ * entry. A block whose exits are all indirect does not, because the dispatch
+ * itself notices -- a pending exit poisons tb_jmp_cache_probe, so the probe
+ * misses into helper_lookup_tb_ptr(), which returns the epilogue. Every block
+ * therefore either polls on entry or is checked as it leaves, which bounds
+ * interrupt latency at one block without looking at the shape of the guest's
+ * control flow graph.
+ *
+ * Which kind a block is is not known until its last exit has been emitted, so
+ * defer the decision to gen_tb_end() and emit the poll retroactively.
+ *
+ * icount needs the counter unconditionally, so it opts out.
+ */
+static bool defer_exit_check(uint32_t cflags)
+{
+ return !(cflags & CF_USE_ICOUNT);
+}
+
static TCGOp *gen_tb_start(DisasContextBase *db, uint32_t cflags)
{
TCGv_i32 count = NULL;
TCGOp *icount_start_insn = NULL;
- if ((cflags & CF_USE_ICOUNT) || !(cflags & CF_NOIRQ)) {
+ tcg_ctx->exit_check_needed = false;
+
+ if ((cflags & CF_USE_ICOUNT) ||
+ (!(cflags & CF_NOIRQ) && !defer_exit_check(cflags))) {
count = tcg_temp_new_i32();
tcg_gen_ld_i32(count, tcg_env,
offsetof(CPUState, neg.icount_decr.u32) -
@@ -76,6 +99,9 @@ static TCGOp *gen_tb_start(DisasContextBase *db, uint32_t cflags)
*/
if (cflags & CF_NOIRQ) {
tcg_ctx->exitreq_label = NULL;
+ } else if (defer_exit_check(cflags)) {
+ /* Emitted retroactively by gen_tb_end(), if this TB emits a goto_tb. */
+ tcg_ctx->exitreq_label = gen_new_label();
} else {
tcg_ctx->exitreq_label = gen_new_label();
tcg_gen_brcondi_i32(TCG_COND_LT, count, 0, tcg_ctx->exitreq_label);
@@ -91,7 +117,8 @@ static TCGOp *gen_tb_start(DisasContextBase *db, uint32_t cflags)
}
static void gen_tb_end(const TranslationBlock *tb, uint32_t cflags,
- TCGOp *icount_start_insn, int num_insns)
+ TCGOp *icount_start_insn, int num_insns,
+ TCGOp *first_insn_start)
{
if (cflags & CF_USE_ICOUNT) {
/*
@@ -102,6 +129,23 @@ static void gen_tb_end(const TranslationBlock *tb, uint32_t cflags,
tcgv_i32_arg(tcg_constant_i32(num_insns)));
}
+ if (tcg_ctx->exitreq_label && defer_exit_check(cflags) &&
+ !(cflags & CF_NOIRQ)) {
+ if (tcg_ctx->exit_check_needed) {
+ TCGv_i32 count = tcg_temp_new_i32();
+ TCGOp *save = tcg_ctx->emit_before_op;
+
+ tcg_ctx->emit_before_op = first_insn_start;
+ tcg_gen_ld_i32(count, tcg_env,
+ offsetof(CPUState, neg.icount_decr.u32) -
+ sizeof(CPUState));
+ tcg_gen_brcondi_i32(TCG_COND_LT, count, 0, tcg_ctx->exitreq_label);
+ tcg_ctx->emit_before_op = save;
+ } else {
+ tcg_ctx->exitreq_label = NULL;
+ }
+ }
+
if (tcg_ctx->exitreq_label) {
gen_set_label(tcg_ctx->exitreq_label);
tcg_gen_exit_tb(tb, TB_EXIT_REQUESTED);
@@ -238,7 +282,8 @@ void translator_loop(CPUState *cpu, TranslationBlock *tb, int *max_insns,
/* Emit code to exit the TB, as indicated by db->is_jmp. */
ops->tb_stop(db, cpu);
- gen_tb_end(tb, cflags, icount_start_insn, db->num_insns);
+ gen_tb_end(tb, cflags, icount_start_insn, db->num_insns,
+ first_insn_start);
/*
* Manage can_do_io for the translation block: set to false before
diff --git ./include/hw/core/cpu.h ./include/hw/core/cpu.h
index bd2cdd2a0b..4272740303 100644
--- ./include/hw/core/cpu.h
+++ ./include/hw/core/cpu.h
@@ -523,9 +523,11 @@ struct CPUState {
* @tb_jmp_cache_probe: base the inline jump cache probe reads.
*
* Normally @tb_jmp_cache. Pointed at a shared page of zeroes to force
- * every inline dispatch to miss and fall back to helper_lookup_tb_ptr();
- * see tcg_cpu_sync_jmp_cache(). NULL before tcg_exec_realizefn() and
- * after tcg_exec_unrealizefn().
+ * every inline dispatch to miss and fall back to helper_lookup_tb_ptr(),
+ * either because a breakpoint is set or because an exit is pending; see
+ * tcg_cpu_sync_jmp_cache(). Only generated code and the accessors in
+ * cpu-exec.c may touch it. NULL before tcg_exec_realizefn() and after
+ * tcg_exec_unrealizefn().
*/
struct CPUJumpCache *tb_jmp_cache_probe;
diff --git ./include/tcg/tcg.h ./include/tcg/tcg.h
index 7669dc1c2d..df08c10544 100644
--- ./include/tcg/tcg.h
+++ ./include/tcg/tcg.h
@@ -389,6 +389,8 @@ struct TCGContext {
struct TCGLabelPoolData *pool_labels;
TCGLabel *exitreq_label;
+ /* Set by goto_tb emission: this TB chains without reaching a check. */
+ bool exit_check_needed;
#ifdef CONFIG_PLUGIN
/*
diff --git ./tcg/tcg-op.c ./tcg/tcg-op.c
index ce77541eab..9ed04d50f3 100644
--- ./tcg/tcg-op.c
+++ ./tcg/tcg-op.c
@@ -2713,6 +2713,13 @@ void tcg_gen_goto_tb(unsigned idx)
tcg_debug_assert((tcg_ctx->goto_tb_issue_mask & (1 << idx)) == 0);
tcg_ctx->goto_tb_issue_mask |= 1 << idx;
#endif
+ /*
+ * A goto_tb chains straight into the destination, with nothing in between
+ * that looks at icount_decr, so this TB has to poll on entry. See
+ * defer_exit_check().
+ */
+ tcg_ctx->exit_check_needed = true;
+
plugin_gen_disable_mem_helpers();
tcg_gen_op1i(INDEX_op_goto_tb, 0, idx);
}
@@ -2801,8 +2808,10 @@ void tcg_gen_lookup_and_goto_ptr_tmp(TCGTemp *pc, const TranslationBlock *tb)
}
/*
- * No icount_decr poll is needed for this exit: the helper is called on
- * every dispatch and returns to the main loop while an exit is pending.
+ * No icount_decr poll is needed for this exit. The helper returns to the
+ * main loop while an exit is pending, and a pending exit poisons
+ * tb_jmp_cache_probe, so the inline path below finds a NULL tb and falls
+ * into that same helper.
*/
plugin_gen_disable_mem_helpers();
diff --git ./tests/tcg/alpha/Makefile.target ./tests/tcg/alpha/Makefile.target
index 1a3f541bec..334a088848 100644
--- ./tests/tcg/alpha/Makefile.target
+++ ./tests/tcg/alpha/Makefile.target
@@ -5,7 +5,8 @@
ALPHA_SRC=$(SRC_PATH)/tests/tcg/alpha
VPATH+=$(ALPHA_SRC)
-ALPHA_TESTS=hello-alpha test-cond test-cmov test-ovf test-cvttq test-xpage-chain
+ALPHA_TESTS=hello-alpha test-cond test-cmov test-ovf test-cvttq test-xpage-chain \
+ test-indirect-irq
TESTS+=$(ALPHA_TESTS)
test-cmov: EXTRA_CFLAGS=-DTEST_CMOV
diff --git ./tests/tcg/alpha/test-indirect-irq.c ./tests/tcg/alpha/test-indirect-irq.c
new file mode 100644
index 0000000000..df8eaed6e3
--- /dev/null
+++ ./tests/tcg/alpha/test-indirect-irq.c
@@ -0,0 +1,55 @@
+/*
+ * A loop whose only back edge is an indirect branch must still be
+ * interruptible.
+ *
+ * Blocks that dispatch indirectly do not emit the icount_decr poll; a pending
+ * exit instead poisons the inline jump cache probe so that the dispatch falls
+ * into helper_lookup_tb_ptr(), which returns to the main loop. If that
+ * mechanism breaks, this program never leaves the loop and the test times
+ * out rather than failing an assertion.
+ *
+ * A computed goto is used deliberately: a plain while(1) would end the block
+ * with a direct backward branch, that is a goto_tb, and a block that emits a
+ * goto_tb still polls -- so it would not exercise the path under test.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+#include <assert.h>
+#include <signal.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+static volatile sig_atomic_t fired;
+static volatile unsigned long iterations;
+
+static void handler(int sig)
+{
+ fired = 1;
+}
+
+int main(void)
+{
+ /*
+ * Indexing a table with a volatile index, rather than jumping through a
+ * volatile pointer: gcc happily proves a single-valued pointer constant
+ * and emits a direct branch, which is the case this test is not about.
+ */
+ void *target[2];
+ volatile int idx = 0;
+
+ assert(signal(SIGALRM, handler) != SIG_ERR);
+ alarm(1);
+
+ target[0] = &&spin;
+ target[1] = &&out;
+spin:
+ iterations++;
+ if (!fired) {
+ goto *target[idx];
+ }
+out:
+
+ printf("interrupted after %lu iterations\n", iterations);
+ return 0;
+}
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH v3 7/7] RFC: tcg: fold a guest displacement into the host addressing mode
2026-08-22 19:08 [PATCH v3 0/7] accel/tcg: cut per-block dispatch overhead Matt Turner
` (5 preceding siblings ...)
2026-08-22 19:08 ` [PATCH v3 6/7] RFC: accel/tcg: poison the jump cache instead of polling for indirect exits Matt Turner
@ 2026-08-22 19:08 ` Matt Turner
2026-08-25 22:52 ` Richard Henderson
2026-08-27 5:02 ` [PATCH v4 0/9] accel/tcg: cut per-block dispatch overhead Matt Turner
` (9 subsequent siblings)
16 siblings, 1 reply; 47+ messages in thread
From: Matt Turner @ 2026-08-22 19:08 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
Nothing in the TCG frontend interface can express a based memory access.
tcg_gen_qemu_ld/st take an address and nothing else, so a target with a
displacement in its load and store encodings -- which is most of them --
has to materialise the address first:
ldq a1,8(a0) -> mov 0x80(%rbp),%rbx reload a0
lea 0x8(%rbx),%r12 address
mov (%r12),%r12 the load
mov %r12,0x88(%rbp) spill a1
The lea is pure loss on a host whose addressing mode has a displacement
field sitting empty. It also needs a register, at the point in a block
where pressure is highest.
Fold it. After optimisation, look for an add of a constant immediately
before a guest access, defining that access's address operand, and move the
constant into a new second constant argument on the op. The add is left for
liveness to remove, so nothing breaks if its result has another use. Only
the immediately preceding op is examined: that is what the frontends emit,
and a window of one op means the pass does not have to reason about what
could have happened in between. The one thing it does check is that the add
did not clobber the base it read, since the access now reads that base
directly.
Targets opt in with TCG_TARGET_HAS_ldst_disp and an out_disp member on
TCGOutOpQemuLdSt. Without it the pass does not run, the displacement stays
zero and the existing out member is called exactly as before, so no other
backend changes behaviour or needs touching.
For x86_64 the displacement goes in the disp32 that prepare_host_addr()
already fills in for guest_base. The fold is refused unless there is no
slow path at all, which means user-only -- softmmu compares the unadjusted
address against the TLB -- and an access needing no alignment test, since
the slow path hands addr_reg to the helper and that register no longer
holds the full guest address. It is also refused if guest_base plus the
displacement leaves disp32.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the
SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, LTO build,
on top of the preceding patches, against a control measured in the same
session:
before: 868,811,832,620 instructions, 79.85s
after: 819,262,147,022 instructions, 77.30s
-5.70% instructions, -3.20% wall
Emitted code shrinks from 50.55MB to 48.80MB over the run, 167.4 to 161.6
bytes per block. Per Alpha opcode, the host bytes emitted for an access
fall as expected and nothing else moves:
ldq 18.3 -> 15.4 ldah 20.9 -> 20.9
ldl 16.6 -> 14.1 lda 12.9 -> 12.9
stq 12.8 -> 9.7 mov 9.8 -> 9.8
The emulated compiler produces byte-identical output and the alpha tests
still pass, including with a non-zero guest_base forced via -B.
RFC because:
- Only wired up for x86_64, and only for qemu_ld and qemu_st; the i128
qemu_ld2 and qemu_st2 pairs are left alone.
- Requiring that no slow path exists is stricter than necessary. Recording
the displacement in TCGLabelQemuLdst and emitting one lea on the slow
path would cover alignment-checked accesses too, at no fast path cost.
- Softmmu wants the displacement folded into the TLB comparison as well,
which is a bigger change than this one.
- A one op window catches everything the frontends emit today but is
trivially defeated by anything scheduled in between.
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
include/tcg/tcg-opc.h | 9 +++-
tcg/tcg-op-ldst.c | 3 +-
tcg/tcg.c | 86 ++++++++++++++++++++++++++++++++++++-
tcg/x86_64/tcg-target.c.inc | 61 ++++++++++++++++++++++++++
tcg/x86_64/tcg-target.h | 3 ++
5 files changed, 158 insertions(+), 4 deletions(-)
diff --git ./include/tcg/tcg-opc.h ./include/tcg/tcg-opc.h
index f3a81d5d7f..92fd34d3e3 100644
--- ./include/tcg/tcg-opc.h
+++ ./include/tcg/tcg-opc.h
@@ -125,8 +125,13 @@ DEF(goto_ptr, 0, 1, 0, TCG_OPF_BB_EXIT | TCG_OPF_BB_END)
DEF(plugin_cb, 0, 0, 1, TCG_OPF_NOT_PRESENT)
DEF(plugin_mem_cb, 0, 1, 1, TCG_OPF_NOT_PRESENT)
-DEF(qemu_ld, 1, 1, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT)
-DEF(qemu_st, 0, 2, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT)
+/*
+ * The second constant argument is a displacement to add to the address,
+ * zero unless a target advertises TCG_TARGET_HAS_ldst_disp and the fold in
+ * fold_ldst_disp() applied.
+ */
+DEF(qemu_ld, 1, 1, 2, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT)
+DEF(qemu_st, 0, 2, 2, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT)
DEF(qemu_ld2, 2, 1, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT)
DEF(qemu_st2, 0, 3, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT)
diff --git ./tcg/tcg-op-ldst.c ./tcg/tcg-op-ldst.c
index 22211ccb45..ffc5e651a6 100644
--- ./tcg/tcg-op-ldst.c
+++ ./tcg/tcg-op-ldst.c
@@ -92,7 +92,8 @@ static MemOp tcg_canonicalize_memop(MemOp op, bool is64, bool st)
static void gen_ldst1(TCGOpcode opc, TCGType type, TCGTemp *v,
TCGTemp *addr, MemOpIdx oi)
{
- TCGOp *op = tcg_gen_op3(opc, type, temp_arg(v), temp_arg(addr), oi);
+ /* The trailing zero is the address displacement; see fold_ldst_disp(). */
+ TCGOp *op = tcg_gen_op4(opc, type, temp_arg(v), temp_arg(addr), oi, 0);
TCGOP_FLAGS(op) = get_memop(oi) & MO_SIZE;
}
diff --git ./tcg/tcg.c ./tcg/tcg.c
index 489df0e738..9e6da41887 100644
--- ./tcg/tcg.c
+++ ./tcg/tcg.c
@@ -1058,6 +1058,13 @@ typedef struct TCGOutOpQemuLdSt {
TCGOutOp base;
void (*out)(TCGContext *s, TCGType type, TCGReg dest,
TCGReg addr, MemOpIdx oi);
+ /*
+ * As out(), for an access at addr + disp. Only required of targets that
+ * define TCG_TARGET_HAS_ldst_disp; for everyone else fold_ldst_disp()
+ * never runs and the displacement is always zero.
+ */
+ void (*out_disp)(TCGContext *s, TCGType type, TCGReg dest,
+ TCGReg addr, MemOpIdx oi, int32_t disp);
} TCGOutOpQemuLdSt;
typedef struct TCGOutOpQemuLdSt2 {
@@ -3574,6 +3581,77 @@ static void move_label_uses(TCGLabel *to, TCGLabel *from)
QSIMPLEQ_CONCAT(&to->branches, &from->branches);
}
+#ifndef TCG_TARGET_HAS_ldst_disp
+#define TCG_TARGET_HAS_ldst_disp 0
+static bool tcg_target_ldst_disp_ok(TCGContext *s, MemOpIdx oi, int64_t disp)
+{
+ return false;
+}
+#endif
+
+/*
+ * Fold "add addr, base, $disp" into the guest access that follows it, so
+ * that the displacement becomes part of the host addressing mode instead of
+ * a separate instruction. Frontends have no way to express this: there is
+ * no displacement operand on tcg_gen_qemu_ld/st, so a based access always
+ * costs an extra add, and an extra register to hold its result.
+ *
+ * Only an add in the op immediately before the access is recognised. That
+ * is what the frontends emit, and a window of one op means no analysis is
+ * needed of what might have happened in between. The add is left in place;
+ * liveness removes it if its result has no other use.
+ */
+static void __attribute__((noinline))
+fold_ldst_disp(TCGContext *s)
+{
+ TCGOp *op;
+
+ if (!TCG_TARGET_HAS_ldst_disp) {
+ return;
+ }
+
+ QTAILQ_FOREACH(op, &s->ops, link) {
+ TCGOp *prev;
+ TCGTemp *cts;
+ int64_t disp;
+
+ switch (op->opc) {
+ case INDEX_op_qemu_ld:
+ case INDEX_op_qemu_st:
+ break;
+ default:
+ continue;
+ }
+
+ prev = QTAILQ_PREV(op, link);
+ if (prev == NULL || prev->opc != INDEX_op_add ||
+ TCGOP_TYPE(prev) != s->addr_type) {
+ continue;
+ }
+
+ /*
+ * The add must define the address operand, and must not have
+ * clobbered the base it read: after the fold the access reads the
+ * base directly, so the base has to still hold its original value.
+ */
+ if (prev->args[0] != op->args[1] || prev->args[0] == prev->args[1]) {
+ continue;
+ }
+
+ cts = arg_temp(prev->args[2]);
+ if (cts->kind != TEMP_CONST) {
+ continue;
+ }
+ disp = cts->val;
+ if (disp == 0 || !tcg_target_ldst_disp_ok(s, op->args[2], disp)) {
+ continue;
+ }
+
+ op->args[1] = prev->args[1];
+ op->args[3] = disp;
+ }
+}
+
/* Reachable analysis : remove unreachable code. */
static void __attribute__((noinline))
reachable_code_pass(TCGContext *s)
@@ -5728,7 +5806,12 @@ static void tcg_reg_alloc_op(TCGContext *s, const TCGOp *op)
const TCGOutOpQemuLdSt *out =
container_of(all_outop[op->opc], TCGOutOpQemuLdSt, base);
- out->out(s, type, new_args[0], new_args[1], new_args[2]);
+ if (new_args[3]) {
+ out->out_disp(s, type, new_args[0], new_args[1],
+ new_args[2], new_args[3]);
+ } else {
+ out->out(s, type, new_args[0], new_args[1], new_args[2]);
+ }
}
break;
@@ -6611,6 +6694,7 @@ int tcg_gen_code(TCGContext *s, TranslationBlock *tb, uint64_t pc_start)
tcg_temp_ebb_reset_freed(s);
tcg_optimize(s);
+ fold_ldst_disp(s);
reachable_code_pass(s);
liveness_pass_0(s);
diff --git ./tcg/x86_64/tcg-target.c.inc ./tcg/x86_64/tcg-target.c.inc
index 2c8f1f3e58..d72c3db13d 100644
--- ./tcg/x86_64/tcg-target.c.inc
+++ ./tcg/x86_64/tcg-target.c.inc
@@ -2027,6 +2027,39 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContext *s, HostAddress *h,
return ldst;
}
+/*
+ * Whether the displacement of a guest access can be folded into the host
+ * addressing mode rather than materialised by a separate lea.
+ *
+ * Folding rewrites the access to use base + disp, so nothing may need a
+ * register holding the complete guest address. The softmmu TLB comparison
+ * does, and so does any slow path, which hands addr_reg to the helper. In
+ * user-only mode prepare_host_addr() creates a slow path only for an
+ * alignment test, so requiring that none is needed rules it out. What is
+ * left to check is guest_base, which shares the disp32 field.
+ */
+static bool tcg_target_ldst_disp_ok(TCGContext *s, MemOpIdx oi, int64_t disp)
+{
+#ifdef CONFIG_USER_ONLY
+ MemOp opc = get_memop(oi);
+ TCGAtomAlign aa;
+ int64_t ofs;
+
+ if (tcg_use_softmmu || s->addr_type != TCG_TYPE_I64) {
+ return false;
+ }
+ aa = atom_and_align_for_opc(s, opc, MO_ATOM_WITHIN16,
+ (opc & MO_SIZE) == MO_128);
+ if (aa.align) {
+ return false;
+ }
+ ofs = (int64_t)x86_guest_base.ofs + disp;
+ return ofs == (int32_t)ofs;
+#else
+ return false;
+#endif
+}
+
static void tcg_out_qemu_ld_direct(TCGContext *s, TCGReg datalo, TCGReg datahi,
HostAddress h, TCGType type, MemOp memop)
{
@@ -2183,9 +2216,23 @@ static void tgen_qemu_ld(TCGContext *s, TCGType type, TCGReg data,
}
}
+static void tgen_qemu_ld_disp(TCGContext *s, TCGType type, TCGReg data,
+ TCGReg addr, MemOpIdx oi, int32_t disp)
+{
+ TCGLabelQemuLdst *ldst;
+ HostAddress h;
+
+ ldst = prepare_host_addr(s, &h, addr, oi, true);
+ /* tcg_target_ldst_disp_ok() has ruled out every slow path. */
+ tcg_debug_assert(ldst == NULL);
+ h.ofs += disp;
+ tcg_out_qemu_ld_direct(s, data, -1, h, type, get_memop(oi));
+}
+
static const TCGOutOpQemuLdSt outop_qemu_ld = {
.base.static_constraint = C_O1_I1(r, L),
.out = tgen_qemu_ld,
+ .out_disp = tgen_qemu_ld_disp,
};
static void tgen_qemu_ld2(TCGContext *s, TCGType type, TCGReg datalo,
@@ -2321,9 +2368,23 @@ static void tgen_qemu_st(TCGContext *s, TCGType type, TCGReg data,
}
}
+static void tgen_qemu_st_disp(TCGContext *s, TCGType type, TCGReg data,
+ TCGReg addr, MemOpIdx oi, int32_t disp)
+{
+ TCGLabelQemuLdst *ldst;
+ HostAddress h;
+
+ ldst = prepare_host_addr(s, &h, addr, oi, false);
+ /* tcg_target_ldst_disp_ok() has ruled out every slow path. */
+ tcg_debug_assert(ldst == NULL);
+ h.ofs += disp;
+ tcg_out_qemu_st_direct(s, data, -1, h, get_memop(oi));
+}
+
static const TCGOutOpQemuLdSt outop_qemu_st = {
.base.static_constraint = C_O0_I2(L, L),
.out = tgen_qemu_st,
+ .out_disp = tgen_qemu_st_disp,
};
static void tgen_qemu_st2(TCGContext *s, TCGType type, TCGReg datalo,
diff --git ./tcg/x86_64/tcg-target.h ./tcg/x86_64/tcg-target.h
index 7ebae56a7d..8f2315c15e 100644
--- ./tcg/x86_64/tcg-target.h
+++ ./tcg/x86_64/tcg-target.h
@@ -30,6 +30,9 @@
#define TCG_TARGET_NB_REGS 32
#define MAX_CODE_GEN_BUFFER_SIZE (2 * GiB)
+/* A guest displacement can go in the disp32 of the addressing mode. */
+#define TCG_TARGET_HAS_ldst_disp 1
+
typedef enum {
TCG_REG_EAX = 0,
TCG_REG_ECX,
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* Re: [PATCH v3 1/7] accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags
2026-08-22 19:08 ` [PATCH v3 1/7] accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags Matt Turner
@ 2026-08-25 21:47 ` Richard Henderson
2026-08-27 4:57 ` Matt Turner
2026-08-26 7:46 ` Alex Bennée
1 sibling, 1 reply; 47+ messages in thread
From: Richard Henderson @ 2026-08-25 21:47 UTC (permalink / raw)
To: Matt Turner, qemu-devel; +Cc: pbonzini, philmd, alex.bennee, zhao1.liu
On 8/22/26 12:08, Matt Turner wrote:
> diff --git ./accel/tcg/tcg-all.c ./accel/tcg/tcg-all.c
> index 7186c10cf0..c9874a286a 100644
> --- ./accel/tcg/tcg-all.c
> +++ ./accel/tcg/tcg-all.c
> @@ -254,6 +254,7 @@ static void tcg_set_one_insn_per_tb(Object *obj, bool value, Error **errp)
> s->one_insn_per_tb = value;
> /* Set the global also: this changes the behaviour */
> qatomic_set(&one_insn_per_tb, value);
> + tcg_update_all_cflags();
> }
...
> diff --git ./util/log.c ./util/log.c
> index 7cffbc1bf8..3fa46a67fa 100644
> --- ./util/log.c
> +++ ./util/log.c
> @@ -27,6 +27,7 @@
> #include "qemu/thread.h"
> #include "qemu/lockable.h"
> #include "qemu/rcu.h"
> +#include "system/tcg.h"
> #ifdef CONFIG_LINUX
> #include <sys/syscall.h>
> #endif
> @@ -301,6 +302,9 @@ static bool qemu_set_log_internal(const char *filename, bool changed_name,
> #endif
> qemu_loglevel = log_flags;
>
> + /* CPU_LOG_TB_NOCHAIN feeds into the per-CPU cflags. */
> + tcg_update_all_cflags();
I'm not keen on these placements. I know they're needed for HMP, but
I'd rather have these calls in HMP, after updating state.
> +void tcg_update_cflags(CPUState *cpu)
> +{
> +}
> +
Where does this get used outside of tcg itself? I think only
tcg_update_all_cflags() should be stubbed, and indeed the only one
visible outside of accel/tcg/.
Phil, is the stub itself better placed in accel/stubs/ or accel/tcg/?
I'm unsure what organization you're working toward.
r~
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v3 2/7] accel/tcg: enlarge the TB jump cache to 64K entries
2026-08-22 19:08 ` [PATCH v3 2/7] accel/tcg: enlarge the TB jump cache to 64K entries Matt Turner
@ 2026-08-25 21:50 ` Richard Henderson
2026-08-27 4:57 ` Matt Turner
0 siblings, 1 reply; 47+ messages in thread
From: Richard Henderson @ 2026-08-25 21:50 UTC (permalink / raw)
To: Matt Turner, qemu-devel; +Cc: pbonzini, philmd, alex.bennee, zhao1.liu
On 8/22/26 12:08, Matt Turner wrote:
> The cost is memory: the cache grows from 64 KiB to 1 MiB per vCPU. That
> is easy to justify for a single-vCPU linux-user process and less obvious
> for system emulation with many vCPUs, so this may want to be sized by
> target or made tunable rather than raised unconditionally.
Why do you believe linux-user processes are single-threaded? I suppose
when emulating /bin a fair few of them are, but...
r~
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v3 4/7] RFC: tcg: probe the TB jump cache inline instead of calling a helper
2026-08-22 19:08 ` [PATCH v3 4/7] RFC: tcg: probe the TB jump cache inline instead of calling a helper Matt Turner
@ 2026-08-25 22:28 ` Richard Henderson
2026-08-27 5:00 ` Matt Turner
0 siblings, 1 reply; 47+ messages in thread
From: Richard Henderson @ 2026-08-25 22:28 UTC (permalink / raw)
To: Matt Turner, qemu-devel; +Cc: pbonzini, philmd, alex.bennee, zhao1.liu
On 8/22/26 12:08, Matt Turner wrote:
> Every indirect branch that cannot use goto_tb ends in
> tcg_gen_lookup_and_goto_ptr(), which calls helper_lookup_tb_ptr(). For an
> emulated compiler that is 8.4 billion helper calls in a single translation
> unit: 24.6% of all TB exits take this path, because jsr/ret/jmp have a
> register destination and because goto_tb is restricted to same-page
> targets.
>
> The helper itself is already tight, but each call pays for a call frame,
> the can_do_io store, the get_tb_cpu_state() indirect call through
> TCGCPUOps, curr_cflags(), and a breakpoint check, before it gets to the
> jump cache probe that almost always hits (95.8% for this workload).
>
> Emit the probe inline instead. The destination PC is already in a TCG
> temp, and the flags, cflags and cs_base the destination must match are
> constants at translation time, so the fast path is a hash, four guarded
> loads and a goto_ptr. Only a miss calls the helper, which still owns
> filling the cache.
>
> tcg_gen_lookup_and_goto_ptr() therefore takes the destination PC and the
> TB being generated, and decides for itself whether to emit the probe or
> the old helper call; there is no second entry point for targets that opt
> in. A target that cannot name its destination in a single temp passes
> NULL and gets the helper. Since the probe hashes and compares the PC as
> one 64-bit value, a 32-bit guest PC also falls back.
>
> The PC a target passes must be exactly what get_tb_cpu_state() reports for
> the destination, which is the whole of the contract. alpha, loongarch,
> mips, ppc and s390x pass their PC register, whose value is that pc by
> construction. The rest pass NULL for now: avr's TB pc is the word address
> doubled, i386's is eip before segmentation, riscv masks it to 32 bits when
> xl is MXL_RV32, hppa derives it from the IAQ, hexagon adjusts it inside a
> hardware loop, and sparc puts npc in cs_base so the guard could not hit
> anyway. Each of those is a one-line change for whoever wants to measure it.
>
> Two details matter for the generated code. The flags and cflags guards are
> folded into a single aligned 64-bit load and compare, since the fields are
> adjacent. And each path emits its own goto_ptr rather than branching to a
> shared one: a temp live across the label is spilled and reloaded on every
> dispatch, which cost 6.3% on its own.
>
> The probe cannot check everything the helper checks, and the one that
> matters is breakpoints. check_for_breakpoints() raises EXCP_DEBUG on an
> exact pc match and selects CF_BP_PAGE cflags for the rest of the page, and
> setting a breakpoint deliberately invalidates no TB, so a block translated
> before the breakpoint was set is still sitting in the jump cache. Rather
> than pay for a breakpoint test on the fast path, give the probe its own
> base pointer, tb_jmp_cache_probe, that nothing else reads, and point it at
> a page of zeroes while any breakpoint is set. Every entry the probe finds
> then has a NULL tb, so every dispatch misses into the helper and the old
> behaviour is restored exactly. cpu_breakpoint_insert() poisons the pointer,
> so the poison takes effect at the next dispatch rather than whenever that
> vCPU next reaches its main loop, which matters because a vCPU chaining
> indirectly need never reach it. The main loop puts the pointer back once the
> last breakpoint is gone; that is a load and a compare per block dispatched
> from the main loop, and nothing at all in generated code.
>
> The flags and cflags constants are safe against the other things that can
> change them. CF_PARALLEL is only ever set by begin_parallel_context(),
> which flushes first, so no block predating it survives to dispatch. gdb
> single-step is only turned on with the CPU stopped, and a block translated
> without CF_SINGLE_STEP can only be re-entered through tb_lookup(), which
> from then on demands the new cflags -- so a stale-cflags block is never the
> one running. What is left is one_insn_per_tb and -d nochain, which the
> monitor can toggle under a running vCPU without a flush; see below.
>
> Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling
> the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, LTO
> build, on top of the preceding three patches:
>
> before: 1,402,667,803,616 instructions
> after: 916,415,123,244 instructions -34.67%
>
> before: 115.56s wall clock
> after: 85.59s wall clock -25.94%
>
> The gap between the two is the point at which this stops being a
> straight-line win: the helper call was highly predictable work that the
> host pipelined well, so removing it retires far fewer instructions than it
> saves time. IPC falls from 2.48 to 2.17 across this patch for that reason.
>
> Despite emitting more code, this also reduces instruction cache pressure,
> because a dispatch no longer jumps into qemu's .text and evicts translated
> code:
>
> before: 11,735,141,703 L1-icache-load-misses
> after: 7,154,863,292 L1-icache-load-misses -39.0%
>
> The mechanism is visible directly in a profile: helper_lookup_tb_ptr()
> falls from 31.01% of samples to 0.35%, and qemu's own .text falls from
> 38.8% to 5.3%, with the balance moving into generated code.
>
> Combined with the three preceding patches, against an unmodified LTO
> build, 1,646,994,254,249 instructions fall to 916,415,123,244, or -44.36%.
> The emulated compiler produces byte-identical output throughout.
>
> Open issues, hence RFC:
>
> - one_insn_per_tb and CPU_LOG_TB_NOCHAIN can be toggled from the monitor
> while a vCPU is inside a block that was translated without them. The
> block keeps dispatching inline with the old cflags until it exits for
> some other reason. Poisoning the probe from
> tcg_update_all_curr_cflags() would close it.
> - The jump cache entry is read without qatomic_read(); entries are
> invalidated concurrently by setting tb to NULL.
> - Only alpha has been measured. The other four targets that pass a PC are
> built and boot-tested only.
>
> v3: Fold the fast path into tcg_gen_lookup_and_goto_ptr() instead of
> adding tcg_gen_lookup_and_goto_ptr_inline() beside it (Richard). It
> now takes the destination PC and the TB unconditionally, from all 38
> call sites, and picks the probe or the helper itself. Translators
> built for both values of TARGET_LONG_BITS -- arm, s390x, microblaze --
> cannot include tcg-op.h, so the common entry point takes a TCGTemp and
> reads the width from it, and tcg-op.h wraps that for everyone else;
> this is the same split as tcg_gen_qemu_ld_*_chk().
>
> Compare cs_base too. v2 listed this as an open issue, and closing it
> is what lets the choice be made generically rather than per target: a
> target that uses cs_base would otherwise have been enabled silently by
> a decision keyed on PC width alone. It costs a load and a compare on
> the fast path, and the numbers above were measured with it in place.
>
> Audited which targets may pass a real PC, the contract being that it
> is exactly what get_tb_cpu_state() reports for the destination. Five
> do; the rest pass NULL and keep the helper call, sparc among them
> because it puts npc in cs_base and so could essentially never hit.
>
> Poison the probe from cpu_breakpoint_insert() rather than only from
> the poisoned CPU's own main loop. gdb inserts a breakpoint into every
> CPU (tcg_insert_gdbstub_breakpoint()), and a thread already inside
> generated code, dispatching indirectly, need never return to the main
> loop -- so it would keep dispatching inline and run past a breakpoint
> another thread had just set. Upstream has no such window: its
> helper_lookup_tb_ptr() sees the new breakpoint at the next indirect
> branch. The un-poison in tcg_cpu_sync_jmp_cache() now re-checks after
> its store, with a barrier, so that it loses the race with a concurrent
> insert in the safe direction.
>
> Signed-off-by: Matt Turner <mattst88@gmail.com>
> ---
> accel/tcg/cpu-exec.c | 102 ++++++++++++++++++
> accel/tcg/internal-common.h | 2 +
> cpu-common.c | 11 ++
> include/hw/core/cpu.h | 9 ++
> include/system/tcg.h | 9 ++
> include/tcg/tcg-op-common.h | 16 ++-
> include/tcg/tcg-op.h | 12 +++
> stubs/tcg-cflags.c | 8 +-
> target/alpha/translate.c | 4 +-
> target/arm/tcg/translate-a64.c | 4 +-
> target/arm/tcg/translate.c | 10 +-
> target/avr/translate.c | 4 +-
> target/hexagon/translate.c | 4 +-
> target/hppa/translate.c | 6 +-
> target/i386/tcg/translate.c | 2 +-
> .../tcg/insn_trans/trans_branch.c.inc | 2 +-
> target/loongarch/tcg/translate.c | 4 +-
> target/m68k/translate.c | 2 +-
> target/microblaze/translate.c | 4 +-
> target/mips/tcg/nanomips_translate.c.inc | 2 +-
> target/mips/tcg/translate.c | 6 +-
> target/or1k/translate.c | 4 +-
> target/ppc/translate.c | 4 +-
> target/riscv/tcg/insn_trans/trans_rvzce.c.inc | 4 +-
> target/riscv/tcg/translate.c | 2 +-
> target/rx/translate.c | 4 +-
> target/s390x/tcg/translate.c | 5 +-
> target/sh4/translate.c | 4 +-
> target/sparc/translate.c | 4 +-
> target/tricore/translate.c | 4 +-
> tcg/tcg-op.c | 92 +++++++++++++++-
> 31 files changed, 300 insertions(+), 50 deletions(-)
>
> diff --git ./accel/tcg/cpu-exec.c ./accel/tcg/cpu-exec.c
> index 148e0f583e..e546f717e8 100644
> --- ./accel/tcg/cpu-exec.c
> +++ ./accel/tcg/cpu-exec.c
> @@ -752,6 +752,99 @@ static inline bool cpu_handle_exception(CPUState *cpu, int *ret)
> return false;
> }
>
> +/*
> + * The inline jump cache probe reads cpu->tb_jmp_cache_probe and takes the
> + * slow path when the entry it finds has a NULL tb. Pointing the probe at a
> + * region that is all zeroes therefore forces every indirect dispatch into
> + * helper_lookup_tb_ptr(), which does the full lookup the inline probe only
> + * approximates. The real jump cache is untouched, so no contents are lost
> + * and recovery is a single store.
> + *
> + * Only ever read from, and only the tb field of one entry per dispatch, so
> + * one shared zero-filled cache is enough for every CPU.
> + */
> +static const CPUJumpCache *tb_jmp_cache_poison(void)
> +{
> + static CPUJumpCache *poison;
> +
> + if (unlikely(poison == NULL)) {
> + /* Raced allocations are harmless: both are all zeroes. */
> + qatomic_cmpxchg(&poison, NULL, g_new0(CPUJumpCache, 1));
> + }
> + return poison;
> +}
Why allocate a jump cache at runtime? Surely this would simply be a
block of 0's in .rodata with
static const CPUJumpCache poison;
I think this needs to be split into many pieces. In particular:
(1) API change for tcg_gen_lookup_and_goto_ptr.
(2) Introduce tb_jmp_cache_probe, and the poisoning.
(3) Implementation of gen_jmp_cache_probe.
> @@ -28,6 +28,8 @@
> #include "tcg/tcg-op-common.h"
> #include "exec/translation-block.h"
> #include "exec/plugin-gen.h"
> +#include "hw/core/cpu.h"
> +#include "../accel/tcg/tb-jmp-cache.h"
> #include "tcg-internal.h"
> #include "tcg-has.h"
>
> @@ -2715,7 +2717,81 @@ void tcg_gen_goto_tb(unsigned idx)
> tcg_gen_op1i(INDEX_op_goto_tb, 0, idx);
> }
>
> -void tcg_gen_lookup_and_goto_ptr(void)
> +static void gen_jmp_cache_probe(TCGv_i64 pc, const TranslationBlock *tb)
> +{
> + TCGv_ptr jc, ent, tbp, ptr;
> + TCGv_i64 h, tmp;
> + TCGLabel *slow;
> + uint64_t fpair;
> +
> + QEMU_BUILD_BUG_ON(sizeof(((CPUJumpCache *)0)->array[0]) != 16);
> + QEMU_BUILD_BUG_ON(offsetof(TranslationBlock, cflags) !=
> + offsetof(TranslationBlock, flags) + 4);
> +
> + jc = tcg_temp_ebb_new_ptr();
> + ent = tcg_temp_ebb_new_ptr();
> + tbp = tcg_temp_ebb_new_ptr();
> + ptr = tcg_temp_ebb_new_ptr();
> + h = tcg_temp_ebb_new_i64();
> + tmp = tcg_temp_ebb_new_i64();
> + slow = gen_new_label();
> +
> + /* h = tb_jmp_cache_hash_func(pc) * sizeof(array[0]) */
> + tcg_gen_shri_i64(h, pc, TB_JMP_CACHE_BITS);
> + tcg_gen_xor_i64(h, h, pc);
> + tcg_gen_andi_i64(h, h, TB_JMP_CACHE_SIZE - 1);
> + tcg_gen_shli_i64(h, h, 4);
This hash function varies between system and user mode.
> +
> + /*
> + * Not cpu->tb_jmp_cache: the probe reads its own base so that the main
> + * loop can poison it, which is how conditions the probe cannot test for
> + * itself force every dispatch back into the helper. See
> + * tcg_cpu_sync_jmp_cache().
> + */
> + tcg_gen_ld_ptr(jc, tcg_env,
> + offsetof(CPUState, tb_jmp_cache_probe) - sizeof(CPUState));
> + tcg_gen_trunc_i64_ptr(ent, h);
> + tcg_gen_add_ptr(ent, jc, ent);
> +
> + tcg_gen_ld_ptr(tbp, ent, offsetof(CPUJumpCache, array[0].tb));
> + tcg_gen_brcondi_ptr(TCG_COND_EQ, tbp, 0, slow);
> +
> + tcg_gen_ld_i64(tmp, ent, offsetof(CPUJumpCache, array[0].pc));
> + tcg_gen_brcond_i64(TCG_COND_NE, tmp, pc, slow);
I think you can test pc first as on hash miss, that's most likely to differ.
> +
> + /*
> + * flags and cflags are adjacent uint32_t, so one aligned 64-bit load
> + * and compare covers both.
> + */
> +#if HOST_BIG_ENDIAN
> + fpair = ((uint64_t)tb->flags << 32) | tb->cflags;
> +#else
> + fpair = ((uint64_t)tb->cflags << 32) | tb->flags;
> +#endif
> + tcg_gen_ld_i64(tmp, tbp, offsetof(TranslationBlock, flags));
> + tcg_gen_brcondi_i64(TCG_COND_NE, tmp, fpair, slow);
I'm not keen on this without an additional _Static_assert that the
offset is aligned. It happens to be right now, but we're not currently
relying on that. :-)
> +
> + /*
> + * The destination must have been translated with the same cs_base, which
> + * the pc alone does not imply on a target that uses it.
> + */
> + tcg_gen_ld_i64(tmp, tbp, offsetof(TranslationBlock, cs_base));
> + tcg_gen_brcondi_i64(TCG_COND_NE, tmp, tb->cs_base, slow);
The comment here should be more generic. Despite the cs_base name,
consider this target-specific tb->flags2.
> @@ -2724,7 +2800,21 @@ void tcg_gen_lookup_and_goto_ptr(void)
> return;
> }
>
> + /*
> + * No icount_decr poll is needed for this exit: the helper is called on
> + * every dispatch and returns to the main loop while an exit is pending.
> + */
> plugin_gen_disable_mem_helpers();
> +
> + /*
> + * The inline probe hashes and compares the pc as a single 64-bit value.
> + * A target with a 32-bit guest PC keeps the helper call.
> + */
> + if (pc && pc->type == TCG_TYPE_I64) {
> + gen_jmp_cache_probe(temp_tcgv_i64(pc), tb);
> + return;
> + }
Just extend, clearly.
r~
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v3 7/7] RFC: tcg: fold a guest displacement into the host addressing mode
2026-08-22 19:08 ` [PATCH v3 7/7] RFC: tcg: fold a guest displacement into the host addressing mode Matt Turner
@ 2026-08-25 22:52 ` Richard Henderson
2026-08-27 4:57 ` Matt Turner
0 siblings, 1 reply; 47+ messages in thread
From: Richard Henderson @ 2026-08-25 22:52 UTC (permalink / raw)
To: Matt Turner, qemu-devel; +Cc: pbonzini, philmd, alex.bennee, zhao1.liu
On 8/22/26 12:08, Matt Turner wrote:
> RFC because:
>
> - Only wired up for x86_64, and only for qemu_ld and qemu_st; the i128
> qemu_ld2 and qemu_st2 pairs are left alone.
> - Requiring that no slow path exists is stricter than necessary. Recording
> the displacement in TCGLabelQemuLdst and emitting one lea on the slow
> path would cover alignment-checked accesses too, at no fast path cost.
> - Softmmu wants the displacement folded into the TLB comparison as well,
> which is a bigger change than this one.
> - A one op window catches everything the frontends emit today but is
> trivially defeated by anything scheduled in between.
Plausible. Several times I've considered exposing complex addressing
modes to the translators so that things that do LEA are kept intact for
awhile. There are plenty of host-specific code sequences for x << s + y
+ c, even before we fold that into the memory access. Then expose the
host memory path to tcg ops, somehow, and finally implement a simple CSE
pass. But yeah, hand waving is as far as I've ever gone.
On the second point, I guess you also assuming the offset is also
aligned? I.e. for X + 8*N, you can test X for 8-byte alignment without
constructing the complete address.
> +static bool tcg_target_ldst_disp_ok(TCGContext *s, MemOpIdx oi, int64_t disp)
> +{
> +#ifdef CONFIG_USER_ONLY
> + MemOp opc = get_memop(oi);
> + TCGAtomAlign aa;
> + int64_t ofs;
> +
> + if (tcg_use_softmmu || s->addr_type != TCG_TYPE_I64) {
> + return false;
> + }
> + aa = atom_and_align_for_opc(s, opc, MO_ATOM_WITHIN16,
> + (opc & MO_SIZE) == MO_128);
> + if (aa.align) {
> + return false;
> + }
> + ofs = (int64_t)x86_guest_base.ofs + disp;
> + return ofs == (int32_t)ofs;
> +#else
> + return false;
> +#endif
> +}
These compilation mode tests belong...
> +static void __attribute__((noinline))
> +fold_ldst_disp(TCGContext *s)
> +{
> + TCGOp *op;
> +
> + if (!TCG_TARGET_HAS_ldst_disp) {
> + return;
> + }
... here, before we step over the loop. You might as well pass MemOp to
the target function and not MemOpIdx -- nothing about the mmu_idx is
relevant.
Ideally, the atom_and_align test would also be done generically, not
requiring each target to replicate that boilerplate.
r~
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v3 1/7] accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags
2026-08-22 19:08 ` [PATCH v3 1/7] accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags Matt Turner
2026-08-25 21:47 ` Richard Henderson
@ 2026-08-26 7:46 ` Alex Bennée
2026-08-27 4:57 ` Matt Turner
1 sibling, 1 reply; 47+ messages in thread
From: Alex Bennée @ 2026-08-26 7:46 UTC (permalink / raw)
To: Matt Turner; +Cc: qemu-devel, richard.henderson, pbonzini, philmd, zhao1.liu
Matt Turner <mattst88@gmail.com> writes:
> curr_cflags() is called once per TB dispatch, from helper_lookup_tb_ptr()
> and from the cpu_exec() loop. It recomputes the same value every time:
>
> uint32_t cflags = cpu->tcg_cflags;
> if (unlikely(cpu_single_stepping(cpu))) { ... }
> else if (qatomic_read(&one_insn_per_tb)) { ... }
> else if (qemu_loglevel_mask(CPU_LOG_TB_NOCHAIN)) { ... }
>
> That is three loads and three branches on the hottest path in the
> interpreter, for state that changes only when gdb enables single-step,
> when one-insn-per-tb is toggled, or when the log mask changes.
>
> None of the three has to be sampled at dispatch time. Fold each into
> CPUState::tcg_cflags where it changes and curr_cflags() becomes a single
> load of a field that TB lookup has to read anyway.
>
> The derived bits -- CF_COUNT_MASK, CF_NO_GOTO_TB, CF_NO_GOTO_PTR and
> CF_SINGLE_STEP -- are never set by tcg_cflags_set(), so tcg_update_cflags()
> can recompute them in place without disturbing the rest, and conversely
> tcg_cflags_set() ORs in its bits without disturbing them.
>
> There are three places to call it:
>
> - tcg_exec_realizefn(), so that a CPU created after the command line has
> been parsed starts out with the right value. This covers user-only,
> where tcg_cpu_init_cflags() is not reached. linux-user's cpu_copy()
> copies tcg_cflags wholesale, so a cloned thread inherits it.
>
> - cpu_single_step(), which changes one CPU and runs either on that CPU's
> thread or with it stopped.
>
> - tcg_set_one_insn_per_tb() and qemu_set_log_internal(), which change
> every CPU. Both can be reached from the monitor while the vCPUs are
> running -- 'one-insn-per-tb on' and 'log nochain' -- so the update is
> queued with async_safe_run_on_cpu() and each CPU writes its own cflags
> with the others halted.
>
> Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling
> the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, in a
> build configured with --enable-lto:
>
> before: 1,646,994,254,249 instructions
> after: 1,562,204,796,597 instructions -5.15%
>
> That workload issues 8.4 billion dispatches, so the per-call saving is
> small but the aggregate is not. The emulated compiler produces
> byte-identical output before and after.
>
> Wall clock does not move: 133.19s to 132.58s, a 0.46% difference against a
> run-to-run spread larger than that. The removed work is a few predictable
> loads and branches that the host executes largely in parallel with the
> surrounding dispatch, so this patch is worth taking for the instruction
> count and for what it enables, not for a time saving that can be measured
> on its own.
>
> Signed-off-by: Matt Turner <mattst88@gmail.com>
> ---
> accel/tcg/cpu-exec-common.c | 33 ++++++++++++++++++++++++++++++---
> accel/tcg/cpu-exec.c | 3 +++
> accel/tcg/internal-common.h | 11 +++++++++--
> accel/tcg/tcg-all.c | 1 +
> cpu-target.c | 3 +++
> include/system/tcg.h | 12 ++++++++++++
> stubs/meson.build | 1 +
> stubs/tcg-cflags.c | 16 ++++++++++++++++
> util/log.c | 4 ++++
> 9 files changed, 79 insertions(+), 5 deletions(-)
> create mode 100644 stubs/tcg-cflags.c
>
> diff --git ./accel/tcg/cpu-exec-common.c ./accel/tcg/cpu-exec-common.c
> index 44e84344f3..dd2be475e2 100644
> --- ./accel/tcg/cpu-exec-common.c
> +++ ./accel/tcg/cpu-exec-common.c
> @@ -36,9 +36,16 @@ void tcg_cflags_set(CPUState *cpu, uint32_t flags)
> cpu->tcg_cflags |= flags;
> }
>
> -uint32_t curr_cflags(CPUState *cpu)
> +/*
> + * The bits of CPUState::tcg_cflags that tcg_cflags_set() never sets, because
> + * they are derived from gdb single-step, one-insn-per-tb and -d nochain.
> + */
> +#define CF_DERIVED (CF_COUNT_MASK | CF_NO_GOTO_TB | CF_NO_GOTO_PTR | \
> + CF_SINGLE_STEP)
> +
> +void tcg_update_cflags(CPUState *cpu)
> {
> - uint32_t cflags = cpu->tcg_cflags;
> + uint32_t cflags = cpu->tcg_cflags & ~CF_DERIVED;
>
> /*
> * Record gdb single-step. We should be exiting the TB by raising
> @@ -55,7 +62,27 @@ uint32_t curr_cflags(CPUState *cpu)
> cflags |= CF_NO_GOTO_TB;
> }
>
> - return cflags;
> + cpu->tcg_cflags = cflags;
> +}
> +
> +static void tcg_update_cflags_work(CPUState *cpu, run_on_cpu_data data)
> +{
> + tcg_update_cflags(cpu);
> +}
> +
> +void tcg_update_all_cflags(void)
> +{
> + CPUState *cpu;
> +
> + /*
> + * one-insn-per-tb and -d nochain can both be changed from the monitor
> + * while the vCPUs are running. Have each CPU update its own cflags
> + * with the others halted, so that no dispatch can read a value that
> + * another thread is in the middle of writing.
> + */
> + CPU_FOREACH(cpu) {
> + async_safe_run_on_cpu(cpu, tcg_update_cflags_work,
> RUN_ON_CPU_NULL);
I don't think this is wrong but are we really seeing cross-vCPU updates
of cpu->cflags? I suspect async_run_on_cpu would be enough to trigger an
update from a non-vCPU thread to the vCPU.
You could even pass the sub-set of flags down in the user data and maybe
avoid having to use global atomics for those flags.
> + }
> }
>
> /* exit the current TB, but without causing any exception to be raised */
> diff --git ./accel/tcg/cpu-exec.c ./accel/tcg/cpu-exec.c
> index 257211235d..148e0f583e 100644
> --- ./accel/tcg/cpu-exec.c
> +++ ./accel/tcg/cpu-exec.c
> @@ -1068,6 +1068,9 @@ bool tcg_exec_realizefn(CPUState *cpu, Error **errp)
> tcg_target_initialized = true;
> }
>
> + /* Pick up one-insn-per-tb and -d nochain from the command line. */
> + tcg_update_cflags(cpu);
> +
> cpu->tb_jmp_cache = g_new0(CPUJumpCache, 1);
> tlb_init(cpu);
> #ifndef CONFIG_USER_ONLY
> diff --git ./accel/tcg/internal-common.h ./accel/tcg/internal-common.h
> index 9e7be2d78d..853d1b51ee 100644
> --- ./accel/tcg/internal-common.h
> +++ ./accel/tcg/internal-common.h
> @@ -69,8 +69,15 @@ void tlb_destroy(CPUState *cpu);
> bool tcg_exec_realizefn(CPUState *cpu, Error **errp);
> void tcg_exec_unrealizefn(CPUState *cpu);
>
> -/* current cflags for hashing/comparison */
> -uint32_t curr_cflags(CPUState *cpu);
> +/*
> + * Current cflags for hashing/comparison. Everything that feeds into the
> + * value is folded into CPUState::tcg_cflags when it changes, by
> + * tcg_update_cflags(), so that TB dispatch only has to load it.
> + */
> +static inline uint32_t curr_cflags(CPUState *cpu)
> +{
> + return cpu->tcg_cflags;
> +}
>
> void tb_check_watchpoint(CPUState *cpu, uintptr_t retaddr);
>
> diff --git ./accel/tcg/tcg-all.c ./accel/tcg/tcg-all.c
> index 7186c10cf0..c9874a286a 100644
> --- ./accel/tcg/tcg-all.c
> +++ ./accel/tcg/tcg-all.c
> @@ -254,6 +254,7 @@ static void tcg_set_one_insn_per_tb(Object *obj, bool value, Error **errp)
> s->one_insn_per_tb = value;
> /* Set the global also: this changes the behaviour */
> qatomic_set(&one_insn_per_tb, value);
> + tcg_update_all_cflags();
> }
>
> static void tcg_accel_class_init(ObjectClass *oc, const void *data)
> diff --git ./cpu-target.c ./cpu-target.c
> index 4783845c9b..50be591acf 100644
> --- ./cpu-target.c
> +++ ./cpu-target.c
> @@ -24,6 +24,7 @@
> #include "exec/replay-core.h"
> #include "exec/log.h"
> #include "hw/core/cpu.h"
> +#include "system/tcg.h"
> #include "trace/trace-root.h"
>
> /* enable or disable single step mode. EXCP_DEBUG is returned by the
> @@ -35,6 +36,8 @@ void cpu_single_step(CPUState *cpu, unsigned flags)
> cpu->singlestep_flags, flags);
> cpu->singlestep_flags = flags;
>
> + tcg_update_cflags(cpu);
> +
> #if !defined(CONFIG_USER_ONLY)
> const AccelOpsClass *ops = cpus_get_accel();
> if (ops->update_guest_debug) {
> diff --git ./include/system/tcg.h ./include/system/tcg.h
> index 7622dcea30..2c2dbc753b 100644
> --- ./include/system/tcg.h
> +++ ./include/system/tcg.h
> @@ -17,6 +17,18 @@ extern bool tcg_allowed;
> #define tcg_enabled() 0
> #endif
>
> +/*
> + * Recompute the parts of CPUState::tcg_cflags that TB dispatch consumes but
> + * tcg_cflags_set() does not provide: gdb single-step, one-insn-per-tb and
> + * the CPU_LOG_TB_NOCHAIN log flag. Call whenever one of those changes.
> + *
> + * tcg_update_cflags() updates one CPU and must be called from that CPU's
> + * thread, or with it stopped. tcg_update_all_cflags() updates every CPU
> + * and is safe to call from the monitor while the vCPUs run.
> + */
> +void tcg_update_cflags(CPUState *cpu);
> +void tcg_update_all_cflags(void);
> +
> /**
> * qemu_tcg_mttcg_enabled:
> * Check whether we are running MultiThread TCG or not.
> diff --git ./stubs/meson.build ./stubs/meson.build
> index 3b2f2680b1..0025e79226 100644
> --- ./stubs/meson.build
> +++ ./stubs/meson.build
> @@ -3,6 +3,7 @@
> # below, so that it is clear who needs the stubbed functionality.
>
> stub_ss.add(files('cpu-get-clock.c'))
> +stub_ss.add(files('tcg-cflags.c'))
> stub_ss.add(files('fdset.c'))
> stub_ss.add(files('iothread-lock.c'))
> stub_ss.add(files('is-daemonized.c'))
> diff --git ./stubs/tcg-cflags.c ./stubs/tcg-cflags.c
> new file mode 100644
> index 0000000000..cb278e94aa
> --- /dev/null
> +++ ./stubs/tcg-cflags.c
> @@ -0,0 +1,16 @@
> +/*
> + * Stub for tcg_update_all_cflags(), for binaries that link util/log.c
> + * or cpu-target.c but not TCG.
> + *
> + * SPDX-License-Identifier: GPL-2.0-or-later
> + */
> +#include "qemu/osdep.h"
> +#include "system/tcg.h"
> +
> +void tcg_update_cflags(CPUState *cpu)
> +{
> +}
> +
> +void tcg_update_all_cflags(void)
> +{
> +}
> diff --git ./util/log.c ./util/log.c
> index 7cffbc1bf8..3fa46a67fa 100644
> --- ./util/log.c
> +++ ./util/log.c
> @@ -27,6 +27,7 @@
> #include "qemu/thread.h"
> #include "qemu/lockable.h"
> #include "qemu/rcu.h"
> +#include "system/tcg.h"
> #ifdef CONFIG_LINUX
> #include <sys/syscall.h>
> #endif
> @@ -301,6 +302,9 @@ static bool qemu_set_log_internal(const char *filename, bool changed_name,
> #endif
> qemu_loglevel = log_flags;
>
> + /* CPU_LOG_TB_NOCHAIN feeds into the per-CPU cflags. */
> + tcg_update_all_cflags();
> +
> daemonized = is_daemonized();
> need_to_open_file = false;
> if (!daemonized) {
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v3 5/7] RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds
2026-08-22 19:08 ` [PATCH v3 5/7] RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds Matt Turner
@ 2026-08-26 7:51 ` Alex Bennée
2026-08-27 4:57 ` Matt Turner
0 siblings, 1 reply; 47+ messages in thread
From: Alex Bennée @ 2026-08-26 7:51 UTC (permalink / raw)
To: Matt Turner; +Cc: qemu-devel, richard.henderson, pbonzini, philmd, zhao1.liu
Matt Turner <mattst88@gmail.com> writes:
> translator_use_goto_tb() refuses to chain unless the destination is on the
> same page as the start of the TB. For guests whose text is much larger than
> a page this is expensive: an emulated alpha gcc compiling a 255k line
> translation unit takes the indirect dispatch path for 8.4 billion of its
> 34.2 billion TB exits, and a large share of those are ordinary direct
> branches that simply crossed an 8 KiB page boundary.
>
> The restriction was made unconditional by d3a2a1d803 ("accel/tcg:
> Introduce translator_use_goto_tb"), whose rationale was:
>
> Various targets avoid the page crossing test for CONFIG_USER_ONLY,
> but that is wrong: mmap and mprotect can change page permissions.
>
> That is true, but in user-only builds the invalidation path already covers
> it. There are no page tables: every mmap, mprotect and munmap reaches
> page_set_flags(), which calls tb_invalidate_phys_range() whenever the flags
> actually change, and tb_phys_invalidate() calls tb_jmp_unlink() to reset
> incoming jumps. A chained cross-page jump is therefore broken whenever the
> destination page's permissions change. This is not true in system mode,
> where TBs are keyed by physical address and a page table change invalidates
> nothing, so the restriction is kept there.
> tests/tcg/alpha/Makefile.target | 17 +++-
> tests/tcg/alpha/gdbstub/xpage-bp.py | 34 +++++++
> tests/tcg/alpha/test-xpage-chain.c | 144 ++++++++++++++++++++++++++++
<snip>
Given alpha linux-user isn't widely built or used it would be better if
we could make the xpage chaining tests multiarch so they are exercised
on all *-user targets.
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v3 2/7] accel/tcg: enlarge the TB jump cache to 64K entries
2026-08-25 21:50 ` Richard Henderson
@ 2026-08-27 4:57 ` Matt Turner
0 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-08-27 4:57 UTC (permalink / raw)
To: Richard Henderson; +Cc: qemu-devel, pbonzini, philmd, alex.bennee, zhao1.liu
On Tue, Aug 25, 2026 at 5:50 PM Richard Henderson
<richard.henderson@linaro.org> wrote:
>
> On 8/22/26 12:08, Matt Turner wrote:
> > The cost is memory: the cache grows from 64 KiB to 1 MiB per vCPU. That
> > is easy to justify for a single-vCPU linux-user process and less obvious
> > for system emulation with many vCPUs, so this may want to be sized by
> > target or made tunable rather than raised unconditionally.
>
> Why do you believe linux-user processes are single-threaded? I suppose
> when emulating /bin a fair few of them are, but...
I don't -- I'd conflated "one process" with "one CPUState". linux-user
creates a CPUState per guest thread, so a threaded guest pays the
megabyte per thread, just as system emulation pays it per vCPU. Commit
message fixed.
The sizing question stays open, which is why the patch says so. I
don't have a threaded workload where 64 KiB is the better trade and
would welcome one.
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v3 1/7] accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags
2026-08-26 7:46 ` Alex Bennée
@ 2026-08-27 4:57 ` Matt Turner
0 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-08-27 4:57 UTC (permalink / raw)
To: Alex Bennée
Cc: qemu-devel, richard.henderson, pbonzini, philmd, zhao1.liu
On Wed, Aug 26, 2026 at 3:46 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>
> Matt Turner <mattst88@gmail.com> writes:
>
> > curr_cflags() is called once per TB dispatch, from helper_lookup_tb_ptr()
> > and from the cpu_exec() loop. It recomputes the same value every time:
> >
> > uint32_t cflags = cpu->tcg_cflags;
> > if (unlikely(cpu_single_stepping(cpu))) { ... }
> > else if (qatomic_read(&one_insn_per_tb)) { ... }
> > else if (qemu_loglevel_mask(CPU_LOG_TB_NOCHAIN)) { ... }
> >
> > That is three loads and three branches on the hottest path in the
> > interpreter, for state that changes only when gdb enables single-step,
> > when one-insn-per-tb is toggled, or when the log mask changes.
> >
> > None of the three has to be sampled at dispatch time. Fold each into
> > CPUState::tcg_cflags where it changes and curr_cflags() becomes a single
> > load of a field that TB lookup has to read anyway.
> >
> > The derived bits -- CF_COUNT_MASK, CF_NO_GOTO_TB, CF_NO_GOTO_PTR and
> > CF_SINGLE_STEP -- are never set by tcg_cflags_set(), so tcg_update_cflags()
> > can recompute them in place without disturbing the rest, and conversely
> > tcg_cflags_set() ORs in its bits without disturbing them.
> >
> > There are three places to call it:
> >
> > - tcg_exec_realizefn(), so that a CPU created after the command line has
> > been parsed starts out with the right value. This covers user-only,
> > where tcg_cpu_init_cflags() is not reached. linux-user's cpu_copy()
> > copies tcg_cflags wholesale, so a cloned thread inherits it.
> >
> > - cpu_single_step(), which changes one CPU and runs either on that CPU's
> > thread or with it stopped.
> >
> > - tcg_set_one_insn_per_tb() and qemu_set_log_internal(), which change
> > every CPU. Both can be reached from the monitor while the vCPUs are
> > running -- 'one-insn-per-tb on' and 'log nochain' -- so the update is
> > queued with async_safe_run_on_cpu() and each CPU writes its own cflags
> > with the others halted.
> >
> > Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling
> > the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, in a
> > build configured with --enable-lto:
> >
> > before: 1,646,994,254,249 instructions
> > after: 1,562,204,796,597 instructions -5.15%
> >
> > That workload issues 8.4 billion dispatches, so the per-call saving is
> > small but the aggregate is not. The emulated compiler produces
> > byte-identical output before and after.
> >
> > Wall clock does not move: 133.19s to 132.58s, a 0.46% difference against a
> > run-to-run spread larger than that. The removed work is a few predictable
> > loads and branches that the host executes largely in parallel with the
> > surrounding dispatch, so this patch is worth taking for the instruction
> > count and for what it enables, not for a time saving that can be measured
> > on its own.
> >
> > Signed-off-by: Matt Turner <mattst88@gmail.com>
> > ---
> > accel/tcg/cpu-exec-common.c | 33 ++++++++++++++++++++++++++++++---
> > accel/tcg/cpu-exec.c | 3 +++
> > accel/tcg/internal-common.h | 11 +++++++++--
> > accel/tcg/tcg-all.c | 1 +
> > cpu-target.c | 3 +++
> > include/system/tcg.h | 12 ++++++++++++
> > stubs/meson.build | 1 +
> > stubs/tcg-cflags.c | 16 ++++++++++++++++
> > util/log.c | 4 ++++
> > 9 files changed, 79 insertions(+), 5 deletions(-)
> > create mode 100644 stubs/tcg-cflags.c
> >
> > diff --git ./accel/tcg/cpu-exec-common.c ./accel/tcg/cpu-exec-common.c
> > index 44e84344f3..dd2be475e2 100644
> > --- ./accel/tcg/cpu-exec-common.c
> > +++ ./accel/tcg/cpu-exec-common.c
> > @@ -36,9 +36,16 @@ void tcg_cflags_set(CPUState *cpu, uint32_t flags)
> > cpu->tcg_cflags |= flags;
> > }
> >
> > -uint32_t curr_cflags(CPUState *cpu)
> > +/*
> > + * The bits of CPUState::tcg_cflags that tcg_cflags_set() never sets, because
> > + * they are derived from gdb single-step, one-insn-per-tb and -d nochain.
> > + */
> > +#define CF_DERIVED (CF_COUNT_MASK | CF_NO_GOTO_TB | CF_NO_GOTO_PTR | \
> > + CF_SINGLE_STEP)
> > +
> > +void tcg_update_cflags(CPUState *cpu)
> > {
> > - uint32_t cflags = cpu->tcg_cflags;
> > + uint32_t cflags = cpu->tcg_cflags & ~CF_DERIVED;
> >
> > /*
> > * Record gdb single-step. We should be exiting the TB by raising
> > @@ -55,7 +62,27 @@ uint32_t curr_cflags(CPUState *cpu)
> > cflags |= CF_NO_GOTO_TB;
> > }
> >
> > - return cflags;
> > + cpu->tcg_cflags = cflags;
> > +}
> > +
> > +static void tcg_update_cflags_work(CPUState *cpu, run_on_cpu_data data)
> > +{
> > + tcg_update_cflags(cpu);
> > +}
> > +
> > +void tcg_update_all_cflags(void)
> > +{
> > + CPUState *cpu;
> > +
> > + /*
> > + * one-insn-per-tb and -d nochain can both be changed from the monitor
> > + * while the vCPUs are running. Have each CPU update its own cflags
> > + * with the others halted, so that no dispatch can read a value that
> > + * another thread is in the middle of writing.
> > + */
> > + CPU_FOREACH(cpu) {
> > + async_safe_run_on_cpu(cpu, tcg_update_cflags_work,
> > RUN_ON_CPU_NULL);
>
> I don't think this is wrong but are we really seeing cross-vCPU updates
> of cpu->cflags?
Fixed in v4. The monitor path was the cross-vCPU one -- v3 had the
monitor thread storing into every cpu->tcg_cflags -- and v4 queues the
update instead, so each vCPU writes its own field from its own thread.
The one foreign writer left is cpu_single_step(), and it's the same
one that was already there. gdb_continue_partial() in gdbstub/user.c
walks CPU_FOREACH and can hit a thread that's still running, since
gdb_handlesig() only parks the thread that trapped. But before this
patch, curr_cflags() on CPU X read X->singlestep_flags and
cpu_single_step() stored to it from whichever thread gdb was on. Now
the store lands in X->tcg_cflags. Same width, same plain accesses,
same writer, same reader -- the patch changes which field carries the
state, not how it's synchronized. Nothing here wants atomics that
didn't want them before.
I did fix the commit message, which claimed cpu_single_step() always
runs on the owning thread or with the CPU stopped. True in system
mode, not in the user-mode gdb_continue_partial() case.
> I suspect async_run_on_cpu would be enough to trigger an
> update from a non-vCPU thread to the vCPU.
Agreed, fixed in v4. async_safe_run_on_cpu() was overkill.
> You could even pass the sub-set of flags down in the user data and maybe
> avoid having to use global atomics for those flags.
I left this, but tell me if you'd rather have it. The only atomic is
qatomic_read(&one_insn_per_tb), and the patch already takes it off the
hot path -- it ran on every dispatch, now it runs at realize, at a gdb
step change, or on an HMP command. Passing a snapshot would remove
that last read, but one_insn_per_tb and qemu_loglevel are read by
other code too, so the queued CPU would be working from a snapshot
while everyone else sees the live global. For a path this cold that
seemed like the worse trade.
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v3 1/7] accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags
2026-08-25 21:47 ` Richard Henderson
@ 2026-08-27 4:57 ` Matt Turner
0 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-08-27 4:57 UTC (permalink / raw)
To: Richard Henderson; +Cc: qemu-devel, pbonzini, philmd, alex.bennee, zhao1.liu
On Tue, Aug 25, 2026 at 5:47 PM Richard Henderson
<richard.henderson@linaro.org> wrote:
>
>
> On 8/22/26 12:08, Matt Turner wrote:
> > diff --git ./accel/tcg/tcg-all.c ./accel/tcg/tcg-all.c
> > index 7186c10cf0..c9874a286a 100644
> > --- ./accel/tcg/tcg-all.c
> > +++ ./accel/tcg/tcg-all.c
> > @@ -254,6 +254,7 @@ static void tcg_set_one_insn_per_tb(Object *obj, bool value, Error **errp)
> > s->one_insn_per_tb = value;
> > /* Set the global also: this changes the behaviour */
> > qatomic_set(&one_insn_per_tb, value);
> > + tcg_update_all_cflags();
> > }
> ...
> > diff --git ./util/log.c ./util/log.c
> > index 7cffbc1bf8..3fa46a67fa 100644
> > --- ./util/log.c
> > +++ ./util/log.c
> > @@ -27,6 +27,7 @@
> > #include "qemu/thread.h"
> > #include "qemu/lockable.h"
> > #include "qemu/rcu.h"
> > +#include "system/tcg.h"
> > #ifdef CONFIG_LINUX
> > #include <sys/syscall.h>
> > #endif
> > @@ -301,6 +302,9 @@ static bool qemu_set_log_internal(const char *filename, bool changed_name,
> > #endif
> > qemu_loglevel = log_flags;
> >
> > + /* CPU_LOG_TB_NOCHAIN feeds into the per-CPU cflags. */
> > + tcg_update_all_cflags();
>
> I'm not keen on these placements. I know they're needed for HMP, but
> I'd rather have these calls in HMP, after updating state.
>
> > +void tcg_update_cflags(CPUState *cpu)
> > +{
> > +}
> > +
> Where does this get used outside of tcg itself? I think only
> tcg_update_all_cflags() should be stubbed, and indeed the only one
> visible outside of accel/tcg/.
cpu_single_step(), in cpu-target.c, which is in both user_ss and
system_ss and always linked. So both entry points need stubbing.
> Phil, is the stub itself better placed in accel/stubs/ or accel/tcg/?
> I'm unsure what organization you're working toward.
v4 moves it to accel/stubs/tcg-stub.c, alongside the other accelerator
stubs. Verified --disable-tcg links. Easy to move again if Phil
prefers elsewhere.
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v3 5/7] RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds
2026-08-26 7:51 ` Alex Bennée
@ 2026-08-27 4:57 ` Matt Turner
0 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-08-27 4:57 UTC (permalink / raw)
To: Alex Bennée
Cc: qemu-devel, richard.henderson, pbonzini, philmd, zhao1.liu
On Wed, Aug 26, 2026 at 3:51 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>
> Matt Turner <mattst88@gmail.com> writes:
>
> > translator_use_goto_tb() refuses to chain unless the destination is on the
> > same page as the start of the TB. For guests whose text is much larger than
> > a page this is expensive: an emulated alpha gcc compiling a 255k line
> > translation unit takes the indirect dispatch path for 8.4 billion of its
> > 34.2 billion TB exits, and a large share of those are ordinary direct
> > branches that simply crossed an 8 KiB page boundary.
> >
> > The restriction was made unconditional by d3a2a1d803 ("accel/tcg:
> > Introduce translator_use_goto_tb"), whose rationale was:
> >
> > Various targets avoid the page crossing test for CONFIG_USER_ONLY,
> > but that is wrong: mmap and mprotect can change page permissions.
> >
> > That is true, but in user-only builds the invalidation path already covers
> > it. There are no page tables: every mmap, mprotect and munmap reaches
> > page_set_flags(), which calls tb_invalidate_phys_range() whenever the flags
> > actually change, and tb_phys_invalidate() calls tb_jmp_unlink() to reset
> > incoming jumps. A chained cross-page jump is therefore broken whenever the
> > destination page's permissions change. This is not true in system mode,
> > where TBs are keyed by physical address and a page table change invalidates
> > nothing, so the restriction is kept there.
> > tests/tcg/alpha/Makefile.target | 17 +++-
> > tests/tcg/alpha/gdbstub/xpage-bp.py | 34 +++++++
> > tests/tcg/alpha/test-xpage-chain.c | 144 ++++++++++++++++++++++++++++
> <snip>
>
> Given alpha linux-user isn't widely built or used it would be better if
> we could make the xpage chaining tests multiarch so they are exercised
> on all *-user targets.
Done. The explicit branch is gone with the move: a fall-through off
the end of a page is a cross-page goto_tb just the same, so the test
writes the last instruction of page A and the first of page B, and
needs no per-arch branch encoding or displacement arithmetic -- just
"set the return value" and "return".
13 architectures supply those two, and the rest skip. I checked each
encoding against the cross assembler and ran the test under qemu-user
on aarch64, alpha, arm, hppa, loongarch64, m68k, mips, ppc, ppc64le,
riscv64, s390x, sh4, sparc64 and x86_64 (ppc64 ELFv1 skips). The
gdbstub half moved with it.
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v3 7/7] RFC: tcg: fold a guest displacement into the host addressing mode
2026-08-25 22:52 ` Richard Henderson
@ 2026-08-27 4:57 ` Matt Turner
0 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-08-27 4:57 UTC (permalink / raw)
To: Richard Henderson; +Cc: qemu-devel, pbonzini, philmd, alex.bennee, zhao1.liu
On Tue, Aug 25, 2026 at 6:52 PM Richard Henderson
<richard.henderson@linaro.org> wrote:
>
> On 8/22/26 12:08, Matt Turner wrote:
> > RFC because:
> >
> > - Only wired up for x86_64, and only for qemu_ld and qemu_st; the i128
> > qemu_ld2 and qemu_st2 pairs are left alone.
> > - Requiring that no slow path exists is stricter than necessary. Recording
> > the displacement in TCGLabelQemuLdst and emitting one lea on the slow
> > path would cover alignment-checked accesses too, at no fast path cost.
> > - Softmmu wants the displacement folded into the TLB comparison as well,
> > which is a bigger change than this one.
> > - A one op window catches everything the frontends emit today but is
> > trivially defeated by anything scheduled in between.
>
> Plausible. Several times I've considered exposing complex addressing
> modes to the translators so that things that do LEA are kept intact for
> awhile. There are plenty of host-specific code sequences for x << s + y
> + c, even before we fold that into the memory access. Then expose the
> host memory path to tcg ops, somehow, and finally implement a simple CSE
> pass. But yeah, hand waving is as far as I've ever gone.
That's the shape I kept wanting. A one-op peephole is a poor
substitute, and it loses the moment anything is scheduled between the
add and the access. I went this way because it needed no frontend
changes; happy to look at the op instead if you'd rather.
> On the second point, I guess you also assuming the offset is also
> aligned? I.e. for X + 8*N, you can test X for 8-byte alignment without
> constructing the complete address.
Right, and that's what makes the second bullet worth doing. If disp is
a multiple of the required alignment, X + disp is aligned exactly when
X is, so the fast path test can stay on the base and never needs the
full address. That covers every displacement a frontend emits for a
struct or stack access.
What's left is the slow path, which hands addr_reg to the helper --
and addr_reg is now the base. Recording the displacement in
TCGLabelQemuLdst and emitting one lea there fixes it at no fast path
cost. v4 spells that out in the bullet but still refuses any access
needing a test.
> > +static bool tcg_target_ldst_disp_ok(TCGContext *s, MemOpIdx oi, int64_t disp)
> > +{
> > +#ifdef CONFIG_USER_ONLY
> > + MemOp opc = get_memop(oi);
> > + TCGAtomAlign aa;
> > + int64_t ofs;
> > +
> > + if (tcg_use_softmmu || s->addr_type != TCG_TYPE_I64) {
> > + return false;
> > + }
> > + aa = atom_and_align_for_opc(s, opc, MO_ATOM_WITHIN16,
> > + (opc & MO_SIZE) == MO_128);
> > + if (aa.align) {
> > + return false;
> > + }
> > + ofs = (int64_t)x86_guest_base.ofs + disp;
> > + return ofs == (int32_t)ofs;
> > +#else
> > + return false;
> > +#endif
> > +}
> These compilation mode tests belong...
>
> > +static void __attribute__((noinline))
> > +fold_ldst_disp(TCGContext *s)
> > +{
> > + TCGOp *op;
> > +
> > + if (!TCG_TARGET_HAS_ldst_disp) {
> > + return;
> > + }
> ... here, before we step over the loop. You might as well pass MemOp to
> the target function and not MemOpIdx -- nothing about the mmu_idx is
> relevant.
>
> Ideally, the atom_and_align test would also be done generically, not
> requiring each target to replicate that boilerplate.
All three done. x86_64 is now just:
ofs = (int64_t)x86_guest_base.ofs + disp;
return ofs == (int32_t)ofs;
The alignment one needed a compromise. atom_and_align_for_opc() takes
host_atom and allow_two_ops and both feed aa.align, so a generic
caller either gets them from the target -- the same boilerplate, moved
to tcg-target.h -- or answers without them. I answered without them: a
small ldst_disp_needs_align() that gives the answer for the most
restrictive
host.
Exact for MO_ATOM_NONE and the IFALIGN cases, i.e. everything
frontends emit by default, so the alpha numbers don't move.
Conservative for MO_ATOM_WITHIN16 and MO_ATOM_SUBALIGN: x86 could take
those folds and no longer does. One #define of the host atomicity next
to TCG_TARGET_HAS_ldst_disp gets that back if you'd rather have it
exact.
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v3 4/7] RFC: tcg: probe the TB jump cache inline instead of calling a helper
2026-08-25 22:28 ` Richard Henderson
@ 2026-08-27 5:00 ` Matt Turner
0 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-08-27 5:00 UTC (permalink / raw)
To: Richard Henderson; +Cc: qemu-devel, pbonzini, philmd, alex.bennee, zhao1.liu
On Tue, Aug 25, 2026 at 6:28 PM Richard Henderson
<richard.henderson@linaro.org> wrote:
>
> On 8/22/26 12:08, Matt Turner wrote:
> > Every indirect branch that cannot use goto_tb ends in
> > tcg_gen_lookup_and_goto_ptr(), which calls helper_lookup_tb_ptr(). For an
> > emulated compiler that is 8.4 billion helper calls in a single translation
> > unit: 24.6% of all TB exits take this path, because jsr/ret/jmp have a
> > register destination and because goto_tb is restricted to same-page
> > targets.
> >
> > The helper itself is already tight, but each call pays for a call frame,
> > the can_do_io store, the get_tb_cpu_state() indirect call through
> > TCGCPUOps, curr_cflags(), and a breakpoint check, before it gets to the
> > jump cache probe that almost always hits (95.8% for this workload).
> >
> > Emit the probe inline instead. The destination PC is already in a TCG
> > temp, and the flags, cflags and cs_base the destination must match are
> > constants at translation time, so the fast path is a hash, four guarded
> > loads and a goto_ptr. Only a miss calls the helper, which still owns
> > filling the cache.
> >
> > tcg_gen_lookup_and_goto_ptr() therefore takes the destination PC and the
> > TB being generated, and decides for itself whether to emit the probe or
> > the old helper call; there is no second entry point for targets that opt
> > in. A target that cannot name its destination in a single temp passes
> > NULL and gets the helper. Since the probe hashes and compares the PC as
> > one 64-bit value, a 32-bit guest PC also falls back.
> >
> > The PC a target passes must be exactly what get_tb_cpu_state() reports for
> > the destination, which is the whole of the contract. alpha, loongarch,
> > mips, ppc and s390x pass their PC register, whose value is that pc by
> > construction. The rest pass NULL for now: avr's TB pc is the word address
> > doubled, i386's is eip before segmentation, riscv masks it to 32 bits when
> > xl is MXL_RV32, hppa derives it from the IAQ, hexagon adjusts it inside a
> > hardware loop, and sparc puts npc in cs_base so the guard could not hit
> > anyway. Each of those is a one-line change for whoever wants to measure it.
> >
> > Two details matter for the generated code. The flags and cflags guards are
> > folded into a single aligned 64-bit load and compare, since the fields are
> > adjacent. And each path emits its own goto_ptr rather than branching to a
> > shared one: a temp live across the label is spilled and reloaded on every
> > dispatch, which cost 6.3% on its own.
> >
> > The probe cannot check everything the helper checks, and the one that
> > matters is breakpoints. check_for_breakpoints() raises EXCP_DEBUG on an
> > exact pc match and selects CF_BP_PAGE cflags for the rest of the page, and
> > setting a breakpoint deliberately invalidates no TB, so a block translated
> > before the breakpoint was set is still sitting in the jump cache. Rather
> > than pay for a breakpoint test on the fast path, give the probe its own
> > base pointer, tb_jmp_cache_probe, that nothing else reads, and point it at
> > a page of zeroes while any breakpoint is set. Every entry the probe finds
> > then has a NULL tb, so every dispatch misses into the helper and the old
> > behaviour is restored exactly. cpu_breakpoint_insert() poisons the pointer,
> > so the poison takes effect at the next dispatch rather than whenever that
> > vCPU next reaches its main loop, which matters because a vCPU chaining
> > indirectly need never reach it. The main loop puts the pointer back once the
> > last breakpoint is gone; that is a load and a compare per block dispatched
> > from the main loop, and nothing at all in generated code.
> >
> > The flags and cflags constants are safe against the other things that can
> > change them. CF_PARALLEL is only ever set by begin_parallel_context(),
> > which flushes first, so no block predating it survives to dispatch. gdb
> > single-step is only turned on with the CPU stopped, and a block translated
> > without CF_SINGLE_STEP can only be re-entered through tb_lookup(), which
> > from then on demands the new cflags -- so a stale-cflags block is never the
> > one running. What is left is one_insn_per_tb and -d nochain, which the
> > monitor can toggle under a running vCPU without a flush; see below.
> >
> > Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling
> > the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, LTO
> > build, on top of the preceding three patches:
> >
> > before: 1,402,667,803,616 instructions
> > after: 916,415,123,244 instructions -34.67%
> >
> > before: 115.56s wall clock
> > after: 85.59s wall clock -25.94%
> >
> > The gap between the two is the point at which this stops being a
> > straight-line win: the helper call was highly predictable work that the
> > host pipelined well, so removing it retires far fewer instructions than it
> > saves time. IPC falls from 2.48 to 2.17 across this patch for that reason.
> >
> > Despite emitting more code, this also reduces instruction cache pressure,
> > because a dispatch no longer jumps into qemu's .text and evicts translated
> > code:
> >
> > before: 11,735,141,703 L1-icache-load-misses
> > after: 7,154,863,292 L1-icache-load-misses -39.0%
> >
> > The mechanism is visible directly in a profile: helper_lookup_tb_ptr()
> > falls from 31.01% of samples to 0.35%, and qemu's own .text falls from
> > 38.8% to 5.3%, with the balance moving into generated code.
> >
> > Combined with the three preceding patches, against an unmodified LTO
> > build, 1,646,994,254,249 instructions fall to 916,415,123,244, or -44.36%.
> > The emulated compiler produces byte-identical output throughout.
> >
> > Open issues, hence RFC:
> >
> > - one_insn_per_tb and CPU_LOG_TB_NOCHAIN can be toggled from the monitor
> > while a vCPU is inside a block that was translated without them. The
> > block keeps dispatching inline with the old cflags until it exits for
> > some other reason. Poisoning the probe from
> > tcg_update_all_curr_cflags() would close it.
> > - The jump cache entry is read without qatomic_read(); entries are
> > invalidated concurrently by setting tb to NULL.
> > - Only alpha has been measured. The other four targets that pass a PC are
> > built and boot-tested only.
> >
> > v3: Fold the fast path into tcg_gen_lookup_and_goto_ptr() instead of
> > adding tcg_gen_lookup_and_goto_ptr_inline() beside it (Richard). It
> > now takes the destination PC and the TB unconditionally, from all 38
> > call sites, and picks the probe or the helper itself. Translators
> > built for both values of TARGET_LONG_BITS -- arm, s390x, microblaze --
> > cannot include tcg-op.h, so the common entry point takes a TCGTemp and
> > reads the width from it, and tcg-op.h wraps that for everyone else;
> > this is the same split as tcg_gen_qemu_ld_*_chk().
> >
> > Compare cs_base too. v2 listed this as an open issue, and closing it
> > is what lets the choice be made generically rather than per target: a
> > target that uses cs_base would otherwise have been enabled silently by
> > a decision keyed on PC width alone. It costs a load and a compare on
> > the fast path, and the numbers above were measured with it in place.
> >
> > Audited which targets may pass a real PC, the contract being that it
> > is exactly what get_tb_cpu_state() reports for the destination. Five
> > do; the rest pass NULL and keep the helper call, sparc among them
> > because it puts npc in cs_base and so could essentially never hit.
> >
> > Poison the probe from cpu_breakpoint_insert() rather than only from
> > the poisoned CPU's own main loop. gdb inserts a breakpoint into every
> > CPU (tcg_insert_gdbstub_breakpoint()), and a thread already inside
> > generated code, dispatching indirectly, need never return to the main
> > loop -- so it would keep dispatching inline and run past a breakpoint
> > another thread had just set. Upstream has no such window: its
> > helper_lookup_tb_ptr() sees the new breakpoint at the next indirect
> > branch. The un-poison in tcg_cpu_sync_jmp_cache() now re-checks after
> > its store, with a barrier, so that it loses the race with a concurrent
> > insert in the safe direction.
> >
> > Signed-off-by: Matt Turner <mattst88@gmail.com>
> > ---
> > accel/tcg/cpu-exec.c | 102 ++++++++++++++++++
> > accel/tcg/internal-common.h | 2 +
> > cpu-common.c | 11 ++
> > include/hw/core/cpu.h | 9 ++
> > include/system/tcg.h | 9 ++
> > include/tcg/tcg-op-common.h | 16 ++-
> > include/tcg/tcg-op.h | 12 +++
> > stubs/tcg-cflags.c | 8 +-
> > target/alpha/translate.c | 4 +-
> > target/arm/tcg/translate-a64.c | 4 +-
> > target/arm/tcg/translate.c | 10 +-
> > target/avr/translate.c | 4 +-
> > target/hexagon/translate.c | 4 +-
> > target/hppa/translate.c | 6 +-
> > target/i386/tcg/translate.c | 2 +-
> > .../tcg/insn_trans/trans_branch.c.inc | 2 +-
> > target/loongarch/tcg/translate.c | 4 +-
> > target/m68k/translate.c | 2 +-
> > target/microblaze/translate.c | 4 +-
> > target/mips/tcg/nanomips_translate.c.inc | 2 +-
> > target/mips/tcg/translate.c | 6 +-
> > target/or1k/translate.c | 4 +-
> > target/ppc/translate.c | 4 +-
> > target/riscv/tcg/insn_trans/trans_rvzce.c.inc | 4 +-
> > target/riscv/tcg/translate.c | 2 +-
> > target/rx/translate.c | 4 +-
> > target/s390x/tcg/translate.c | 5 +-
> > target/sh4/translate.c | 4 +-
> > target/sparc/translate.c | 4 +-
> > target/tricore/translate.c | 4 +-
> > tcg/tcg-op.c | 92 +++++++++++++++-
> > 31 files changed, 300 insertions(+), 50 deletions(-)
> >
> > diff --git ./accel/tcg/cpu-exec.c ./accel/tcg/cpu-exec.c
> > index 148e0f583e..e546f717e8 100644
> > --- ./accel/tcg/cpu-exec.c
> > +++ ./accel/tcg/cpu-exec.c
> > @@ -752,6 +752,99 @@ static inline bool cpu_handle_exception(CPUState *cpu, int *ret)
> > return false;
> > }
> >
> > +/*
> > + * The inline jump cache probe reads cpu->tb_jmp_cache_probe and takes the
> > + * slow path when the entry it finds has a NULL tb. Pointing the probe at a
> > + * region that is all zeroes therefore forces every indirect dispatch into
> > + * helper_lookup_tb_ptr(), which does the full lookup the inline probe only
> > + * approximates. The real jump cache is untouched, so no contents are lost
> > + * and recovery is a single store.
> > + *
> > + * Only ever read from, and only the tb field of one entry per dispatch, so
> > + * one shared zero-filled cache is enough for every CPU.
> > + */
> > +static const CPUJumpCache *tb_jmp_cache_poison(void)
> > +{
> > + static CPUJumpCache *poison;
> > +
> > + if (unlikely(poison == NULL)) {
> > + /* Raced allocations are harmless: both are all zeroes. */
> > + qatomic_cmpxchg(&poison, NULL, g_new0(CPUJumpCache, 1));
> > + }
> > + return poison;
> > +}
>
> Why allocate a jump cache at runtime? Surely this would simply be a
> block of 0's in .rodata with
>
> static const CPUJumpCache poison;
Changed to be a static object now, but not const. A zero-initialized
const aggregate goes in .rodata -- a megabyte of real zeroes in every
emulator binary. Without const it lands in .bss, costs nothing on
disk, and faults in only the pages a poisoned run touches. Nothing
writes to it; there's a comment saying why.
> I think this needs to be split into many pieces. In particular:
>
> (1) API change for tcg_gen_lookup_and_goto_ptr.
>
> (2) Introduce tb_jmp_cache_probe, and the poisoning.
>
> (3) Implementation of gen_jmp_cache_probe.
Done, in the order you gave.
> > @@ -28,6 +28,8 @@
> > #include "tcg/tcg-op-common.h"
> > #include "exec/translation-block.h"
> > #include "exec/plugin-gen.h"
> > +#include "hw/core/cpu.h"
> > +#include "../accel/tcg/tb-jmp-cache.h"
> > #include "tcg-internal.h"
> > #include "tcg-has.h"
> >
> > @@ -2715,7 +2717,81 @@ void tcg_gen_goto_tb(unsigned idx)
> > tcg_gen_op1i(INDEX_op_goto_tb, 0, idx);
> > }
> >
> > -void tcg_gen_lookup_and_goto_ptr(void)
> > +static void gen_jmp_cache_probe(TCGv_i64 pc, const TranslationBlock *tb)
> > +{
> > + TCGv_ptr jc, ent, tbp, ptr;
> > + TCGv_i64 h, tmp;
> > + TCGLabel *slow;
> > + uint64_t fpair;
> > +
> > + QEMU_BUILD_BUG_ON(sizeof(((CPUJumpCache *)0)->array[0]) != 16);
> > + QEMU_BUILD_BUG_ON(offsetof(TranslationBlock, cflags) !=
> > + offsetof(TranslationBlock, flags) + 4);
> > +
> > + jc = tcg_temp_ebb_new_ptr();
> > + ent = tcg_temp_ebb_new_ptr();
> > + tbp = tcg_temp_ebb_new_ptr();
> > + ptr = tcg_temp_ebb_new_ptr();
> > + h = tcg_temp_ebb_new_i64();
> > + tmp = tcg_temp_ebb_new_i64();
> > + slow = gen_new_label();
> > +
> > + /* h = tb_jmp_cache_hash_func(pc) * sizeof(array[0]) */
> > + tcg_gen_shri_i64(h, pc, TB_JMP_CACHE_BITS);
> > + tcg_gen_xor_i64(h, h, pc);
> > + tcg_gen_andi_i64(h, h, TB_JMP_CACHE_SIZE - 1);
> > + tcg_gen_shli_i64(h, h, 4);
>
> This hash function varies between system and user mode.
Fixed.
> > +
> > + /*
> > + * Not cpu->tb_jmp_cache: the probe reads its own base so that the main
> > + * loop can poison it, which is how conditions the probe cannot test for
> > + * itself force every dispatch back into the helper. See
> > + * tcg_cpu_sync_jmp_cache().
> > + */
> > + tcg_gen_ld_ptr(jc, tcg_env,
> > + offsetof(CPUState, tb_jmp_cache_probe) - sizeof(CPUState));
> > + tcg_gen_trunc_i64_ptr(ent, h);
> > + tcg_gen_add_ptr(ent, jc, ent);
> > +
> > + tcg_gen_ld_ptr(tbp, ent, offsetof(CPUJumpCache, array[0].tb));
> > + tcg_gen_brcondi_ptr(TCG_COND_EQ, tbp, 0, slow);
> > +
> > + tcg_gen_ld_i64(tmp, ent, offsetof(CPUJumpCache, array[0].pc));
> > + tcg_gen_brcond_i64(TCG_COND_NE, tmp, pc, slow);
>
> I think you can test pc first as on hash miss, that's most likely to differ.
Smart! Fixed.
>
> > +
> > + /*
> > + * flags and cflags are adjacent uint32_t, so one aligned 64-bit load
> > + * and compare covers both.
> > + */
> > +#if HOST_BIG_ENDIAN
> > + fpair = ((uint64_t)tb->flags << 32) | tb->cflags;
> > +#else
> > + fpair = ((uint64_t)tb->cflags << 32) | tb->flags;
> > +#endif
> > + tcg_gen_ld_i64(tmp, tbp, offsetof(TranslationBlock, flags));
> > + tcg_gen_brcondi_i64(TCG_COND_NE, tmp, fpair, slow);
>
> I'm not keen on this without an additional _Static_assert that the
> offset is aligned. It happens to be right now, but we're not currently
> relying on that. :-)
Added, next to the adjacency assert.
> > +
> > + /*
> > + * The destination must have been translated with the same cs_base, which
> > + * the pc alone does not imply on a target that uses it.
> > + */
> > + tcg_gen_ld_i64(tmp, tbp, offsetof(TranslationBlock, cs_base));
> > + tcg_gen_brcondi_i64(TCG_COND_NE, tmp, tb->cs_base, slow);
>
> The comment here should be more generic. Despite the cs_base name,
> consider this target-specific tb->flags2.
Done.
> > @@ -2724,7 +2800,21 @@ void tcg_gen_lookup_and_goto_ptr(void)
> > return;
> > }
> >
> > + /*
> > + * No icount_decr poll is needed for this exit: the helper is called on
> > + * every dispatch and returns to the main loop while an exit is pending.
> > + */
> > plugin_gen_disable_mem_helpers();
> > +
> > + /*
> > + * The inline probe hashes and compares the pc as a single 64-bit value.
> > + * A target with a 32-bit guest PC keeps the helper call.
> > + */
> > + if (pc && pc->type == TCG_TYPE_I64) {
> > + gen_jmp_cache_probe(temp_tcgv_i64(pc), tb);
> > + return;
> > + }
>
> Just extend, clearly.
Done. Nice, the high half folds to a compare against zero.
^ permalink raw reply [flat|nested] 47+ messages in thread
* [PATCH v4 0/9] accel/tcg: cut per-block dispatch overhead
2026-08-22 19:08 [PATCH v3 0/7] accel/tcg: cut per-block dispatch overhead Matt Turner
` (6 preceding siblings ...)
2026-08-22 19:08 ` [PATCH v3 7/7] RFC: tcg: fold a guest displacement into the host addressing mode Matt Turner
@ 2026-08-27 5:02 ` Matt Turner
2026-09-01 3:47 ` [PATCH v5 " Matt Turner
2026-08-27 5:02 ` [PATCH v4 1/9] accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags Matt Turner
` (8 subsequent siblings)
16 siblings, 1 reply; 47+ messages in thread
From: Matt Turner @ 2026-08-27 5:02 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
For guests running large amounts of code, most of what TCG executes is not
translated guest work but the fixed overhead around it. Blocks are short and
there are a great many of them, so the constant cost at each end of a block
(the interrupt poll and the can_do_io stores on entry, the dispatch on exit)
ends up dominating everything else.
The workload throughout is qemu-alpha running an emulated alpha gcc 16.2.0
compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64
host, in a --static --enable-lto --target-list=alpha-linux-user build. It
executes 34.2 billion TBs at 6.04 guest instructions each, and 24.6% of its
TB exits cannot use goto_tb. That is a representative shape for any guest
whose text is much larger than a page: indirect calls and returns
everywhere, plus direct branches that merely crossed a page boundary.
The first three patches are ordinary cleanups that stand on their own; patch
3 picked up review tags in v2. Patches 4 and 5 are preparation split out of
v3's patch 4 at Richard's request and move nothing on their own. The
remaining four are marked RFC individually and are where the interesting
questions are.
1 accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags
curr_cflags() recomputed three unlikely tests on every one of the run's
8.4 billion dispatches, from state that changes only when gdb enables
single-step, when one-insn-per-tb is toggled, or when the log mask
moves. Fold each into tcg_cflags where it changes. -5.15%
2 accel/tcg: enlarge the TB jump cache to 64K entries
4096 entries is too small for a guest running a large program;
tb_htable_lookup() is 6.10% of samples. 16 bits is the knee of the
sizing curve, at 1 MiB per CPUState. -5.92%, -8.71% wall
3 accel/tcg: skip the can_do_io stores in user-only builds
Two stores per TB that nothing in a user-only build reads: 68 billion
of them over the run. -4.57%, -4.53% wall
4 tcg: pass the destination to tcg_gen_lookup_and_goto_ptr()
Preparation. The destination PC is already in a TCG temp at every one
of the 38 call sites; give the helper wrapper the option of taking it
rather than discarding it. Five targets pass it; the rest pass NULL
and keep today's behavior.
5 accel/tcg: give the TB jump cache a second base pointer for generated
code
Preparation. CPUState::tb_jmp_cache_probe is a base pointer only
generated code reads. Pointing it at a shared zero-filled cache makes
every lookup through it miss, which is how the conditions an inline
probe cannot check force it back into the helper. Nothing reads it
yet.
6 RFC: tcg: probe the TB jump cache inline instead of calling a helper
95.8% of those 8.4 billion helper_lookup_tb_ptr() calls hit the jump
cache. Emit the probe inline (hash, four guarded loads, goto_ptr) and
call the helper only on a miss. -34.67%, -25.94% wall
7 RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds
translator_use_goto_tb() refuses to chain across a page. In user-only
builds the invalidation path already covers what that was protecting
against: every mmap/mprotect/munmap reaches page_set_flags(), which
invalidates and unlinks. Lift it there for runs that can never acquire
a breakpoint, keep it for system mode. -2.75%, -4.84% wall
8 RFC: accel/tcg: poison the jump cache instead of polling for indirect
exits
A block only needs the icount_decr poll if it can leave by goto_tb;
every other exit already passes through a dispatch. Poison the probe
pointer from patch 5 when an exit is requested, so every dispatch
misses into the helper, which returns the epilogue. Emit the poll only
in blocks that emitted a goto_tb. -2.52%, -1.96% wall
9 RFC: tcg: fold a guest displacement into the host addressing mode
tcg_gen_qemu_ld/st cannot express a based access, so a target with a
displacement in its encodings materializes the address with an lea
that the host addressing mode would have done for free. Fold a
preceding constant add into a new argument on the op, opt-in per
backend, wired up for x86_64 user-only. -5.70%, -3.20% wall
Each percentage is against the patch before it. Every stage was measured in
one session on the same host, so end to end, from an unmodified LTO build of
the same base to the full series:
instructions retired: 1,646,994,254,249 -> 819,262,147,022 -50.26%
wall clock: 133.19s -> 77.30s -41.96%
Those are the v3 measurements. The machine they were taken on is busy, so v4
has not been re-measured. Nothing in the v4 changes is expected to move them
-- the splits are pure reorganization, the new alignment test in patch 9
reaches the same answer for everything the alpha frontend emits, and the
emulated compiler's output is unchanged -- but they are not a measurement of
this posting and should not be read as one.
The two figures do not track each other, and that is the interesting part:
what the series removes is cheap, well-predicted, highly pipelined work, so
it retires far more instructions than it saves time. Patch 6 also cuts L1
icache load misses by 39.0%, because a dispatch no longer jumps into qemu's
.text and evicts translated code; qemu's own .text falls from 38.8% to 5.3%
of profile samples.
Every step builds and runs on its own, so the series bisects, and the
emulated compiler produces byte-identical assembly output at every step,
which is the correctness check these patches most need. Three new tests
cover the hazards the series creates, all under tests/tcg/multiarch as of
this revision: test-xpage-chain.c and gdbstub/xpage-bp.py (patch 7) and
test-indirect-irq.c (patch 8). Each fails or hangs if the mechanism it
covers is removed, which is what makes them tests of the new behavior rather
than of the old.
Changes since v3
================
The biggest change is that v3's patch 4 is split three ways, as Richard
asked: the tcg_gen_lookup_and_goto_ptr() API change is now patch 4, the
tb_jmp_cache_probe base pointer and its poison are patch 5, and the inline
probe itself is patch 6. The other structural change is that both guest
tests move from tests/tcg/alpha to tests/tcg/multiarch, as Alex asked, so
every *-user target runs them. tests/tcg/alpha is now byte-identical to
master again.
1 Update the cflags from the HMP handlers for 'log' and
'one-insn-per-tb' rather than from qemu_set_log_internal() and the
accelerator property setter; those are the paths that reach a running
vCPU, and the monitor is the only thing that does (Richard). Queue the
per-CPU update with async_run_on_cpu() rather than
async_safe_run_on_cpu(): halting the other vCPUs buys nothing, since
the queued work already runs on the owning CPU's own thread (Alex).
Alex also asked whether there are cross-vCPU updates of tcg_cflags at
all; with this change the monitor path has none, and the only
remaining writer from another thread is cpu_single_step(), which is
neither new nor made worse here. Stub moved to accel/stubs/, where the
other accelerator stubs live (Philippe).
2 Commit message only: a linux-user process is not a single vCPU. The
cache is per CPUState and linux-user creates one per guest thread, so
a threaded guest pays the 1 MiB per thread, exactly as system
emulation pays it per vCPU (Richard).
3 Unchanged.
4 New, split out of v3's patch 4. No functional change from v3. Its
commit message no longer claims most targets can simply pass a PC:
five do, six cannot because their TB pc is derived (avr doubles it,
i386's is pre-segmentation, riscv masks it, hppa derives it from the
IAQ, hexagon adjusts it in hardware loops, sparc puts npc in cs_base),
and seven look like they could but are untested.
5 Also new, split out of v3's patch 4. The poison cache is a static
object rather than one allocated on first use (Richard, who asked for
const; the commit message says why it is plain static and lands in
.bss).
6 What remains of v3's patch 4. Emit the softmmu form of
tb_jmp_cache_hash_func() under CONFIG_SOFTMMU rather than the
user-only form everywhere: v3 was wrong for system mode, and only not
a correctness bug because a wrong index simply misses (Richard).
Compare the pc before testing tb for NULL, assert that
TranslationBlock::flags is 8-byte aligned since folding the two guards
into one 64-bit load relies on it, zero-extend a 32-bit guest PC
instead of falling back to the helper, and describe cs_base in the
probe as a second word of target-specific flags rather than by name
(all Richard).
7 Test moved to tests/tcg/multiarch (Alex). The hand-written branch went
with it: falling off the end of a page is a cross-page goto_tb just
the same, and needs no per-architecture branch encoding or
displacement arithmetic, only "set the return value" and "return".
Built and run under qemu-user on aarch64, alpha, arm, hppa,
loongarch64, m68k, mips, ppc, ppc64le, riscv64, s390x, sh4, sparc64
and x86_64; ppc64 ELFv1 skips, because a function pointer there is a
descriptor rather than a code address.
8 Test likewise moved to tests/tcg/multiarch (Alex). Nothing in it is
architecture specific: the loop is a computed goto, which every
target's compiler supports, so it covers whichever targets go on to
use the inline probe.
9 Hoist the compilation mode tests -- tcg_use_softmmu and the 64-bit
address type -- out of the backend hook into fold_ldst_disp(), so the
loop is not entered at all when the mode rules the fold out. Pass
MemOp rather than MemOpIdx to the hook; nothing about the mmu_idx is
relevant to it. Move the alignment test into generic code as
ldst_disp_needs_align(), so a backend need not repeat the
atom_and_align_for_opc() call; the exact answer depends on the host's
atomicity capabilities, which the generic pass does not know, so it
answers for the most restrictive host. That is the same answer for
everything the frontends actually emit, and conservative for the
handful of MO_ATOM_WITHIN16 and MO_ATOM_SUBALIGN accesses. (All
Richard.) What is left of the x86_64 hook is the guest_base test, so
it now lives beside x86_guest_base under the CONFIG_USER_ONLY that
declares it. Also refuse a displacement that does not fit the int32_t
out_disp() takes, which is unreachable with any real guest_base but
which the interface could not have carried.
What I would most like reviewed
===============================
- Patch 7 reverses a deliberate decision made in d3a2a1d803 on the
strength of an argument about the user-only invalidation paths, plus a
gate on whether gdb can ever attach.
- Patch 8's un-poison in the main loop races a concurrent poison from
another thread. I believe the existing barrier around
icount_decr.u16.high covers it, but my testing was single-threaded user
mode.
- Patch 6 treats cpu flags, cflags and cs_base as translation-time
constants in its guards, reads a jump cache entry without qatomic_read(),
and leaves one_insn_per_tb and -d nochain toggles visible only at the
next non-inline exit.
- Patch 9 only examines the immediately preceding op, refuses any access
with a slow path (so user-only, and no alignment check), and leaves the
i128 pairs alone. Richard asked whether the alignment test could stay on
the base register when the displacement is itself aligned; it can, and
the reason the fold is still refused there is the slow path handing
addr_reg to the helper. Recording the displacement in TCGLabelQemuLdst
and emitting one lea on the slow path would cover alignment-checked
accesses too, at no fast path cost. Not attempted here.
- Patch 2's 1 MiB per CPUState is easy to justify for a single-threaded
linux-user process and less obvious for system emulation with many
vCPUs, or for a heavily threaded guest. It may want to be sized per
target or made tunable rather than raised unconditionally.
Patches 6 and 9 are wired up for alpha and x86_64 respectively; everything
else is target-independent, and no other backend changes behavior or needs
touching.
v3: https://lore.kernel.org/qemu-devel/20260822190818.1829249-1-mattst88@gmail.com/
v2: https://lore.kernel.org/qemu-devel/20260817190038.580257-1-mattst88@gmail.com/
Matt Turner (9):
accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags
accel/tcg: enlarge the TB jump cache to 64K entries
accel/tcg: skip the can_do_io stores in user-only builds
tcg: pass the destination to tcg_gen_lookup_and_goto_ptr()
accel/tcg: give the TB jump cache a second base pointer for generated
code
RFC: tcg: probe the TB jump cache inline instead of calling a helper
RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds
RFC: accel/tcg: poison the jump cache instead of polling for indirect
exits
RFC: tcg: fold a guest displacement into the host addressing mode
accel/stubs/meson.build | 1 +
accel/stubs/tcg-stub.c | 20 ++
accel/tcg/cpu-exec-common.c | 33 +-
accel/tcg/cpu-exec.c | 116 ++++++
accel/tcg/internal-common.h | 13 +-
accel/tcg/tb-jmp-cache.h | 2 +-
accel/tcg/tcg-accel-ops.c | 1 +
accel/tcg/translator.c | 94 ++++-
cpu-common.c | 11 +
cpu-target.c | 3 +
gdbstub/user.c | 14 +
include/gdbstub/user.h | 11 +
include/hw/core/cpu.h | 11 +
include/system/tcg.h | 21 ++
include/tcg/tcg-op-common.h | 15 +-
include/tcg/tcg-op.h | 12 +
include/tcg/tcg-opc.h | 9 +-
include/tcg/tcg.h | 2 +
monitor/hmp-cmds.c | 5 +
system/runstate-hmp-cmds.c | 4 +
target/alpha/translate.c | 4 +-
target/arm/tcg/translate-a64.c | 4 +-
target/arm/tcg/translate.c | 10 +-
target/avr/translate.c | 4 +-
target/hexagon/translate.c | 4 +-
target/hppa/translate.c | 6 +-
target/i386/tcg/translate.c | 2 +-
.../tcg/insn_trans/trans_branch.c.inc | 2 +-
target/loongarch/tcg/translate.c | 4 +-
target/m68k/translate.c | 2 +-
target/microblaze/translate.c | 4 +-
target/mips/tcg/nanomips_translate.c.inc | 2 +-
target/mips/tcg/translate.c | 6 +-
target/or1k/translate.c | 4 +-
target/ppc/translate.c | 4 +-
target/riscv/tcg/insn_trans/trans_rvzce.c.inc | 4 +-
target/riscv/tcg/translate.c | 2 +-
target/rx/translate.c | 4 +-
target/s390x/tcg/translate.c | 5 +-
target/sh4/translate.c | 4 +-
target/sparc/translate.c | 4 +-
target/tricore/translate.c | 4 +-
tcg/tcg-op-ldst.c | 3 +-
tcg/tcg-op.c | 142 +++++++-
tcg/tcg.c | 132 ++++++-
tcg/x86_64/tcg-target.c.inc | 41 +++
tcg/x86_64/tcg-target.h | 3 +
tests/tcg/multiarch/Makefile.target | 12 +-
tests/tcg/multiarch/gdbstub/xpage-bp.py | 37 ++
tests/tcg/multiarch/test-indirect-irq.c | 62 ++++
tests/tcg/multiarch/test-xpage-chain.c | 336 ++++++++++++++++++
51 files changed, 1192 insertions(+), 63 deletions(-)
create mode 100644 accel/stubs/tcg-stub.c
create mode 100644 tests/tcg/multiarch/gdbstub/xpage-bp.py
create mode 100644 tests/tcg/multiarch/test-indirect-irq.c
create mode 100644 tests/tcg/multiarch/test-xpage-chain.c
base-commit: eea8fe61b8be8f3016e522e6af24924a0266ca95
--
2.54.0
^ permalink raw reply [flat|nested] 47+ messages in thread
* [PATCH v4 1/9] accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags
2026-08-22 19:08 [PATCH v3 0/7] accel/tcg: cut per-block dispatch overhead Matt Turner
` (7 preceding siblings ...)
2026-08-27 5:02 ` [PATCH v4 0/9] accel/tcg: cut per-block dispatch overhead Matt Turner
@ 2026-08-27 5:02 ` Matt Turner
2026-08-27 18:51 ` Richard Henderson
2026-08-27 5:02 ` [PATCH v4 2/9] accel/tcg: enlarge the TB jump cache to 64K entries Matt Turner
` (7 subsequent siblings)
16 siblings, 1 reply; 47+ messages in thread
From: Matt Turner @ 2026-08-27 5:02 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
curr_cflags() is called once per TB dispatch, from helper_lookup_tb_ptr()
and from the cpu_exec() loop. It recomputes the same value every time:
uint32_t cflags = cpu->tcg_cflags;
if (unlikely(cpu_single_stepping(cpu))) { ... }
else if (qatomic_read(&one_insn_per_tb)) { ... }
else if (qemu_loglevel_mask(CPU_LOG_TB_NOCHAIN)) { ... }
That is three loads and three branches on the hottest path in the
interpreter, for state that changes only when gdb enables single-step,
when one-insn-per-tb is toggled, or when the log mask changes.
None of the three has to be sampled at dispatch time. Fold each into
CPUState::tcg_cflags where it changes and curr_cflags() becomes a single
load of a field that TB lookup has to read anyway.
The derived bits -- CF_COUNT_MASK, CF_NO_GOTO_TB, CF_NO_GOTO_PTR and
CF_SINGLE_STEP -- are never set by tcg_cflags_set(), so tcg_update_cflags()
can recompute them in place without disturbing the rest, and conversely
tcg_cflags_set() ORs in its bits without disturbing them.
There are three places to call it:
- tcg_exec_realizefn(), so that a CPU created after the command line has
been parsed starts out with the right value. This covers user-only,
where tcg_cpu_init_cflags() is not reached. linux-user's cpu_copy()
copies tcg_cflags wholesale, so a cloned thread inherits it.
- cpu_single_step(), which changes one CPU. gdb is the only caller that
matters; in system mode it runs with the vCPUs stopped, and in user mode
gdb_continue_partial() can reach a thread that is still running, because
gdb_handlesig() stops only the thread that trapped. That is exactly the
plain cross-thread store to another CPU's CPUState that
cpu->singlestep_flags already was, read back by that CPU through
cpu_single_stepping() in curr_cflags(). This patch changes which field
carries it, not who writes it or how.
- hmp_one_insn_per_tb() and hmp_log(), which change every CPU while the
vCPUs are running, so the update is queued with async_run_on_cpu() and
each CPU writes its own cflags from its own thread. The command line
spellings of those two settings need nothing: they are parsed before
any CPU is realized, so tcg_exec_realizefn() picks them up.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling
the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, in a
build configured with --enable-lto:
before: 1,646,994,254,249 instructions
after: 1,562,204,796,597 instructions -5.15%
That workload issues 8.4 billion dispatches, so the per-call saving is
small but the aggregate is not. The emulated compiler produces
byte-identical output before and after.
Wall clock does not move: 133.19s to 132.58s, a 0.46% difference against a
run-to-run spread larger than that. The removed work is a few predictable
loads and branches that the host executes largely in parallel with the
surrounding dispatch, so this patch is worth taking for the instruction
count and for what it enables, not for a time saving that can be measured
on its own.
v4: Update the cflags from the HMP handlers for 'log' and 'one-insn-per-tb'
rather than from qemu_set_log_internal() and the accelerator property
setter. Those are the paths that reach a running vCPU, and the monitor
is the only thing that does. Suggested by Richard Henderson.
v4: Queue the per-CPU update with async_run_on_cpu() rather than
async_safe_run_on_cpu(). Halting the other vCPUs buys nothing: the
queued work already runs on the owning CPU's own thread. Suggested by
Alex Bennee, who also asked whether there are cross-vCPU updates of
tcg_cflags at all. With this change the monitor path has none: the
only remaining writer from another thread is cpu_single_step(), above,
which is neither new nor made worse here.
v4: Move the stub to accel/stubs/, which is where the other accelerator
stubs live.
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
accel/stubs/meson.build | 1 +
accel/stubs/tcg-stub.c | 16 ++++++++++++++++
accel/tcg/cpu-exec-common.c | 33 ++++++++++++++++++++++++++++++---
accel/tcg/cpu-exec.c | 3 +++
accel/tcg/internal-common.h | 11 +++++++++--
cpu-target.c | 3 +++
include/system/tcg.h | 12 ++++++++++++
monitor/hmp-cmds.c | 5 +++++
system/runstate-hmp-cmds.c | 4 ++++
9 files changed, 83 insertions(+), 5 deletions(-)
create mode 100644 accel/stubs/tcg-stub.c
diff --git ./accel/stubs/meson.build ./accel/stubs/meson.build
index 7c6d7ad943..ccad583e64 100644
--- ./accel/stubs/meson.build
+++ ./accel/stubs/meson.build
@@ -4,6 +4,7 @@ stub_ss.add(files(
'nitro-stub.c',
'mshv-stub.c',
'nvmm-stub.c',
+ 'tcg-stub.c',
'whpx-stub.c',
'xen-stub.c',
))
diff --git ./accel/stubs/tcg-stub.c ./accel/stubs/tcg-stub.c
new file mode 100644
index 0000000000..f9e1bd22d6
--- /dev/null
+++ ./accel/stubs/tcg-stub.c
@@ -0,0 +1,16 @@
+/*
+ * Stubs for the TCG entry points in system/tcg.h, for binaries that link
+ * cpu-target.c or the HMP command handlers but not TCG.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+#include "qemu/osdep.h"
+#include "system/tcg.h"
+
+void tcg_update_cflags(CPUState *cpu)
+{
+}
+
+void tcg_update_all_cflags(void)
+{
+}
diff --git ./accel/tcg/cpu-exec-common.c ./accel/tcg/cpu-exec-common.c
index 44e84344f3..9f3517f36b 100644
--- ./accel/tcg/cpu-exec-common.c
+++ ./accel/tcg/cpu-exec-common.c
@@ -36,9 +36,16 @@ void tcg_cflags_set(CPUState *cpu, uint32_t flags)
cpu->tcg_cflags |= flags;
}
-uint32_t curr_cflags(CPUState *cpu)
+/*
+ * The bits of CPUState::tcg_cflags that tcg_cflags_set() never sets, because
+ * they are derived from gdb single-step, one-insn-per-tb and -d nochain.
+ */
+#define CF_DERIVED (CF_COUNT_MASK | CF_NO_GOTO_TB | CF_NO_GOTO_PTR | \
+ CF_SINGLE_STEP)
+
+void tcg_update_cflags(CPUState *cpu)
{
- uint32_t cflags = cpu->tcg_cflags;
+ uint32_t cflags = cpu->tcg_cflags & ~CF_DERIVED;
/*
* Record gdb single-step. We should be exiting the TB by raising
@@ -55,7 +62,27 @@ uint32_t curr_cflags(CPUState *cpu)
cflags |= CF_NO_GOTO_TB;
}
- return cflags;
+ cpu->tcg_cflags = cflags;
+}
+
+static void tcg_update_cflags_work(CPUState *cpu, run_on_cpu_data data)
+{
+ tcg_update_cflags(cpu);
+}
+
+void tcg_update_all_cflags(void)
+{
+ CPUState *cpu;
+
+ /*
+ * one-insn-per-tb and -d nochain can both be changed from the monitor
+ * while the vCPUs are running. Queue the update onto each CPU rather
+ * than writing tcg_cflags from here, so that the field is only ever
+ * written by the CPU that owns it.
+ */
+ CPU_FOREACH(cpu) {
+ async_run_on_cpu(cpu, tcg_update_cflags_work, RUN_ON_CPU_NULL);
+ }
}
/* exit the current TB, but without causing any exception to be raised */
diff --git ./accel/tcg/cpu-exec.c ./accel/tcg/cpu-exec.c
index 257211235d..148e0f583e 100644
--- ./accel/tcg/cpu-exec.c
+++ ./accel/tcg/cpu-exec.c
@@ -1068,6 +1068,9 @@ bool tcg_exec_realizefn(CPUState *cpu, Error **errp)
tcg_target_initialized = true;
}
+ /* Pick up one-insn-per-tb and -d nochain from the command line. */
+ tcg_update_cflags(cpu);
+
cpu->tb_jmp_cache = g_new0(CPUJumpCache, 1);
tlb_init(cpu);
#ifndef CONFIG_USER_ONLY
diff --git ./accel/tcg/internal-common.h ./accel/tcg/internal-common.h
index 9e7be2d78d..853d1b51ee 100644
--- ./accel/tcg/internal-common.h
+++ ./accel/tcg/internal-common.h
@@ -69,8 +69,15 @@ void tlb_destroy(CPUState *cpu);
bool tcg_exec_realizefn(CPUState *cpu, Error **errp);
void tcg_exec_unrealizefn(CPUState *cpu);
-/* current cflags for hashing/comparison */
-uint32_t curr_cflags(CPUState *cpu);
+/*
+ * Current cflags for hashing/comparison. Everything that feeds into the
+ * value is folded into CPUState::tcg_cflags when it changes, by
+ * tcg_update_cflags(), so that TB dispatch only has to load it.
+ */
+static inline uint32_t curr_cflags(CPUState *cpu)
+{
+ return cpu->tcg_cflags;
+}
void tb_check_watchpoint(CPUState *cpu, uintptr_t retaddr);
diff --git ./cpu-target.c ./cpu-target.c
index 4783845c9b..50be591acf 100644
--- ./cpu-target.c
+++ ./cpu-target.c
@@ -24,6 +24,7 @@
#include "exec/replay-core.h"
#include "exec/log.h"
#include "hw/core/cpu.h"
+#include "system/tcg.h"
#include "trace/trace-root.h"
/* enable or disable single step mode. EXCP_DEBUG is returned by the
@@ -35,6 +36,8 @@ void cpu_single_step(CPUState *cpu, unsigned flags)
cpu->singlestep_flags, flags);
cpu->singlestep_flags = flags;
+ tcg_update_cflags(cpu);
+
#if !defined(CONFIG_USER_ONLY)
const AccelOpsClass *ops = cpus_get_accel();
if (ops->update_guest_debug) {
diff --git ./include/system/tcg.h ./include/system/tcg.h
index 7622dcea30..2c2dbc753b 100644
--- ./include/system/tcg.h
+++ ./include/system/tcg.h
@@ -17,6 +17,18 @@ extern bool tcg_allowed;
#define tcg_enabled() 0
#endif
+/*
+ * Recompute the parts of CPUState::tcg_cflags that TB dispatch consumes but
+ * tcg_cflags_set() does not provide: gdb single-step, one-insn-per-tb and
+ * the CPU_LOG_TB_NOCHAIN log flag. Call whenever one of those changes.
+ *
+ * tcg_update_cflags() updates one CPU and must be called from that CPU's
+ * thread, or with it stopped. tcg_update_all_cflags() updates every CPU
+ * and is safe to call from the monitor while the vCPUs run.
+ */
+void tcg_update_cflags(CPUState *cpu);
+void tcg_update_all_cflags(void);
+
/**
* qemu_tcg_mttcg_enabled:
* Check whether we are running MultiThread TCG or not.
diff --git ./monitor/hmp-cmds.c ./monitor/hmp-cmds.c
index 4e8d996dbb..b83551ea54 100644
--- ./monitor/hmp-cmds.c
+++ ./monitor/hmp-cmds.c
@@ -39,6 +39,7 @@
#include "system/hw_accel.h"
#include "system/memory.h"
#include "system/system.h"
+#include "system/tcg.h"
#include "disas/disas.h"
/* Please update hmp-commands.hx when adding or changing commands */
@@ -335,7 +336,11 @@ void hmp_log(Monitor *mon, const QDict *qdict)
if (!qemu_set_log(mask, &err)) {
error_report_err(err);
+ return;
}
+
+ /* CPU_LOG_TB_NOCHAIN feeds into the per-CPU cflags. */
+ tcg_update_all_cflags();
}
void hmp_gdbserver(Monitor *mon, const QDict *qdict)
diff --git ./system/runstate-hmp-cmds.c ./system/runstate-hmp-cmds.c
index 02d1d42bf3..86754a37f8 100644
--- ./system/runstate-hmp-cmds.c
+++ ./system/runstate-hmp-cmds.c
@@ -22,6 +22,7 @@
#include "qapi/qapi-commands-run-state.h"
#include "qobject/qdict.h"
#include "qemu/accel.h"
+#include "system/tcg.h"
void hmp_info_status(Monitor *mon, const QDict *qdict)
{
@@ -64,6 +65,9 @@ void hmp_one_insn_per_tb(Monitor *mon, const QDict *qdict)
/* If the property exists then setting it can never fail */
object_property_set_bool(OBJECT(accel), "one-insn-per-tb",
newval, &error_abort);
+
+ /* one-insn-per-tb feeds into the per-CPU cflags. */
+ tcg_update_all_cflags();
}
void hmp_watchdog_action(Monitor *mon, const QDict *qdict)
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH v4 2/9] accel/tcg: enlarge the TB jump cache to 64K entries
2026-08-22 19:08 [PATCH v3 0/7] accel/tcg: cut per-block dispatch overhead Matt Turner
` (8 preceding siblings ...)
2026-08-27 5:02 ` [PATCH v4 1/9] accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags Matt Turner
@ 2026-08-27 5:02 ` Matt Turner
2026-08-27 5:02 ` [PATCH v4 3/9] accel/tcg: skip the can_do_io stores in user-only builds Matt Turner
` (6 subsequent siblings)
16 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-08-27 5:02 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
The per-CPU TB jump cache has held 4096 entries since it was introduced.
That is too small for guests running large programs: an emulated compiler
misses often enough that the fallback qht lookup shows up prominently in
a profile.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling
the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host. The
compile performs 34.2 billion TB executions, of which 8.4 billion take
the indirect dispatch path.
Sizing curve, on top of the preceding patch, instructions retired and
wall clock:
12 bits ( 64 KiB): 1,562,204,796,597 132.58s
14 bits ( 256 KiB): 1,493,318,515,396 -4.41% 124.67s -5.97%
16 bits ( 1 MiB): 1,469,772,951,575 -5.92% 121.04s -8.71%
18 bits ( 4 MiB): 1,462,309,832,762 -6.39% 119.82s -9.62%
16 bits is the knee. 18 buys another 0.47% of instructions for four times
the memory. It does show a further 1.01% of wall clock, which is outside
the 0.70% run-to-run spread at 16 bits, so the effect is probably real --
but paying four times the memory for it is a poor trade, and instructions
retired does not account for the data cache pressure of a 4 MiB table.
In a perf profile the mechanism is visible directly: tb_htable_lookup(),
which is where qht_lookup_custom() lands once it is inlined in an LTO
build, falls from 6.10% of samples to 1.66%.
The cost is memory: the cache grows from 64 KiB to 1 MiB, once per
CPUState. In linux-user that is per guest thread rather than per process,
so a threaded guest pays it as many times as it has threads, exactly as
system emulation pays it per vCPU. The allocation is g_new0(), so the
pages are faulted in as the cache is touched and a thread that runs a
small amount of code touches a small part of it, but the address space is
committed either way.
So this may still want to be tunable, or scaled from the number of CPUs,
rather than raised unconditionally. I do not have a threaded workload where
the smaller cache is the better trade, and would welcome one.
v4: Fix the claim that a linux-user process is a single vCPU. The cache is
per CPUState, and linux-user creates one per guest thread. Pointed out
by Richard Henderson.
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
accel/tcg/tb-jmp-cache.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git ./accel/tcg/tb-jmp-cache.h ./accel/tcg/tb-jmp-cache.h
index c3a505e394..268dacd7ba 100644
--- ./accel/tcg/tb-jmp-cache.h
+++ ./accel/tcg/tb-jmp-cache.h
@@ -12,7 +12,7 @@
#include "qemu/rcu.h"
#include "exec/cpu-common.h"
-#define TB_JMP_CACHE_BITS 12
+#define TB_JMP_CACHE_BITS 16
#define TB_JMP_CACHE_SIZE (1 << TB_JMP_CACHE_BITS)
/*
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH v4 3/9] accel/tcg: skip the can_do_io stores in user-only builds
2026-08-22 19:08 [PATCH v3 0/7] accel/tcg: cut per-block dispatch overhead Matt Turner
` (9 preceding siblings ...)
2026-08-27 5:02 ` [PATCH v4 2/9] accel/tcg: enlarge the TB jump cache to 64K entries Matt Turner
@ 2026-08-27 5:02 ` Matt Turner
2026-08-27 5:02 ` [PATCH v4 4/9] tcg: pass the destination to tcg_gen_lookup_and_goto_ptr() Matt Turner
` (5 subsequent siblings)
16 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-08-27 5:02 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
Every translation block stores to cpu->neg.can_do_io twice: false before
the first instruction, true before the last one. Nothing reads it in a
user-only build. There is no memory-mapped I/O in linux-user, and every
reader is in system_ss: cputlb.c, watchpoint.c, icount-common.c and
tcg-accel-ops-icount.c.
Two stores per TB is not much on its own, but TBs are short. An emulated
alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines,
-O2) executes 34.2 billion TBs at 6.04 guest instructions each, so this is
68 billion stores for nothing.
Measured on an x86-64 host, LTO build, on top of the preceding two
patches:
before: 1,469,772,951,575 instructions
after: 1,402,667,803,616 instructions -4.57%
before: 121.04s wall clock
after: 115.56s wall clock -4.53%
The emulated compiler produces byte-identical output.
v3: Use #ifndef CONFIG_USER_ONLY again rather than
if (IS_ENABLED(CONFIG_USER_ONLY)). QEMU's IS_ENABLED() is IS_EMPTY(),
which is only true for a symbol Meson defines empty; CONFIG_USER_ONLY
is defined as 1, so the test was always false and v2 emitted the two
stores after all. The measurements above are from the working form.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
accel/tcg/translator.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git ./accel/tcg/translator.c ./accel/tcg/translator.c
index 57daded60f..6c8fcd7a20 100644
--- ./accel/tcg/translator.c
+++ ./accel/tcg/translator.c
@@ -21,12 +21,14 @@
#include "disas/disas.h"
#include "tb-internal.h"
+#ifndef CONFIG_USER_ONLY
static void set_can_do_io(DisasContextBase *db, bool val)
{
QEMU_BUILD_BUG_ON(sizeof_field(CPUState, neg.can_do_io) != 1);
tcg_gen_st8_i32(tcg_constant_i32(val), tcg_env,
offsetof(CPUState, neg.can_do_io) - sizeof(CPUState));
}
+#endif
bool translator_io_start(DisasContextBase *db)
{
@@ -210,17 +212,25 @@ void translator_loop(CPUState *cpu, TranslationBlock *tb, int *max_insns,
/*
* Manage can_do_io for the translation block: set to false before
* the first insn and set to true before the last insn.
+ *
+ * Nothing reads can_do_io in user-only builds. There is no MMIO
+ * there, and every reader (cputlb.c, watchpoint.c, icount) is in
+ * system_ss, so skip the two stores per TB entirely.
*/
if (db->num_insns == 1) {
tcg_debug_assert(first_insn_start == db->insn_start);
} else {
tcg_debug_assert(first_insn_start != db->insn_start);
+#ifndef CONFIG_USER_ONLY
tcg_ctx->emit_before_op = first_insn_start;
set_can_do_io(db, false);
+#endif
}
+#ifndef CONFIG_USER_ONLY
tcg_ctx->emit_before_op = db->insn_start;
set_can_do_io(db, true);
tcg_ctx->emit_before_op = NULL;
+#endif
/* May be used by disas_log or plugin callbacks. */
tb->size = db->pc_next - db->pc_first;
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH v4 4/9] tcg: pass the destination to tcg_gen_lookup_and_goto_ptr()
2026-08-22 19:08 [PATCH v3 0/7] accel/tcg: cut per-block dispatch overhead Matt Turner
` (10 preceding siblings ...)
2026-08-27 5:02 ` [PATCH v4 3/9] accel/tcg: skip the can_do_io stores in user-only builds Matt Turner
@ 2026-08-27 5:02 ` Matt Turner
2026-08-27 23:12 ` Richard Henderson
2026-08-27 5:02 ` [PATCH v4 5/9] accel/tcg: give the TB jump cache a second base pointer for generated code Matt Turner
` (4 subsequent siblings)
16 siblings, 1 reply; 47+ messages in thread
From: Matt Turner @ 2026-08-27 5:02 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
tcg_gen_lookup_and_goto_ptr() takes no arguments and emits a call to
helper_lookup_tb_ptr(), which recovers the destination PC from env by
calling back into the target through TCGCPUOps::get_tb_cpu_state(). At
translation time the caller already has the destination PC in a temp, and
knows the flags, cflags and cs_base any destination it may reach has to
match, because they are the ones the block being generated was translated
with.
Pass both, so that a later patch can use them to look the destination up
inline. Nothing reads them yet and the generated code does not change.
The contract on @pc is the whole of the interface: it must hold exactly
what get_tb_cpu_state() reports as the pc for the destination block. Five
targets keep their PC in a temp whose value is that pc by construction and
so can pass it: alpha, loongarch, mips, ppc and s390x. Everything else
passes NULL and keeps today's behavior.
For six of those the TB pc is derived and passing the PC temp would be
wrong: avr's TB pc is the word address doubled, i386's is eip before
segmentation, riscv masks it to 32 bits when xl is MXL_RV32, hppa derives
it from the IAQ, hexagon adjusts it inside a hardware loop, and sparc puts
npc in cs_base. The remaining seven -- arm, m68k, microblaze, or1k, rx, sh4
and tricore -- look like they could pass it, but I have not convinced
myself of the contract for them and have nothing to test them with. Each is
a one-line change for whoever wants it.
The common entry point takes a TCGTemp rather than a TCGv and reads the
width from it, because the translators that are built for both values of
TARGET_LONG_BITS -- arm, s390x, microblaze -- cannot include tcg-op.h.
tcg-op.h wraps it for everyone else. This is the same split as
tcg_gen_qemu_ld_*_chk().
v4: Split out of "tcg: probe the TB jump cache inline instead of calling a
helper", which did the API change and the inline probe in one patch.
Requested by Richard Henderson.
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
include/tcg/tcg-op-common.h | 15 ++++++++++++---
include/tcg/tcg-op.h | 12 ++++++++++++
target/alpha/translate.c | 4 ++--
target/arm/tcg/translate-a64.c | 4 ++--
target/arm/tcg/translate.c | 10 +++++-----
target/avr/translate.c | 4 ++--
target/hexagon/translate.c | 4 ++--
target/hppa/translate.c | 6 +++---
target/i386/tcg/translate.c | 2 +-
.../loongarch/tcg/insn_trans/trans_branch.c.inc | 2 +-
target/loongarch/tcg/translate.c | 4 ++--
target/m68k/translate.c | 2 +-
target/microblaze/translate.c | 4 ++--
target/mips/tcg/nanomips_translate.c.inc | 2 +-
target/mips/tcg/translate.c | 6 +++---
target/or1k/translate.c | 4 ++--
target/ppc/translate.c | 4 ++--
target/riscv/tcg/insn_trans/trans_rvzce.c.inc | 4 ++--
target/riscv/tcg/translate.c | 2 +-
target/rx/translate.c | 4 ++--
target/s390x/tcg/translate.c | 5 +++--
target/sh4/translate.c | 4 ++--
target/sparc/translate.c | 4 ++--
target/tricore/translate.c | 4 ++--
tcg/tcg-op.c | 3 ++-
25 files changed, 71 insertions(+), 48 deletions(-)
diff --git ./include/tcg/tcg-op-common.h ./include/tcg/tcg-op-common.h
index 9b321f959c..34102b3b7a 100644
--- ./include/tcg/tcg-op-common.h
+++ ./include/tcg/tcg-op-common.h
@@ -75,15 +75,24 @@ void tcg_gen_exit_tb(const TranslationBlock *tb, unsigned idx);
void tcg_gen_goto_tb(unsigned idx);
/**
- * tcg_gen_lookup_and_goto_ptr() - look up the current TB, jump to it if valid
- * @addr: Guest address of the target TB
+ * tcg_gen_lookup_and_goto_ptr() - look up the destination TB, jump to it
+ * @pc: temp holding the destination guest PC, or NULL
+ * @tb: the translation block being generated
*
* If the TB is not valid, jump to the epilogue.
*
+ * The lookup is a call to helper_lookup_tb_ptr(). @pc and @tb describe the
+ * destination for a faster lookup that a later patch adds, and neither is
+ * used yet. When @pc is non-NULL it must hold exactly the value
+ * get_tb_cpu_state() reports as the pc for the destination, and the
+ * destination must match @tb's flags, cflags and cs_base. A target whose
+ * pc is derived rather than being that key -- avr's word address, i386's
+ * eip before segmentation -- must pass NULL.
+ *
* This operation is optional. If the TCG backend does not implement goto_ptr,
* this op is equivalent to calling tcg_gen_exit_tb() with 0 as the argument.
*/
-void tcg_gen_lookup_and_goto_ptr(void);
+void tcg_gen_lookup_and_goto_ptr_tmp(TCGTemp *pc, const TranslationBlock *tb);
void tcg_gen_plugin_cb(unsigned from);
void tcg_gen_plugin_mem_cb(TCGv_i64 addr, unsigned meminfo);
diff --git ./include/tcg/tcg-op.h ./include/tcg/tcg-op.h
index 3721164236..24d567bd2f 100644
--- ./include/tcg/tcg-op.h
+++ ./include/tcg/tcg-op.h
@@ -49,6 +49,18 @@ typedef TCGv_i64 TCGv;
#error Unhandled TARGET_LONG_BITS value
#endif
+/*
+ * See tcg_gen_lookup_and_goto_ptr_tmp(). @pc may be NULL, for a target
+ * whose guest PC is not directly the key a destination block is found by.
+ * A translator that is built for more than one value of TARGET_LONG_BITS,
+ * and so cannot include this header, calls the _tmp() form directly.
+ */
+static inline void
+tcg_gen_lookup_and_goto_ptr(TCGv pc, const TranslationBlock *tb)
+{
+ tcg_gen_lookup_and_goto_ptr_tmp(pc ? tcgv_tl_temp(pc) : NULL, tb);
+}
+
#if TARGET_LONG_BITS == 64
#define tcg_gen_movi_tl tcg_gen_movi_i64
#define tcg_gen_mov_tl tcg_gen_mov_i64
diff --git ./target/alpha/translate.c ./target/alpha/translate.c
index c66e3f9c14..822f5cc120 100644
--- ./target/alpha/translate.c
+++ ./target/alpha/translate.c
@@ -449,7 +449,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_idx, int32_t disp)
tcg_gen_exit_tb(ctx->base.tb, tb_slot_idx);
} else {
gen_pc_disp(ctx, cpu_pc, disp);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_pc, ctx->base.tb);
}
}
@@ -2917,7 +2917,7 @@ static void alpha_tr_tb_stop(DisasContextBase *dcbase, CPUState *cpu)
gen_pc_disp(ctx, cpu_pc, 0);
/* FALLTHRU */
case DISAS_PC_UPDATED:
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_pc, ctx->base.tb);
break;
case DISAS_PC_UPDATED_NOCHAIN:
tcg_gen_exit_tb(NULL, 0);
diff --git ./target/arm/tcg/translate-a64.c ./target/arm/tcg/translate-a64.c
index 4f9a93950b..d1dd33a1af 100644
--- ./target/arm/tcg/translate-a64.c
+++ ./target/arm/tcg/translate-a64.c
@@ -562,7 +562,7 @@ static void gen_goto_tb(DisasContext *s, unsigned tb_slot_idx, int64_t diff)
if (s->ss_active) {
gen_step_complete_exception(s);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, s->base.tb);
s->base.is_jmp = DISAS_NORETURN;
}
}
@@ -11250,7 +11250,7 @@ static void aarch64_tr_tb_stop(DisasContextBase *dcbase, CPUState *cpu)
gen_a64_update_pc(dc, 4);
/* fall through */
case DISAS_JUMP:
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, dc->base.tb);
break;
case DISAS_NORETURN:
case DISAS_SWI:
diff --git ./target/arm/tcg/translate.c ./target/arm/tcg/translate.c
index c866148383..ca701b9cbc 100644
--- ./target/arm/tcg/translate.c
+++ ./target/arm/tcg/translate.c
@@ -1306,9 +1306,9 @@ void write_neon_element64(TCGv_i64 src, int reg, int ele, MemOp memop)
}
}
-static void gen_goto_ptr(void)
+static void gen_goto_ptr(DisasContext *s)
{
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr_tmp(NULL, s->base.tb);
}
/* This will end the TB but doesn't guarantee we'll return to
@@ -1336,7 +1336,7 @@ static void gen_goto_tb(DisasContext *s, unsigned tb_slot_idx, int64_t diff)
tcg_gen_exit_tb(s->base.tb, tb_slot_idx);
} else {
gen_update_pc(s, diff);
- gen_goto_ptr();
+ gen_goto_ptr(s);
}
s->base.is_jmp = DISAS_NORETURN;
}
@@ -1373,7 +1373,7 @@ static void gen_jmp_tb(DisasContext *s, int64_t diff, int tbno)
* and don't chain to another TB.
*/
gen_update_pc(s, diff);
- gen_goto_ptr();
+ gen_goto_ptr(s);
s->base.is_jmp = DISAS_NORETURN;
break;
default:
@@ -6858,7 +6858,7 @@ static void arm_tr_tb_stop(DisasContextBase *dcbase, CPUState *cpu)
gen_update_pc(dc, curr_insn_len(dc));
/* fall through */
case DISAS_JUMP:
- gen_goto_ptr();
+ gen_goto_ptr(dc);
break;
case DISAS_UPDATE_EXIT:
gen_update_pc(dc, curr_insn_len(dc));
diff --git ./target/avr/translate.c ./target/avr/translate.c
index 3c57606097..8f2e0baa67 100644
--- ./target/avr/translate.c
+++ ./target/avr/translate.c
@@ -992,7 +992,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_idx,
tcg_gen_exit_tb(tb, tb_slot_idx);
} else {
tcg_gen_movi_i32(cpu_pc, dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
ctx->base.is_jmp = DISAS_NORETURN;
}
@@ -2778,7 +2778,7 @@ static void avr_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
/* fall through */
case DISAS_LOOKUP:
if (!force_exit) {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
break;
}
/* fall through */
diff --git ./target/hexagon/translate.c ./target/hexagon/translate.c
index 06a8159d28..cc230b08d1 100644
--- ./target/hexagon/translate.c
+++ ./target/hexagon/translate.c
@@ -181,7 +181,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_idx,
if (move_to_pc) {
tcg_gen_movi_tl(hex_gpr[HEX_REG_PC], dest);
}
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
}
@@ -218,7 +218,7 @@ static void gen_end_tb(DisasContext *ctx)
gen_set_label(skip);
gen_goto_tb(ctx, 1, ctx->next_PC, false);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
ctx->base.is_jmp = DISAS_NORETURN;
diff --git ./target/hppa/translate.c ./target/hppa/translate.c
index 002189ddfb..cf8f1a2c13 100644
--- ./target/hppa/translate.c
+++ ./target/hppa/translate.c
@@ -816,7 +816,7 @@ static void gen_goto_tb(DisasContext *ctx, int which,
tcg_gen_goto_tb(which);
tcg_gen_exit_tb(ctx->base.tb, which);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
}
@@ -2027,7 +2027,7 @@ static bool do_ibranch(DisasContext *ctx, unsigned link,
store_psw_xb(ctx, PSW_B);
}
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
ctx->base.is_jmp = DISAS_NORETURN;
return nullify_end(ctx);
}
@@ -4838,7 +4838,7 @@ static void hppa_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
}
/* FALLTHRU */
case DISAS_IAQ_N_UPDATED:
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
break;
case DISAS_EXIT:
tcg_gen_exit_tb(NULL, 0);
diff --git ./target/i386/tcg/translate.c ./target/i386/tcg/translate.c
index 2115c5cd24..66a0ee3cdf 100644
--- ./target/i386/tcg/translate.c
+++ ./target/i386/tcg/translate.c
@@ -2005,7 +2005,7 @@ gen_eob(DisasContext *s, int mode)
} else if (mode == DISAS_JUMP &&
/* give irqs a chance to happen */
!inhibit_reset) {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, s->base.tb);
} else {
tcg_gen_exit_tb(NULL, 0);
}
diff --git ./target/loongarch/tcg/insn_trans/trans_branch.c.inc ./target/loongarch/tcg/insn_trans/trans_branch.c.inc
index da07778658..57d9d47353 100644
--- ./target/loongarch/tcg/insn_trans/trans_branch.c.inc
+++ ./target/loongarch/tcg/insn_trans/trans_branch.c.inc
@@ -27,7 +27,7 @@ static bool trans_jirl(DisasContext *ctx, arg_jirl *a)
tcg_gen_mov_tl(cpu_pc, addr);
tcg_gen_movi_tl(dest, make_address_pc(ctx, ctx->base.pc_next + 4));
gen_set_gpr(a->rd, dest, EXT_NONE);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_pc, ctx->base.tb);
ctx->base.is_jmp = DISAS_NORETURN;
return true;
}
diff --git ./target/loongarch/tcg/translate.c ./target/loongarch/tcg/translate.c
index 124dce6269..a45a51852a 100644
--- ./target/loongarch/tcg/translate.c
+++ ./target/loongarch/tcg/translate.c
@@ -111,7 +111,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_idx, vaddr dest)
tcg_gen_exit_tb(ctx->base.tb, tb_slot_idx);
} else {
tcg_gen_movi_tl(cpu_pc, dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_pc, ctx->base.tb);
}
}
@@ -311,7 +311,7 @@ static void loongarch_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
switch (ctx->base.is_jmp) {
case DISAS_STOP:
tcg_gen_movi_tl(cpu_pc, ctx->base.pc_next);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_pc, ctx->base.tb);
break;
case DISAS_TOO_MANY:
gen_goto_tb(ctx, 0, ctx->base.pc_next);
diff --git ./target/m68k/translate.c ./target/m68k/translate.c
index 138c89d3e5..73691bc0d1 100644
--- ./target/m68k/translate.c
+++ ./target/m68k/translate.c
@@ -6095,7 +6095,7 @@ static void m68k_tr_tb_stop(DisasContextBase *dcbase, CPUState *cpu)
if (dc->ss_active) {
gen_raise_exception_format2(dc, EXCP_TRACE, dc->pc_prev);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, dc->base.tb);
}
break;
case DISAS_EXIT:
diff --git ./target/microblaze/translate.c ./target/microblaze/translate.c
index 8b219afb5d..851b372f8f 100644
--- ./target/microblaze/translate.c
+++ ./target/microblaze/translate.c
@@ -127,7 +127,7 @@ static void gen_goto_tb(DisasContext *dc, unsigned tb_slot_idx, vaddr dest)
tcg_gen_exit_tb(dc->base.tb, tb_slot_idx);
} else {
tcg_gen_movi_i32(cpu_pc, dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr_tmp(NULL, dc->base.tb);
}
dc->base.is_jmp = DISAS_NORETURN;
}
@@ -1764,7 +1764,7 @@ static void mb_tr_tb_stop(DisasContextBase *dcb, CPUState *cs)
/* Indirect jump (or direct jump w/ goto_tb disabled) */
tcg_gen_mov_i32(cpu_pc, cpu_btarget);
tcg_gen_discard_i32(cpu_btarget);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr_tmp(NULL, dc->base.tb);
return;
default:
diff --git ./target/mips/tcg/nanomips_translate.c.inc ./target/mips/tcg/nanomips_translate.c.inc
index 4b0b01ba37..007e29f9ac 100644
--- ./target/mips/tcg/nanomips_translate.c.inc
+++ ./target/mips/tcg/nanomips_translate.c.inc
@@ -2406,7 +2406,7 @@ static void gen_compute_nanomips_pbalrsc_branch(DisasContext *ctx, int rs,
/* unconditional branch to register */
tcg_gen_mov_tl(cpu_PC, btarget);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_PC, ctx->base.tb);
}
/* nanoMIPS Branches */
diff --git ./target/mips/tcg/translate.c ./target/mips/tcg/translate.c
index e3467d1525..73abfbb5d4 100644
--- ./target/mips/tcg/translate.c
+++ ./target/mips/tcg/translate.c
@@ -4374,7 +4374,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_idx,
tcg_gen_exit_tb(ctx->base.tb, tb_slot_idx);
} else {
gen_save_pc(dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_PC, ctx->base.tb);
}
}
@@ -11014,7 +11014,7 @@ static void gen_branch(DisasContext *ctx, int insn_bytes)
} else {
tcg_gen_mov_tl(cpu_PC, btarget);
}
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_PC, ctx->base.tb);
break;
default:
LOG_DISAS("unknown branch 0x%x\n", proc_hflags);
@@ -15244,7 +15244,7 @@ static void mips_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
switch (ctx->base.is_jmp) {
case DISAS_STOP:
gen_save_pc(ctx->base.pc_next);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_PC, ctx->base.tb);
break;
case DISAS_NEXT:
case DISAS_TOO_MANY:
diff --git ./target/or1k/translate.c ./target/or1k/translate.c
index eb4485312f..4907284a6d 100644
--- ./target/or1k/translate.c
+++ ./target/or1k/translate.c
@@ -1605,7 +1605,7 @@ static void openrisc_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
/* The jump destination is indirect/computed; use jmp_pc. */
tcg_gen_mov_i32(cpu_pc, jmp_pc);
tcg_gen_discard_i32(jmp_pc);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, dc->base.tb);
break;
}
/* The jump destination is direct; use jmp_pc_imm.
@@ -1622,7 +1622,7 @@ static void openrisc_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
break;
}
tcg_gen_movi_i32(cpu_pc, jmp_dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, dc->base.tb);
break;
case DISAS_EXIT:
diff --git ./target/ppc/translate.c ./target/ppc/translate.c
index 06ed2adf10..42924281b0 100644
--- ./target/ppc/translate.c
+++ ./target/ppc/translate.c
@@ -3664,7 +3664,7 @@ static void gen_lookup_and_goto_ptr(DisasContext *ctx)
pmu_count_insns(ctx);
}
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_nip, ctx->base.tb);
}
}
@@ -6690,7 +6690,7 @@ static void ppc_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
pmu_count_insns(ctx);
}
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(cpu_nip, ctx->base.tb);
break;
case DISAS_EXIT_UPDATE:
diff --git ./target/riscv/tcg/insn_trans/trans_rvzce.c.inc ./target/riscv/tcg/insn_trans/trans_rvzce.c.inc
index 71b4ca5473..3f1e7c039e 100644
--- ./target/riscv/tcg/insn_trans/trans_rvzce.c.inc
+++ ./target/riscv/tcg/insn_trans/trans_rvzce.c.inc
@@ -213,7 +213,7 @@ static bool gen_pop(DisasContext *ctx, arg_cmpp *a, bool ret, bool ret_val)
}
#endif
tcg_gen_mov_tl(cpu_pc, ret_addr);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
ctx->base.is_jmp = DISAS_NORETURN;
}
@@ -334,7 +334,7 @@ static bool trans_cm_jalt(DisasContext *ctx, arg_cm_jalt *a)
tcg_gen_mov_tl(cpu_pc, addr);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
ctx->base.is_jmp = DISAS_NORETURN;
return true;
}
diff --git ./target/riscv/tcg/translate.c ./target/riscv/tcg/translate.c
index 9684dbe752..8475ab43b4 100644
--- ./target/riscv/tcg/translate.c
+++ ./target/riscv/tcg/translate.c
@@ -287,7 +287,7 @@ static void lookup_and_goto_ptr(DisasContext *ctx)
gen_helper_itrigger_match(tcg_env);
}
#endif
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
static void exit_tb(DisasContext *ctx)
diff --git ./target/rx/translate.c ./target/rx/translate.c
index 132d495710..e5a9783d84 100644
--- ./target/rx/translate.c
+++ ./target/rx/translate.c
@@ -161,7 +161,7 @@ static void gen_goto_tb(DisasContext *dc, unsigned tb_slot_idx, vaddr dest)
tcg_gen_exit_tb(dc->base.tb, tb_slot_idx);
} else {
tcg_gen_movi_i32(cpu_pc, dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, dc->base.tb);
}
dc->base.is_jmp = DISAS_NORETURN;
}
@@ -2242,7 +2242,7 @@ static void rx_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
gen_goto_tb(ctx, 0, dcbase->pc_next);
break;
case DISAS_JUMP:
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
break;
case DISAS_UPDATE:
tcg_gen_movi_i32(cpu_pc, ctx->base.pc_next);
diff --git ./target/s390x/tcg/translate.c ./target/s390x/tcg/translate.c
index 1b6023168b..607c039419 100644
--- ./target/s390x/tcg/translate.c
+++ ./target/s390x/tcg/translate.c
@@ -1162,7 +1162,7 @@ static DisasJumpType help_branch(DisasContext *s, DisasCompare *c,
tcg_gen_goto_tb(0);
tcg_gen_exit_tb(s->base.tb, 0);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr_tmp(tcgv_i64_temp(psw_addr), s->base.tb);
}
gen_set_label(lab);
@@ -6477,7 +6477,8 @@ static void s390x_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
if (dc->exit_to_mainloop) {
tcg_gen_exit_tb(NULL, 0);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr_tmp(tcgv_i64_temp(psw_addr),
+ dc->base.tb);
}
break;
default:
diff --git ./target/sh4/translate.c ./target/sh4/translate.c
index 373950fd66..a4be456bd9 100644
--- ./target/sh4/translate.c
+++ ./target/sh4/translate.c
@@ -242,7 +242,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_idx, vaddr dest)
if (use_exit_tb(ctx)) {
tcg_gen_exit_tb(NULL, 0);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
}
ctx->base.is_jmp = DISAS_NORETURN;
@@ -258,7 +258,7 @@ static void gen_jump(DisasContext * ctx)
if (use_exit_tb(ctx)) {
tcg_gen_exit_tb(NULL, 0);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
ctx->base.is_jmp = DISAS_NORETURN;
} else {
diff --git ./target/sparc/translate.c ./target/sparc/translate.c
index 3156be6a94..2ae0a02c44 100644
--- ./target/sparc/translate.c
+++ ./target/sparc/translate.c
@@ -376,7 +376,7 @@ static void gen_goto_tb(DisasContext *s, unsigned tb_slot_idx,
/* jump to another page: we can use an indirect jump */
tcg_gen_movi_tl(cpu_pc, pc);
tcg_gen_movi_tl(cpu_npc, npc);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, s->base.tb);
}
}
@@ -5807,7 +5807,7 @@ static void sparc_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
tcg_gen_movi_tl(cpu_npc, dc->npc);
}
if (may_lookup) {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, dc->base.tb);
} else {
tcg_gen_exit_tb(NULL, 0);
}
diff --git ./target/tricore/translate.c ./target/tricore/translate.c
index 8cd6b58f66..1d7f54f6df 100644
--- ./target/tricore/translate.c
+++ ./target/tricore/translate.c
@@ -2857,7 +2857,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_index, vaddr dest)
tcg_gen_exit_tb(ctx->base.tb, tb_slot_index);
} else {
gen_save_pc(dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
}
ctx->base.is_jmp = DISAS_NORETURN;
}
@@ -8478,7 +8478,7 @@ static void tricore_tr_tb_stop(DisasContextBase *dcbase, CPUState *cpu)
tcg_gen_exit_tb(NULL, 0);
break;
case DISAS_JUMP:
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_lookup_and_goto_ptr(NULL, ctx->base.tb);
break;
case DISAS_NORETURN:
break;
diff --git ./tcg/tcg-op.c ./tcg/tcg-op.c
index 28d3b2a847..2fda6e5c07 100644
--- ./tcg/tcg-op.c
+++ ./tcg/tcg-op.c
@@ -2715,7 +2715,7 @@ void tcg_gen_goto_tb(unsigned idx)
tcg_gen_op1i(INDEX_op_goto_tb, 0, idx);
}
-void tcg_gen_lookup_and_goto_ptr(void)
+void tcg_gen_lookup_and_goto_ptr_tmp(TCGTemp *pc, const TranslationBlock *tb)
{
TCGv_ptr ptr;
@@ -2725,6 +2725,7 @@ void tcg_gen_lookup_and_goto_ptr(void)
}
plugin_gen_disable_mem_helpers();
+
ptr = tcg_temp_ebb_new_ptr();
gen_helper_lookup_tb_ptr(ptr, tcg_env);
tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr));
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH v4 5/9] accel/tcg: give the TB jump cache a second base pointer for generated code
2026-08-22 19:08 [PATCH v3 0/7] accel/tcg: cut per-block dispatch overhead Matt Turner
` (11 preceding siblings ...)
2026-08-27 5:02 ` [PATCH v4 4/9] tcg: pass the destination to tcg_gen_lookup_and_goto_ptr() Matt Turner
@ 2026-08-27 5:02 ` Matt Turner
2026-08-27 20:03 ` Richard Henderson
2026-08-27 5:02 ` [PATCH v4 6/9] RFC: tcg: probe the TB jump cache inline instead of calling a helper Matt Turner
` (3 subsequent siblings)
16 siblings, 1 reply; 47+ messages in thread
From: Matt Turner @ 2026-08-27 5:02 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
Add CPUState::tb_jmp_cache_probe, a base pointer that only generated code
will read. Normally it is cpu->tb_jmp_cache. Pointing it instead at a
shared, permanently zero-filled CPUJumpCache makes every entry generated
code finds have a NULL tb, so every lookup done through it misses. The
real jump cache is not touched, so nothing is lost and recovery is a single
store.
Nothing reads it yet. The next patch probes the jump cache from generated
code, and that probe cannot check everything helper_lookup_tb_ptr() checks;
poisoning this pointer is how the conditions it cannot check force it back
into the helper. The one that matters here is breakpoints.
check_for_breakpoints() raises EXCP_DEBUG on an exact pc match and selects
CF_BP_PAGE cflags for the rest of the page, and inserting a breakpoint
deliberately invalidates no TB, so a block translated before the breakpoint
was set is still sitting in the jump cache and would be dispatched to
directly.
cpu_breakpoint_insert() poisons the target CPU, rather than leaving it to
that CPU's own main loop, because gdb inserts a breakpoint into every CPU
(tcg_insert_gdbstub_breakpoint()) and a thread already inside generated
code dispatching to itself need never return to its main loop. The main
loop puts the pointer back once the last breakpoint is gone, re-checking
after the store so that it loses a race with a concurrent insert in the
safe direction.
The poison cache is a plain static rather than a const one so that it lands
in .bss: a megabyte of const zeroes would be a megabyte of .rodata in every
emulator binary, whereas .bss costs nothing on disk and faults in only the
handful of pages a poisoned run happens to probe.
v4: Split out of "tcg: probe the TB jump cache inline instead of calling a
helper". Requested by Richard Henderson.
v4: Make the poison a static object rather than allocating one on first
use. Suggested by Richard Henderson, who asked for const; see above for
why it is not.
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
accel/stubs/tcg-stub.c | 6 ++-
accel/tcg/cpu-exec.c | 96 +++++++++++++++++++++++++++++++++++++
accel/tcg/internal-common.h | 2 +
cpu-common.c | 11 +++++
include/hw/core/cpu.h | 9 ++++
include/system/tcg.h | 9 ++++
6 files changed, 132 insertions(+), 1 deletion(-)
diff --git ./accel/stubs/tcg-stub.c ./accel/stubs/tcg-stub.c
index f9e1bd22d6..8298e4a1f5 100644
--- ./accel/stubs/tcg-stub.c
+++ ./accel/stubs/tcg-stub.c
@@ -1,6 +1,6 @@
/*
* Stubs for the TCG entry points in system/tcg.h, for binaries that link
- * cpu-target.c or the HMP command handlers but not TCG.
+ * cpu-target.c, cpu-common.c or the HMP command handlers but not TCG.
*
* SPDX-License-Identifier: GPL-2.0-or-later
*/
@@ -14,3 +14,7 @@ void tcg_update_cflags(CPUState *cpu)
void tcg_update_all_cflags(void)
{
}
+
+void tcg_cpu_poison_jmp_cache(CPUState *cpu)
+{
+}
diff --git ./accel/tcg/cpu-exec.c ./accel/tcg/cpu-exec.c
index 148e0f583e..c2a9679cd7 100644
--- ./accel/tcg/cpu-exec.c
+++ ./accel/tcg/cpu-exec.c
@@ -752,6 +752,93 @@ static inline bool cpu_handle_exception(CPUState *cpu, int *ret)
return false;
}
+/*
+ * The inline jump cache probe reads cpu->tb_jmp_cache_probe and takes the
+ * slow path when the entry it finds has a NULL tb. Pointing the probe at a
+ * region that is all zeroes therefore forces every indirect dispatch into
+ * helper_lookup_tb_ptr(), which does the full lookup the inline probe only
+ * approximates. The real jump cache is untouched, so no contents are lost
+ * and recovery is a single store.
+ *
+ * Only ever read from, and only one entry per dispatch, so one shared
+ * zero-filled cache is enough for every CPU. Not const: that would put a
+ * megabyte of zeroes in .rodata and so in the binary, where .bss costs
+ * nothing on disk and only faults in the handful of pages a poisoned run
+ * happens to probe.
+ */
+static CPUJumpCache tb_jmp_cache_poison;
+
+/*
+ * Whether the generated code may dispatch to the next block by itself.
+ *
+ * The inline probe matches on the destination pc and on the flags and
+ * cflags the dispatching block was translated with. It does not consult
+ * cpu->breakpoints, so it must not run while one is set: setting a
+ * breakpoint deliberately invalidates nothing, and check_for_breakpoints()
+ * both raises EXCP_DEBUG on an exact match and picks CF_BP_PAGE cflags for
+ * the rest of the page. A block translated before the breakpoint was set is
+ * therefore still in the jump cache, and dispatching to it inline would step
+ * straight over the breakpoint.
+ */
+static bool tcg_cpu_may_dispatch(CPUState *cpu)
+{
+ return QTAILQ_EMPTY(&cpu->breakpoints);
+}
+
+/*
+ * Poison @cpu's probe, from any thread. Called when a breakpoint is
+ * inserted, which is what makes the poison take effect at the dispatch
+ * after the insert rather than whenever @cpu next reaches its main loop:
+ * a vCPU chaining indirectly need never reach it, and would run past a
+ * breakpoint another thread had just set.
+ *
+ * A plain store is enough. The value only ever costs a slow path that is
+ * correct on its own, and the generated code re-reads the base on every
+ * dispatch. Un-poisoning is tcg_cpu_sync_jmp_cache()'s job.
+ */
+void tcg_cpu_poison_jmp_cache(CPUState *cpu)
+{
+ if (qatomic_read(&cpu->tb_jmp_cache_probe) != NULL) {
+ qatomic_set(&cpu->tb_jmp_cache_probe, &tb_jmp_cache_poison);
+ }
+}
+
+/*
+ * Called from the main loop, which is the only context that can establish
+ * that no reason to be poisoned is left. Cheap enough to call every time
+ * round: the common case is a load, a compare and no store at all.
+ */
+void tcg_cpu_sync_jmp_cache(CPUState *cpu)
+{
+ CPUJumpCache *want;
+
+ if (qatomic_read(&cpu->tb_jmp_cache_probe) == NULL) {
+ return; /* not realized, or already unrealized */
+ }
+
+ want = tcg_cpu_may_dispatch(cpu)
+ ? cpu->tb_jmp_cache
+ : &tb_jmp_cache_poison;
+
+ if (qatomic_read(&cpu->tb_jmp_cache_probe) != want) {
+ qatomic_set(&cpu->tb_jmp_cache_probe, want);
+
+ if (want == cpu->tb_jmp_cache) {
+ /*
+ * Un-poisoning races a concurrent tcg_cpu_poison_jmp_cache():
+ * the reason may have appeared after tcg_cpu_may_dispatch() read
+ * it, and the poison may have landed before the store above.
+ * Order that store against the re-read below, so that the race
+ * is lost in the safe direction.
+ */
+ smp_mb();
+ if (!tcg_cpu_may_dispatch(cpu)) {
+ tcg_cpu_poison_jmp_cache(cpu);
+ }
+ }
+ }
+}
+
void tcg_kick_vcpu_thread(CPUState *cpu)
{
/*
@@ -964,6 +1051,13 @@ cpu_exec_loop(CPUState *cpu, SyncClocks *sc)
break;
}
+ /*
+ * Reaching here means the main loop has just re-evaluated
+ * everything the inline probe assumes, so this is where the
+ * probe is allowed to come back after a poison.
+ */
+ tcg_cpu_sync_jmp_cache(cpu);
+
tb = tb_lookup(cpu, s);
if (tb == NULL) {
CPUJumpCache *jc;
@@ -1072,6 +1166,7 @@ bool tcg_exec_realizefn(CPUState *cpu, Error **errp)
tcg_update_cflags(cpu);
cpu->tb_jmp_cache = g_new0(CPUJumpCache, 1);
+ qatomic_set(&cpu->tb_jmp_cache_probe, cpu->tb_jmp_cache);
tlb_init(cpu);
#ifndef CONFIG_USER_ONLY
tcg_iommu_init_notifier_list(cpu);
@@ -1089,5 +1184,6 @@ void tcg_exec_unrealizefn(CPUState *cpu)
#endif /* !CONFIG_USER_ONLY */
tlb_destroy(cpu);
+ qatomic_set(&cpu->tb_jmp_cache_probe, NULL);
g_free_rcu(cpu->tb_jmp_cache, rcu);
}
diff --git ./accel/tcg/internal-common.h ./accel/tcg/internal-common.h
index 853d1b51ee..9d1f6712d6 100644
--- ./accel/tcg/internal-common.h
+++ ./accel/tcg/internal-common.h
@@ -144,6 +144,8 @@ void page_table_config_init(void);
G_NORETURN void cpu_io_recompile(CPUState *cpu, uintptr_t retaddr);
#endif /* CONFIG_USER_ONLY */
+void tcg_cpu_sync_jmp_cache(CPUState *cpu);
+
void tb_phys_invalidate(TranslationBlock *tb, tb_page_addr_t page_addr);
void tb_set_jmp_target(TranslationBlock *tb, int n, uintptr_t addr);
diff --git ./cpu-common.c ./cpu-common.c
index adb76b3a78..3aed0156e6 100644
--- ./cpu-common.c
+++ ./cpu-common.c
@@ -22,6 +22,7 @@
#include "exec/cpu-common.h"
#include "hw/core/cpu.h"
#include "qemu/lockable.h"
+#include "system/tcg.h"
#include "trace/trace-root.h"
QemuMutex qemu_cpu_list_lock;
@@ -429,6 +430,16 @@ int cpu_breakpoint_insert(CPUState *cpu, vaddr pc, int flags,
*breakpoint = bp;
}
+ /*
+ * Nothing is invalidated here, so blocks translated before this point
+ * are still live and still dispatch to each other without consulting
+ * cpu->breakpoints. Stop the ones that can: a TCG vCPU dispatching
+ * inline reads a base pointer that this poisons, so the next dispatch
+ * takes the slow path and sees the new breakpoint. @cpu may be another
+ * thread, and may be running.
+ */
+ tcg_cpu_poison_jmp_cache(cpu);
+
trace_breakpoint_insert(cpu->cpu_index, pc, flags);
return 0;
}
diff --git ./include/hw/core/cpu.h ./include/hw/core/cpu.h
index 81af7b9ee1..bd2cdd2a0b 100644
--- ./include/hw/core/cpu.h
+++ ./include/hw/core/cpu.h
@@ -519,6 +519,15 @@ struct CPUState {
MemoryRegion *memory;
struct CPUJumpCache *tb_jmp_cache;
+ /*
+ * @tb_jmp_cache_probe: base the inline jump cache probe reads.
+ *
+ * Normally @tb_jmp_cache. Pointed at a shared page of zeroes to force
+ * every inline dispatch to miss and fall back to helper_lookup_tb_ptr();
+ * see tcg_cpu_sync_jmp_cache(). NULL before tcg_exec_realizefn() and
+ * after tcg_exec_unrealizefn().
+ */
+ struct CPUJumpCache *tb_jmp_cache_probe;
GArray *gdb_regs;
int gdb_num_regs;
diff --git ./include/system/tcg.h ./include/system/tcg.h
index 2c2dbc753b..bf05db1329 100644
--- ./include/system/tcg.h
+++ ./include/system/tcg.h
@@ -29,6 +29,15 @@ extern bool tcg_allowed;
void tcg_update_cflags(CPUState *cpu);
void tcg_update_all_cflags(void);
+/*
+ * Force @cpu's generated code back into the slow dispatch path, which
+ * re-checks everything the inline jump cache probe assumes. Safe to call
+ * from any thread, and a no-op for a CPU that is not running TCG. Call
+ * whenever something the probe cannot see changes under a running vCPU;
+ * the main loop undoes it once the reason is gone.
+ */
+void tcg_cpu_poison_jmp_cache(CPUState *cpu);
+
/**
* qemu_tcg_mttcg_enabled:
* Check whether we are running MultiThread TCG or not.
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH v4 6/9] RFC: tcg: probe the TB jump cache inline instead of calling a helper
2026-08-22 19:08 [PATCH v3 0/7] accel/tcg: cut per-block dispatch overhead Matt Turner
` (12 preceding siblings ...)
2026-08-27 5:02 ` [PATCH v4 5/9] accel/tcg: give the TB jump cache a second base pointer for generated code Matt Turner
@ 2026-08-27 5:02 ` Matt Turner
2026-08-27 23:34 ` Richard Henderson
2026-08-27 5:02 ` [PATCH v4 7/9] RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds Matt Turner
` (2 subsequent siblings)
16 siblings, 1 reply; 47+ messages in thread
From: Matt Turner @ 2026-08-27 5:02 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
Every indirect branch that cannot use goto_tb ends in
tcg_gen_lookup_and_goto_ptr(), which calls helper_lookup_tb_ptr(). For an
emulated compiler that is 8.4 billion helper calls in a single translation
unit: 24.6% of all TB exits take this path, because jsr/ret/jmp have a
register destination and because goto_tb is restricted to same-page
targets.
The helper itself is already tight, but each call pays for a call frame,
the can_do_io store, the get_tb_cpu_state() indirect call through
TCGCPUOps, curr_cflags(), and a breakpoint check, before it gets to the
jump cache probe that almost always hits (95.8% for this workload).
Emit the probe inline instead. The two preceding patches supply what it
needs: the destination PC is in a TCG temp, the flags, cflags and cs_base
the destination must match are constants at translation time, and
tb_jmp_cache_probe is a base pointer the main loop can poison. The fast
path is therefore a hash, four guarded loads and a goto_ptr. Only a miss
calls the helper, which still owns filling the cache.
Two details matter for the generated code. The flags and cflags guards are
folded into a single aligned 64-bit load and compare, since the fields are
adjacent. And each path emits its own goto_ptr rather than branching to a
shared one: a temp live across the label is spilled and reloaded on every
dispatch, which cost 6.3% on its own.
The flags and cflags constants are safe against the other things that can
change them. CF_PARALLEL is only ever set by begin_parallel_context(),
which flushes first, so no block predating it survives to dispatch. gdb
single-step is only turned on with the CPU stopped, and a block translated
without CF_SINGLE_STEP can only be re-entered through tb_lookup(), which
from then on demands the new cflags -- so a stale-cflags block is never the
one running. What is left is one_insn_per_tb and -d nochain, which the
monitor can toggle under a running vCPU without a flush; see below.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling
the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, LTO
build, on top of the preceding patches:
before: 1,402,667,803,616 instructions
after: 916,415,123,244 instructions -34.67%
before: 115.56s wall clock
after: 85.59s wall clock -25.94%
The gap between the two is the point at which this stops being a
straight-line win: the helper call was highly predictable work that the
host pipelined well, so removing it retires far fewer instructions than it
saves time. IPC falls from 2.48 to 2.17 across this patch for that reason.
Despite emitting more code, this also reduces instruction cache pressure,
because a dispatch no longer jumps into qemu's .text and evicts translated
code:
before: 11,735,141,703 L1-icache-load-misses
after: 7,154,863,292 L1-icache-load-misses -39.0%
The mechanism is visible directly in a profile: helper_lookup_tb_ptr()
falls from 31.01% of samples to 0.35%, and qemu's own .text falls from
38.8% to 5.3%, with the balance moving into generated code.
Combined with the preceding patches, against an unmodified LTO build,
1,646,994,254,249 instructions fall to 916,415,123,244, or -44.36%. The
emulated compiler produces byte-identical output throughout.
Open issues, hence RFC:
- one_insn_per_tb and CPU_LOG_TB_NOCHAIN can be toggled from the monitor
while a vCPU is inside a block that was translated without them. The
block keeps dispatching inline with the old cflags until it exits for
some other reason. Poisoning the probe from tcg_update_all_cflags()
would close it.
- The jump cache entry is read without qatomic_read(); entries are
invalidated concurrently by setting tb to NULL.
- Only alpha has been measured. The other four targets that pass a PC are
built and boot-tested only.
v4: Split out of the patch that also changed the
tcg_gen_lookup_and_goto_ptr() API and introduced tb_jmp_cache_probe,
which are now the two preceding patches. Requested by Richard
Henderson.
v4: Emit the softmmu form of tb_jmp_cache_hash_func() under
CONFIG_SOFTMMU rather than the user-only form everywhere. v3 emitted
the user-only hash unconditionally, which was wrong for system mode
and was only not a correctness bug because a wrong index simply
misses. Caught by Richard Henderson. tcg-op.c is compiled once per
build rather than once per target, but CONFIG_SOFTMMU is set for it,
and TARGET_PAGE_BITS -- a load from target_page here -- is fixed long
before any translation happens.
v4: Compare the pc before testing tb for NULL. On a hash miss the pc is
the field most likely to differ, and an unused entry has a zero pc
that only pc 0 can match, so the tb test buys nothing ahead of it.
Suggested by Richard Henderson.
v4: Assert that offsetof(TranslationBlock, flags) is 8-byte aligned, since
folding the flags and cflags guards into one 64-bit load relies on it
and nothing else does. Requested by Richard Henderson.
v4: Zero-extend a 32-bit guest PC instead of falling back to the helper.
Suggested by Richard Henderson. The high half then folds to a compare
against zero.
v4: Describe cs_base in the probe as a second word of target-specific
flags rather than by name. Suggested by Richard Henderson.
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
include/tcg/tcg-op-common.h | 14 ++--
tcg/tcg-op.c | 126 ++++++++++++++++++++++++++++++++++++
2 files changed, 133 insertions(+), 7 deletions(-)
diff --git ./include/tcg/tcg-op-common.h ./include/tcg/tcg-op-common.h
index 34102b3b7a..b8399229c9 100644
--- ./include/tcg/tcg-op-common.h
+++ ./include/tcg/tcg-op-common.h
@@ -81,13 +81,13 @@ void tcg_gen_goto_tb(unsigned idx);
*
* If the TB is not valid, jump to the epilogue.
*
- * The lookup is a call to helper_lookup_tb_ptr(). @pc and @tb describe the
- * destination for a faster lookup that a later patch adds, and neither is
- * used yet. When @pc is non-NULL it must hold exactly the value
- * get_tb_cpu_state() reports as the pc for the destination, and the
- * destination must match @tb's flags, cflags and cs_base. A target whose
- * pc is derived rather than being that key -- avr's word address, i386's
- * eip before segmentation -- must pass NULL.
+ * The lookup is normally a call to helper_lookup_tb_ptr(). If @pc is
+ * non-NULL the jump cache is probed inline instead, and only a miss reaches
+ * the helper. @pc must then hold exactly the value get_tb_cpu_state()
+ * reports as the pc for the destination; a target whose pc is derived
+ * (avr's word address, i386's eip before segmentation) must pass NULL. The
+ * destination is required to match @tb's flags, cflags and cs_base, which
+ * is what makes them constants in the probe.
*
* This operation is optional. If the TCG backend does not implement goto_ptr,
* this op is equivalent to calling tcg_gen_exit_tb() with 0 as the argument.
diff --git ./tcg/tcg-op.c ./tcg/tcg-op.c
index 2fda6e5c07..cf7b6882d8 100644
--- ./tcg/tcg-op.c
+++ ./tcg/tcg-op.c
@@ -28,6 +28,8 @@
#include "tcg/tcg-op-common.h"
#include "exec/translation-block.h"
#include "exec/plugin-gen.h"
+#include "hw/core/cpu.h"
+#include "../accel/tcg/tb-hash.h"
#include "tcg-internal.h"
#include "tcg-has.h"
@@ -2715,6 +2717,112 @@ void tcg_gen_goto_tb(unsigned idx)
tcg_gen_op1i(INDEX_op_goto_tb, 0, idx);
}
+static void gen_jmp_cache_hash(TCGv_i64 h, TCGv_i64 pc)
+{
+#ifdef CONFIG_SOFTMMU
+ /*
+ * tb_jmp_cache_hash_func(), softmmu form. TARGET_PAGE_BITS is a load
+ * from target_page in this translation unit, but it is decided long
+ * before any translation happens, so it is a constant here.
+ */
+ int shift = TARGET_PAGE_BITS - TB_JMP_PAGE_BITS;
+ TCGv_i64 tmp = tcg_temp_ebb_new_i64();
+
+ tcg_gen_shri_i64(tmp, pc, shift);
+ tcg_gen_xor_i64(tmp, tmp, pc);
+ tcg_gen_shri_i64(h, tmp, shift);
+ tcg_gen_andi_i64(h, h, TB_JMP_PAGE_MASK);
+ tcg_gen_andi_i64(tmp, tmp, TB_JMP_ADDR_MASK);
+ tcg_gen_or_i64(h, h, tmp);
+ tcg_temp_free_i64(tmp);
+#else
+ /* tb_jmp_cache_hash_func(), user-only form. */
+ tcg_gen_shri_i64(h, pc, TB_JMP_CACHE_BITS);
+ tcg_gen_xor_i64(h, h, pc);
+ tcg_gen_andi_i64(h, h, TB_JMP_CACHE_SIZE - 1);
+#endif
+}
+
+static void gen_jmp_cache_probe(TCGv_i64 pc, const TranslationBlock *tb)
+{
+ TCGv_ptr jc, ent, tbp, ptr;
+ TCGv_i64 h, tmp;
+ TCGLabel *slow;
+ uint64_t fpair;
+
+ QEMU_BUILD_BUG_ON(sizeof(((CPUJumpCache *)0)->array[0]) != 16);
+ QEMU_BUILD_BUG_ON(offsetof(CPUJumpCache, array[0].pc) % 8 != 0);
+ /* One 64-bit load has to cover both, so they must be adjacent... */
+ QEMU_BUILD_BUG_ON(offsetof(TranslationBlock, cflags) !=
+ offsetof(TranslationBlock, flags) + 4);
+ /* ...and aligned, which nothing else currently relies on. */
+ QEMU_BUILD_BUG_ON(offsetof(TranslationBlock, flags) % 8 != 0);
+
+ jc = tcg_temp_ebb_new_ptr();
+ ent = tcg_temp_ebb_new_ptr();
+ tbp = tcg_temp_ebb_new_ptr();
+ ptr = tcg_temp_ebb_new_ptr();
+ h = tcg_temp_ebb_new_i64();
+ tmp = tcg_temp_ebb_new_i64();
+ slow = gen_new_label();
+
+ /* ent = &jc->array[tb_jmp_cache_hash_func(pc)] */
+ gen_jmp_cache_hash(h, pc);
+ tcg_gen_shli_i64(h, h, 4);
+
+ /*
+ * Not cpu->tb_jmp_cache: the probe reads its own base so that the main
+ * loop can poison it, which is how conditions the probe cannot test for
+ * itself force every dispatch back into the helper. See
+ * tcg_cpu_sync_jmp_cache().
+ */
+ tcg_gen_ld_ptr(jc, tcg_env,
+ offsetof(CPUState, tb_jmp_cache_probe) - sizeof(CPUState));
+ tcg_gen_trunc_i64_ptr(ent, h);
+ tcg_gen_add_ptr(ent, jc, ent);
+
+ /*
+ * The pc first: on a hash miss it is the field most likely to differ,
+ * and an entry whose tb is NULL has a zero pc that only pc 0 matches.
+ */
+ tcg_gen_ld_i64(tmp, ent, offsetof(CPUJumpCache, array[0].pc));
+ tcg_gen_brcond_i64(TCG_COND_NE, tmp, pc, slow);
+
+ tcg_gen_ld_ptr(tbp, ent, offsetof(CPUJumpCache, array[0].tb));
+ tcg_gen_brcondi_ptr(TCG_COND_EQ, tbp, 0, slow);
+
+ /*
+ * flags and cflags are adjacent uint32_t, so one aligned 64-bit load
+ * and compare covers both.
+ */
+#if HOST_BIG_ENDIAN
+ fpair = ((uint64_t)tb->flags << 32) | tb->cflags;
+#else
+ fpair = ((uint64_t)tb->cflags << 32) | tb->flags;
+#endif
+ tcg_gen_ld_i64(tmp, tbp, offsetof(TranslationBlock, flags));
+ tcg_gen_brcondi_i64(TCG_COND_NE, tmp, fpair, slow);
+
+ /*
+ * cs_base is a second word of target-specific flags despite the name,
+ * and the pc alone does not imply it on a target that uses it.
+ */
+ tcg_gen_ld_i64(tmp, tbp, offsetof(TranslationBlock, cs_base));
+ tcg_gen_brcondi_i64(TCG_COND_NE, tmp, tb->cs_base, slow);
+
+ tcg_gen_ld_ptr(ptr, tbp, offsetof(TranslationBlock, tc.ptr));
+ tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr));
+
+ /*
+ * Emit a second goto_ptr rather than branching to a shared one: a temp
+ * live across the label would be spilled and reloaded on every dispatch.
+ */
+ gen_set_label(slow);
+ ptr = tcg_temp_ebb_new_ptr();
+ gen_helper_lookup_tb_ptr(ptr, tcg_env);
+ tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr));
+}
+
void tcg_gen_lookup_and_goto_ptr_tmp(TCGTemp *pc, const TranslationBlock *tb)
{
TCGv_ptr ptr;
@@ -2726,6 +2834,24 @@ void tcg_gen_lookup_and_goto_ptr_tmp(TCGTemp *pc, const TranslationBlock *tb)
plugin_gen_disable_mem_helpers();
+ if (pc) {
+ TCGv_i64 pc64;
+
+ /*
+ * The jump cache is keyed on a vaddr, so a 32-bit guest PC is
+ * compared as its zero-extension. The high half then folds to a
+ * constant compare against zero.
+ */
+ if (pc->type == TCG_TYPE_I32) {
+ pc64 = tcg_temp_ebb_new_i64();
+ tcg_gen_extu_i32_i64(pc64, temp_tcgv_i32(pc));
+ } else {
+ pc64 = temp_tcgv_i64(pc);
+ }
+ gen_jmp_cache_probe(pc64, tb);
+ return;
+ }
+
ptr = tcg_temp_ebb_new_ptr();
gen_helper_lookup_tb_ptr(ptr, tcg_env);
tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr));
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH v4 7/9] RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds
2026-08-22 19:08 [PATCH v3 0/7] accel/tcg: cut per-block dispatch overhead Matt Turner
` (13 preceding siblings ...)
2026-08-27 5:02 ` [PATCH v4 6/9] RFC: tcg: probe the TB jump cache inline instead of calling a helper Matt Turner
@ 2026-08-27 5:02 ` Matt Turner
2026-08-27 5:02 ` [PATCH v4 8/9] RFC: accel/tcg: poison the jump cache instead of polling for indirect exits Matt Turner
2026-08-27 5:02 ` [PATCH v4 9/9] RFC: tcg: fold a guest displacement into the host addressing mode Matt Turner
16 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-08-27 5:02 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
translator_use_goto_tb() refuses to chain unless the destination is on the
same page as the start of the TB. For guests whose text is much larger than
a page this is expensive: an emulated alpha gcc compiling a 255k line
translation unit takes the indirect dispatch path for 8.4 billion of its
34.2 billion TB exits, and a large share of those are ordinary direct
branches that simply crossed an 8 KiB page boundary.
The restriction was made unconditional by d3a2a1d803 ("accel/tcg:
Introduce translator_use_goto_tb"), whose rationale was:
Various targets avoid the page crossing test for CONFIG_USER_ONLY,
but that is wrong: mmap and mprotect can change page permissions.
That is true, but in user-only builds the invalidation path already covers
it. There are no page tables: every mmap, mprotect and munmap reaches
page_set_flags(), which calls tb_invalidate_phys_range() whenever the flags
actually change, and tb_phys_invalidate() calls tb_jmp_unlink() to reset
incoming jumps. A chained cross-page jump is therefore broken whenever the
destination page's permissions change. This is not true in system mode,
where TBs are keyed by physical address and a page table change invalidates
nothing, so the restriction is kept there.
The rule protects one more thing, which the original rationale does not
mention: it guarantees that execution cannot enter a page without a TB
lookup, and so without check_for_breakpoints(). That is what makes a
breakpoint set after a block was translated take effect, since insertion
deliberately invalidates nothing. A link established before the breakpoint
was set would jump straight over it.
So the chaining is only enabled for a run that can never acquire a
breakpoint. In user-only mode every breakpoint comes from gdb -- BP_CPU is
g_assert_not_reached() there, and the guest cannot ask for one -- and gdb
has to be requested with -g before the first block is translated, even
though with suspend=n it may connect later. gdb_may_set_breakpoints()
reports whether it was, and is fixed for the lifetime of the process.
Add tests/tcg/multiarch/test-xpage-chain.c to cover both hazards directly.
It writes the last instruction of one page and the first of the next, so
that the fall-through between them is a cross-page goto_tb, runs it 200000
times so the chain is established, then checks that mprotect(PROT_NONE)
makes the next call fault, and that different code written into the page
once it is mapped back runs rather than a stale translation.
The two instructions -- set the return value register, and return -- are
all the architecture specific code there is; thirteen architectures supply
them and the rest skip.
The test detects the hazard it is meant to detect: with the
tb_invalidate_phys_range() call in page_set_flags() commented out, it fails
both phases, executing page B after PROT_NONE and returning the stale
result.
Run with -b, the same binary stops once the chain is established and lets
tests/tcg/multiarch/gdbstub/xpage-bp.py set a breakpoint on the far side of it,
which the next call has to stop on. With gdb_may_set_breakpoints() forced to
false so that the chaining stays on under gdb, that breakpoint is missed and
the test fails, which is what makes it a test of the gate rather than of
gdb.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the
SQLite 3.45.1 amalgamation on an x86-64 host, LTO build, on top of the
preceding patches:
before: 916,415,123,244 instructions
after: 891,254,240,071 instructions -2.75%
before: 85.59s wall clock
after: 81.45s wall clock -4.84%
Note that this is worth more in time than in instructions, the reverse of
the preceding patch: a chained jump replaces a cache probe whose loads can
miss, so the instructions it removes are more expensive than average.
Measured before the inline jump cache probe, when a missed chain cost a
helper call rather than an inline probe, the same change was worth -7.9%.
RFC because this reverses a deliberate decision and the reasoning above
wants review from someone who knows the invalidation paths better than I
do.
v3: Only take the shortcut when no gdbstub was requested. The same-page
rule also forces a lookup, and so a breakpoint check, on entry to every
page; without that, a chain established before a breakpoint was set runs
past it. Reported by Richard Henderson.
v3: Change translator_use_goto_tb() rather than translator_is_same_page().
i386, riscv and s390x call translator_is_same_page() for something else
-- enforcing that only a single-insn TB may cross a page -- and v2
changed their TB boundaries in user-only mode as a side effect. alpha
does not call it, so the numbers above are unaffected.
v3: Add the gdbstub half of the test.
v4: Move the test to tests/tcg/multiarch so that every *-user target runs
it, rather than only alpha. Requested by Alex Bennee. The direct branch
is gone with it: a fall-through off the end of a page is a cross-page
goto_tb just the same, and needs no per-architecture branch encoding or
displacement arithmetic, only "set the return value" and "return".
Built and run under qemu-user on aarch64, alpha, arm, hppa,
loongarch64, m68k, mips, ppc, ppc64le, riscv64, s390x, sh4, sparc64
and x86_64; ppc64 ELFv1 skips, because a function pointer there is a
descriptor rather than a code address.
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
accel/tcg/translator.c | 33 ++-
gdbstub/user.c | 14 +
include/gdbstub/user.h | 11 +
tests/tcg/multiarch/Makefile.target | 12 +-
tests/tcg/multiarch/gdbstub/xpage-bp.py | 37 +++
tests/tcg/multiarch/test-xpage-chain.c | 336 ++++++++++++++++++++++++
6 files changed, 441 insertions(+), 2 deletions(-)
create mode 100644 tests/tcg/multiarch/gdbstub/xpage-bp.py
create mode 100644 tests/tcg/multiarch/test-xpage-chain.c
diff --git ./accel/tcg/translator.c ./accel/tcg/translator.c
index 6c8fcd7a20..8879cd626f 100644
--- ./accel/tcg/translator.c
+++ ./accel/tcg/translator.c
@@ -15,6 +15,9 @@
#include "accel/tcg/cpu-mmu-index.h"
#include "exec/target_page.h"
#include "exec/translator.h"
+#ifdef CONFIG_USER_ONLY
+#include "gdbstub/user.h"
+#endif
#include "exec/plugin-gen.h"
#include "tcg/tcg-op-common.h"
#include "internal-common.h"
@@ -110,6 +113,34 @@ bool translator_is_same_page(const DisasContextBase *db, vaddr addr)
return ((addr ^ db->pc_first) & TARGET_PAGE_MASK) == 0;
}
+/*
+ * Whether a direct jump may be chained to a destination outside the page
+ * the TB started in.
+ *
+ * In user-only mode there are no page tables. Every mmap, mprotect and
+ * munmap goes through page_set_flags(), which calls tb_invalidate_phys_range()
+ * whenever the flags actually change, and tb_phys_invalidate() unlinks
+ * incoming jumps. A cross-page link is therefore broken whenever the
+ * destination page's permissions change.
+ *
+ * What the same-page rule also provides is that execution cannot enter a page
+ * without a TB lookup, and so without check_for_breakpoints(), which is what
+ * makes a breakpoint set after a block was translated take effect. Nothing
+ * invalidates on breakpoint insertion, so a link established beforehand would
+ * jump straight over it. In user-only mode breakpoints only ever come from
+ * gdb -- BP_CPU is g_assert_not_reached() there and the guest has no way to
+ * ask for one -- and gdb has to be requested with -g before the first block
+ * is translated, so a run that has no gdbstub can never acquire a breakpoint.
+ */
+static bool use_cross_page_goto_tb(void)
+{
+#ifdef CONFIG_USER_ONLY
+ return !gdb_may_set_breakpoints();
+#else
+ return false;
+#endif
+}
+
bool translator_use_goto_tb(DisasContextBase *db, vaddr dest)
{
/* Suppress goto_tb if requested. */
@@ -118,7 +149,7 @@ bool translator_use_goto_tb(DisasContextBase *db, vaddr dest)
}
/* Check for the dest on the same page as the start of the TB. */
- return translator_is_same_page(db, dest);
+ return use_cross_page_goto_tb() || translator_is_same_page(db, dest);
}
void translator_loop(CPUState *cpu, TranslationBlock *tb, int *max_insns,
diff --git ./gdbstub/user.c ./gdbstub/user.c
index 9e6f9a6f37..d810f0f38c 100644
--- ./gdbstub/user.c
+++ ./gdbstub/user.c
@@ -470,6 +470,18 @@ static void *gdbserver_accept_thread(void *arg)
#define USAGE "\nUsage: -g {port|path}[,suspend={y|n}]"
+/*
+ * Set before the guest runs and never cleared, so that code translated at
+ * any point can rely on it: with suspend=n gdb may connect long after
+ * startup, and once connected it can insert a breakpoint at any time.
+ */
+static bool gdbserver_requested;
+
+bool gdb_may_set_breakpoints(void)
+{
+ return gdbserver_requested;
+}
+
bool gdbserver_start(const char *args, Error **errp)
{
g_auto(GStrv) argv = g_strsplit(args, ",", 0);
@@ -513,6 +525,8 @@ bool gdbserver_start(const char *args, Error **errp)
return false;
}
+ gdbserver_requested = true;
+
if (suspend) {
if (gdbserver_accept(port, gdb_fd, port_or_path)) {
gdb_handlesig(first_cpu, 0, NULL, NULL, 0);
diff --git ./include/gdbstub/user.h ./include/gdbstub/user.h
index 654986d483..c091cd9758 100644
--- ./include/gdbstub/user.h
+++ ./include/gdbstub/user.h
@@ -11,6 +11,17 @@
#define MAX_SIGINFO_LENGTH 128
+/**
+ * gdb_may_set_breakpoints() - whether a breakpoint can ever be inserted
+ *
+ * In user-only mode every breakpoint comes from gdb, and gdb is only ever
+ * reachable if -g was given at startup, before the guest ran a single
+ * instruction. A run that has no gdbstub can therefore never acquire a
+ * breakpoint, which lets translation take shortcuts that a breakpoint
+ * would invalidate. Stays true once true, even if gdb detaches.
+ */
+bool gdb_may_set_breakpoints(void);
+
/**
* gdb_handlesig() - yield control to gdb
* @cpu: CPU
diff --git ./tests/tcg/multiarch/Makefile.target ./tests/tcg/multiarch/Makefile.target
index ab4bf9c5d5..f8a91fed2c 100644
--- ./tests/tcg/multiarch/Makefile.target
+++ ./tests/tcg/multiarch/Makefile.target
@@ -143,6 +143,15 @@ run-gdbstub-follow-fork-mode-parent: follow-fork-mode
--bin $< --test $(MULTIARCH_SRC)/gdbstub/follow-fork-mode-parent.py, \
following parents on fork)
+# The chaining this exercises is only enabled when no gdbstub was requested,
+# so what is under test here is that requesting one turns it back off.
+run-gdbstub-xpage-bp: test-xpage-chain
+ $(call run-test, $@, $(GDB_SCRIPT) \
+ --gdb $(GDB) \
+ --qemu $(QEMU) --qargs "$(QEMU_OPTS)" \
+ --bin "$< -b" --test $(MULTIARCH_SRC)/gdbstub/xpage-bp.py, \
+ breakpoint behind an established cross-page chain)
+
run-gdbstub-late-attach: late-attach
$(call run-test, $@, env LATE_ATTACH_PY=1 $(GDB_SCRIPT) \
--gdb $(GDB) \
@@ -159,7 +168,8 @@ EXTRA_RUNS += run-gdbstub-sha1 run-gdbstub-qxfer-auxv-read \
run-gdbstub-registers run-gdbstub-prot-none \
run-gdbstub-catch-syscalls run-gdbstub-follow-fork-mode-child \
run-gdbstub-follow-fork-mode-parent \
- run-gdbstub-qxfer-siginfo-read run-gdbstub-late-attach
+ run-gdbstub-qxfer-siginfo-read run-gdbstub-late-attach \
+ run-gdbstub-xpage-bp
# ARM Compatible Semi Hosting Tests
#
diff --git ./tests/tcg/multiarch/gdbstub/xpage-bp.py ./tests/tcg/multiarch/gdbstub/xpage-bp.py
new file mode 100644
index 0000000000..f40024f16d
--- /dev/null
+++ ./tests/tcg/multiarch/gdbstub/xpage-bp.py
@@ -0,0 +1,37 @@
+"""Test that a breakpoint set after a cross-page chain is established is hit.
+
+translator_use_goto_tb() lets a direct branch chain to another page in
+user-only builds, which is only safe because a run with no gdbstub can never
+acquire a breakpoint. This runs with one, so the chaining must be off and
+the breakpoint must still be reached.
+
+This runs as a sourced script (via -x, via run-test.py).
+
+SPDX-License-Identifier: GPL-2.0-or-later
+"""
+from test_gdbstub import main, report
+
+
+def run_test():
+ """Run through the tests one by one"""
+ gdb.Breakpoint("break_here")
+ gdb.execute("continue")
+
+ # The chain exists by now; put a breakpoint on the far side of it.
+ target = int(gdb.parse_and_eval("(unsigned long)page_b_entry"))
+ if target == 0:
+ report(True, "no code emitters for this architecture, skipped")
+ return
+ gdb.execute("break *{}".format(target))
+ gdb.execute("continue")
+
+ pc = int(gdb.parse_and_eval("(unsigned long)$pc"))
+ report(pc == target, "stopped at {:#x}, expected {:#x}".format(pc, target))
+
+ gdb.execute("delete")
+ gdb.execute("continue")
+ exitcode = int(gdb.parse_and_eval("$_exitcode"))
+ report(exitcode == 0, "{} == 0".format(exitcode))
+
+
+main(run_test)
diff --git ./tests/tcg/multiarch/test-xpage-chain.c ./tests/tcg/multiarch/test-xpage-chain.c
new file mode 100644
index 0000000000..a4e34149e7
--- /dev/null
+++ ./tests/tcg/multiarch/test-xpage-chain.c
@@ -0,0 +1,336 @@
+/*
+ * Cross-page TB chaining hazard test.
+ *
+ * Two adjacent pages of hand-written code. The last instruction of page A
+ * sets the return value and falls through into page B, which returns; a TB
+ * always ends at a page boundary, so page A reaches page B through a
+ * cross-page goto_tb.
+ *
+ * Phase 1: run it enough times that QEMU chains TB_A -> TB_B.
+ * Phase 2: mprotect page B away. Re-running must fault.
+ * Phase 3: map it back and write different code into it. Re-running must
+ * execute the NEW code, not a stale chained translation.
+ *
+ * With -b, phases 2 and 3 are replaced by a stop at break_here(), where the
+ * gdbstub test sets a breakpoint on page B -- after the chain exists -- and
+ * checks that re-running the chain still stops on it. See
+ * tests/tcg/multiarch/gdbstub/xpage-bp.py.
+ *
+ * The code the two pages hold is architecture specific, so each
+ * architecture supplies two emitters:
+ *
+ * emit_set_ret(p, val) - set the integer return value register to val
+ * emit_ret(p) - return to the caller
+ *
+ * both writing at @p and returning the number of bytes written. Neither
+ * may contain a branch: the fall-through from page A into page B is the
+ * whole point, and a delay slot must not straddle the boundary. An
+ * architecture that supplies neither skips the test.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+#include <stdbool.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <setjmp.h>
+#include <signal.h>
+#include <stdint.h>
+#include <sys/mman.h>
+#include <unistd.h>
+
+static inline size_t put32(void *p, uint32_t insn)
+{
+ memcpy(p, &insn, sizeof(insn));
+ return sizeof(insn);
+}
+
+static inline size_t put16(void *p, uint16_t insn)
+{
+ memcpy(p, &insn, sizeof(insn));
+ return sizeof(insn);
+}
+
+#if defined(__aarch64__)
+#define HAVE_EMITTERS
+/* movz w0, #val */
+static size_t emit_set_ret(void *p, int val)
+{
+ return put32(p, 0x52800000u | ((uint32_t)val << 5));
+}
+static size_t emit_ret(void *p)
+{
+ return put32(p, 0xd65f03c0u); /* ret */
+}
+#elif defined(__alpha__)
+#define HAVE_EMITTERS
+/* lda $0, val($31) */
+static size_t emit_set_ret(void *p, int val)
+{
+ return put32(p, 0x201f0000u | (uint16_t)val);
+}
+static size_t emit_ret(void *p)
+{
+ return put32(p, 0x6bfa8001u); /* ret */
+}
+#elif defined(__arm__)
+#define HAVE_EMITTERS
+/* mov r0, #val */
+static size_t emit_set_ret(void *p, int val)
+{
+ return put32(p, 0xe3a00000u | (uint8_t)val);
+}
+static size_t emit_ret(void *p)
+{
+ return put32(p, 0xe12fff1eu); /* bx lr */
+}
+#elif defined(__hppa__)
+#define HAVE_EMITTERS
+/* ldi val, %ret0 */
+static size_t emit_set_ret(void *p, int val)
+{
+ return put32(p, 0x341c0000u | ((uint32_t)val << 1));
+}
+static size_t emit_ret(void *p)
+{
+ size_t n = put32(p, 0xe840c000u); /* bv %r0(%rp) */
+ return n + put32((char *)p + n, 0x08000240u); /* nop (delay slot) */
+}
+#elif defined(__i386__) || defined(__x86_64__)
+#define HAVE_EMITTERS
+/* mov $val, %eax */
+static size_t emit_set_ret(void *p, int val)
+{
+ uint32_t imm = val;
+ *(unsigned char *)p = 0xb8;
+ return 1 + put32((char *)p + 1, imm);
+}
+static size_t emit_ret(void *p)
+{
+ *(unsigned char *)p = 0xc3; /* ret */
+ return 1;
+}
+#elif defined(__loongarch64)
+#define HAVE_EMITTERS
+/* ori $a0, $zero, val */
+static size_t emit_set_ret(void *p, int val)
+{
+ return put32(p, 0x03800004u | ((uint32_t)val << 10));
+}
+static size_t emit_ret(void *p)
+{
+ return put32(p, 0x4c000020u); /* jr $ra */
+}
+#elif defined(__m68k__)
+#define HAVE_EMITTERS
+/* moveq #val, %d0 */
+static size_t emit_set_ret(void *p, int val)
+{
+ return put16(p, 0x7000u | (uint8_t)val);
+}
+static size_t emit_ret(void *p)
+{
+ return put16(p, 0x4e75u); /* rts */
+}
+#elif defined(__mips__)
+#define HAVE_EMITTERS
+/* li $v0, val */
+static size_t emit_set_ret(void *p, int val)
+{
+ return put32(p, 0x24020000u | (uint16_t)val);
+}
+static size_t emit_ret(void *p)
+{
+ size_t n = put32(p, 0x03e00008u); /* jr $ra */
+ return n + put32((char *)p + n, 0x00000000u); /* nop (delay slot) */
+}
+/*
+ * ELFv1 function pointers are descriptors rather than code addresses, so
+ * there is nothing to call the raw code through.
+ */
+#elif defined(__powerpc__) && \
+ (!defined(__powerpc64__) || (defined(_CALL_ELF) && _CALL_ELF == 2))
+#define HAVE_EMITTERS
+/* li r3, val */
+static size_t emit_set_ret(void *p, int val)
+{
+ return put32(p, 0x38600000u | (uint16_t)val);
+}
+static size_t emit_ret(void *p)
+{
+ return put32(p, 0x4e800020u); /* blr */
+}
+#elif defined(__riscv)
+#define HAVE_EMITTERS
+/* addi a0, zero, val -- the 4 byte form, never c.li */
+static size_t emit_set_ret(void *p, int val)
+{
+ return put32(p, 0x00000513u | ((uint32_t)val << 20));
+}
+static size_t emit_ret(void *p)
+{
+ return put32(p, 0x00008067u); /* jalr zero, 0(ra) */
+}
+#elif defined(__s390x__)
+#define HAVE_EMITTERS
+/* lghi %r2, val */
+static size_t emit_set_ret(void *p, int val)
+{
+ size_t n = put16(p, 0xa729u);
+ return n + put16((char *)p + n, (uint16_t)val);
+}
+static size_t emit_ret(void *p)
+{
+ return put16(p, 0x07feu); /* br %r14 */
+}
+#elif defined(__sh__)
+#define HAVE_EMITTERS
+/* mov #val, r0 */
+static size_t emit_set_ret(void *p, int val)
+{
+ return put16(p, 0xe000u | (uint8_t)val);
+}
+static size_t emit_ret(void *p)
+{
+ size_t n = put16(p, 0x000bu); /* rts */
+ return n + put16((char *)p + n, 0x0009u); /* nop (delay slot) */
+}
+#elif defined(__sparc__)
+#define HAVE_EMITTERS
+/* mov val, %o0 */
+static size_t emit_set_ret(void *p, int val)
+{
+ return put32(p, 0x90102000u | (uint32_t)(val & 0x1fff));
+}
+static size_t emit_ret(void *p)
+{
+ size_t n = put32(p, 0x81c3e008u); /* retl */
+ return n + put32((char *)p + n, 0x01000000u); /* nop (delay slot) */
+}
+#endif
+
+/* Where the fall-through lands, for the gdbstub test to breakpoint on. */
+void *page_b_entry;
+
+/* Somewhere for the gdbstub test to stop once the chain is established. */
+void __attribute__((noinline)) break_here(void)
+{
+ asm volatile ("");
+}
+
+#ifdef HAVE_EMITTERS
+static sigjmp_buf jb;
+/*
+ * Written by the SIGSEGV handler and read by main(), so it must not be
+ * cached in a register across the faulting call.
+ */
+static volatile sig_atomic_t caught;
+
+static void segv(int sig)
+{
+ caught = 1;
+ siglongjmp(jb, 1);
+}
+#endif
+
+int main(int argc, char **argv)
+{
+ bool bp_mode = argc > 1 && strcmp(argv[1], "-b") == 0;
+#ifndef HAVE_EMITTERS
+ printf("SKIP: no code emitters for this architecture\n");
+ if (bp_mode) {
+ break_here();
+ }
+ return 0;
+#else
+ unsigned char tmp[16];
+ struct sigaction sa;
+ long (*fn)(void);
+ size_t setlen, n;
+ long ps = sysconf(_SC_PAGESIZE);
+ int rc = 0;
+ unsigned char *m = mmap(NULL, 2 * ps, PROT_READ | PROT_WRITE | PROT_EXEC,
+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ if (m == MAP_FAILED) {
+ perror("mmap");
+ return 2;
+ }
+
+ unsigned char *pb = m + ps;
+
+ /*
+ * Page A ends with the store to the return value register, so that the
+ * next instruction executed is the first one on page B.
+ */
+ setlen = emit_set_ret(tmp, 1);
+ memcpy(pb - setlen, tmp, setlen);
+ emit_ret(pb);
+ __builtin___clear_cache((char *)m, (char *)m + 2 * ps);
+
+ page_b_entry = pb;
+ fn = (long (*)(void))(pb - setlen);
+
+ for (int i = 0; i < 200000; i++) {
+ if (fn() != 1) {
+ printf("FAIL: phase 1 wrong result\n");
+ return 1;
+ }
+ }
+ printf("phase 1 ok (chained)\n");
+
+ if (bp_mode) {
+ /*
+ * The chain from page A to page B now exists. gdb puts a breakpoint
+ * on page_b_entry here; the call below has to stop on it rather than
+ * jump over it.
+ */
+ break_here();
+ if (fn() != 1) {
+ printf("FAIL: bp phase wrong result\n");
+ return 1;
+ }
+ printf("bp phase ok\n");
+ return 0;
+ }
+
+ memset(&sa, 0, sizeof(sa));
+ sa.sa_handler = segv;
+ sigemptyset(&sa.sa_mask);
+ if (sigaction(SIGSEGV, &sa, NULL) != 0) {
+ perror("sigaction");
+ return 2;
+ }
+ if (mprotect(pb, ps, PROT_NONE) != 0) {
+ perror("mprotect");
+ return 2;
+ }
+ if (sigsetjmp(jb, 1) == 0) {
+ fn();
+ printf("FAIL: phase 2 executed page B after mprotect(PROT_NONE)\n");
+ rc = 1;
+ } else if (!caught) {
+ printf("FAIL: phase 2 longjmp without entering the handler\n");
+ rc = 1;
+ } else {
+ printf("phase 2 ok (faulted)\n");
+ }
+
+ /* Phase 3: map back, overwrite, expect the new code to run. */
+ if (mprotect(pb, ps, PROT_READ | PROT_WRITE | PROT_EXEC) != 0) {
+ perror("mprotect back");
+ return 2;
+ }
+ n = emit_set_ret(pb, 2);
+ emit_ret(pb + n);
+ __builtin___clear_cache((char *)pb, (char *)pb + ps);
+
+ long r = fn();
+ if (r != 2) {
+ printf("FAIL: phase 3 returned %ld, expected 2 (stale chain)\n", r);
+ rc = 1;
+ } else {
+ printf("phase 3 ok (new code ran)\n");
+ }
+ return rc;
+#endif
+}
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH v4 8/9] RFC: accel/tcg: poison the jump cache instead of polling for indirect exits
2026-08-22 19:08 [PATCH v3 0/7] accel/tcg: cut per-block dispatch overhead Matt Turner
` (14 preceding siblings ...)
2026-08-27 5:02 ` [PATCH v4 7/9] RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds Matt Turner
@ 2026-08-27 5:02 ` Matt Turner
2026-08-27 5:02 ` [PATCH v4 9/9] RFC: tcg: fold a guest displacement into the host addressing mode Matt Turner
16 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-08-27 5:02 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
Every translation block begins by loading cpu->neg.icount_decr.u32, testing
it and branching to the exit path. That is three host instructions at the top
of every TB, and blocks are short: an emulated alpha gcc 16.2.0 compiling a
255k line translation unit executes 34.2 billion of them at 6.04 guest
instructions each.
A block does not need to poll if every way out of it already reaches a check.
A goto_tb does not: it chains straight into its destination, with nothing in
between that looks at icount_decr, so the destination has to poll on entry.
An indirect exit does. The out-of-line path calls helper_lookup_tb_ptr()
every time, so it only needs the helper to return the epilogue while an exit
is pending. The inline probe is already covered by the machinery the
breakpoint patch added: it reads its base pointer from
cpu->tb_jmp_cache_probe and takes the slow path when the entry it finds has a
NULL tb, so pointing that base at a page of zeroes turns every indirect
dispatch into a miss, and a miss lands in the same helper.
So a pending exit becomes one more reason for tcg_cpu_may_dispatch() to say
no. The two places that set icount_decr.u16.high poison the probe; the main
loop puts it back once the flag is clear, on the same pass that already
re-evaluates the breakpoint state. The real tb_jmp_cache is untouched
throughout, so no cache contents are lost, and the fast path pays nothing:
the base was a load from CPUState either way.
The poll is therefore emitted only in blocks that emit a goto_tb. Whether a
block does is not known until its last exit has been generated, so the
decision is deferred and the load and branch are emitted retroactively at the
head of the block in gen_tb_end(), using the same emit_before_op mechanism
the can_do_io stores use. icount opts out and keeps the counter
unconditionally.
Interrupt latency is bounded at one block, as before. It does not depend on
the shape of the guest's control flow graph: a block either polls on entry or
is checked on the way out, and no run of blocks can avoid both. What changes
is where the check sits, not how often one happens.
tests/tcg/multiarch/test-indirect-irq.c is added for this: a loop whose only
back edge is an indirect branch, under alarm(1). That loop's block emits no
goto_tb, so it no longer polls, and the test passes only because the dispatch
notices instead -- it hangs if the poison is removed, which is what makes it a
test of the new mechanism rather than of the old poll. Nothing in it is
architecture specific: the loop is a computed goto, which every target's
compiler supports, so it covers whichever targets go on to use the inline
probe. The other alpha tests still pass and the emulated compiler still
produces byte-identical output.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the
SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, LTO build, on
top of the preceding patches:
before: 891,254,240,071 instructions
after: 868,811,832,620 instructions -2.52%
before: 81.45s wall clock
after: 79.85s wall clock -1.96%
The emulated compiler produces byte-identical output.
RFC because:
- The un-poison in the main loop races a concurrent poison from another
thread. The existing barrier around icount_decr.u16.high covers it -- a
poison that lands after the sync also re-set the flag, and exit_request was
stored before it -- but this deserves more eyes than the single-threaded
user-mode testing I have given it.
- Only the inline probe needs the poison, and only alpha uses the inline
probe today. Targets on the out-of-line path are covered by the helper
check alone, but that has not been measured.
- The shared zero-filled CPUJumpCache is a 1MB allocation that is never
written. A read-only mapping would express that better.
v3: Rebased onto the removal of "only poll for interrupts in blocks that can
close a cycle", which v2 sat on top of and which is dropped: it let a
straight-line run of arbitrary length go unchecked, since a block with no
backward edge polled nowhere (Richard).
The rule is now that a block polls iff it emits a goto_tb, rather than
iff it can close a control flow cycle. That keeps the bound at one block
without any analysis of the guest's control flow graph, so the objection
to the dropped patch does not carry over. The deferred-emission machinery
it needs moves here from that patch; DisasContextBase::needs_exit_check
and the hook in translator_use_goto_tb() are gone with it, and the flag
is now set by tcg_gen_goto_tb() rather than by goto_ptr emission.
All of v2's measurements were dropped: they were taken with the
cycle-analysis patch underneath, which changes both the baseline and
what is left to remove, so none of them described this patch. The
numbers above are a fresh measurement of the series as it now stands.
v4: Moved the test from tests/tcg/alpha/ to tests/tcg/multiarch/: the
mechanism is generic and nothing in the test is alpha specific (Alex).
The performance numbers above are the v3 measurements, not re-run: the
machine they were taken on is busy.
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
accel/tcg/cpu-exec.c | 23 +++++++--
accel/tcg/tcg-accel-ops.c | 1 +
accel/tcg/translator.c | 51 ++++++++++++++++++--
include/hw/core/cpu.h | 8 ++--
include/tcg/tcg.h | 2 +
tcg/tcg-op.c | 13 ++++++
tests/tcg/multiarch/test-indirect-irq.c | 62 +++++++++++++++++++++++++
7 files changed, 151 insertions(+), 9 deletions(-)
create mode 100644 tests/tcg/multiarch/test-indirect-irq.c
diff --git ./accel/tcg/cpu-exec.c ./accel/tcg/cpu-exec.c
index c2a9679cd7..9ca5894108 100644
--- ./accel/tcg/cpu-exec.c
+++ ./accel/tcg/cpu-exec.c
@@ -388,6 +388,16 @@ const void *HELPER(lookup_tb_ptr)(CPUArchState *env)
*/
cpu->neg.can_do_io = true;
+ /*
+ * A block that dispatches indirectly does not emit the icount_decr poll,
+ * so this is where a pending exit is noticed for that path: either the
+ * probe was poisoned and every dispatch arrives here, or the target uses
+ * the out-of-line lookup and always did.
+ */
+ if (unlikely(cpu_loop_exit_requested(cpu))) {
+ return tcg_code_gen_epilogue;
+ }
+
TCGTBCPUState s = cpu->cc->tcg_ops->get_tb_cpu_state(cpu);
s.cflags = curr_cflags(cpu);
@@ -757,8 +767,9 @@ static inline bool cpu_handle_exception(CPUState *cpu, int *ret)
* slow path when the entry it finds has a NULL tb. Pointing the probe at a
* region that is all zeroes therefore forces every indirect dispatch into
* helper_lookup_tb_ptr(), which does the full lookup the inline probe only
- * approximates. The real jump cache is untouched, so no contents are lost
- * and recovery is a single store.
+ * approximates and returns to the main loop while an exit is pending. The
+ * real jump cache is untouched, so no contents are lost and recovery is a
+ * single store.
*
* Only ever read from, and only one entry per dispatch, so one shared
* zero-filled cache is enough for every CPU. Not const: that would put a
@@ -779,10 +790,13 @@ static CPUJumpCache tb_jmp_cache_poison;
* the rest of the page. A block translated before the breakpoint was set is
* therefore still in the jump cache, and dispatching to it inline would step
* straight over the breakpoint.
+ *
+ * A block that dispatches indirectly also does not emit the icount_decr
+ * poll, so the dispatch is where a pending exit has to be noticed.
*/
static bool tcg_cpu_may_dispatch(CPUState *cpu)
{
- return QTAILQ_EMPTY(&cpu->breakpoints);
+ return QTAILQ_EMPTY(&cpu->breakpoints) && !cpu_loop_exit_requested(cpu);
}
/*
@@ -851,6 +865,9 @@ void tcg_kick_vcpu_thread(CPUState *cpu)
/* Ensure cpu_exec will see the exit request after TCG has exited. */
qatomic_store_release(&cpu->neg.icount_decr.u16.high, -1);
+
+ /* Blocks that only dispatch indirectly do not poll; stop them chaining. */
+ tcg_cpu_poison_jmp_cache(cpu);
}
static inline bool icount_exit_request(CPUState *cpu)
diff --git ./accel/tcg/tcg-accel-ops.c ./accel/tcg/tcg-accel-ops.c
index 560fe2554b..9eb9e861ac 100644
--- ./accel/tcg/tcg-accel-ops.c
+++ ./accel/tcg/tcg-accel-ops.c
@@ -106,6 +106,7 @@ void tcg_handle_interrupt(CPUState *cpu, int mask)
qemu_cpu_kick(cpu);
} else {
qatomic_set(&cpu->neg.icount_decr.u16.high, -1);
+ tcg_cpu_poison_jmp_cache(cpu);
}
}
diff --git ./accel/tcg/translator.c ./accel/tcg/translator.c
index 8879cd626f..89d255bd04 100644
--- ./accel/tcg/translator.c
+++ ./accel/tcg/translator.c
@@ -45,12 +45,35 @@ bool translator_io_start(DisasContextBase *db)
return true;
}
+/*
+ * A block that ends in a goto_tb chains straight to its destination: nothing
+ * between the two looks at icount_decr, so the destination has to poll on
+ * entry. A block whose exits are all indirect does not, because the dispatch
+ * itself notices -- a pending exit poisons tb_jmp_cache_probe, so the probe
+ * misses into helper_lookup_tb_ptr(), which returns the epilogue. Every block
+ * therefore either polls on entry or is checked as it leaves, which bounds
+ * interrupt latency at one block without looking at the shape of the guest's
+ * control flow graph.
+ *
+ * Which kind a block is is not known until its last exit has been emitted, so
+ * defer the decision to gen_tb_end() and emit the poll retroactively.
+ *
+ * icount needs the counter unconditionally, so it opts out.
+ */
+static bool defer_exit_check(uint32_t cflags)
+{
+ return !(cflags & CF_USE_ICOUNT);
+}
+
static TCGOp *gen_tb_start(DisasContextBase *db, uint32_t cflags)
{
TCGv_i32 count = NULL;
TCGOp *icount_start_insn = NULL;
- if ((cflags & CF_USE_ICOUNT) || !(cflags & CF_NOIRQ)) {
+ tcg_ctx->exit_check_needed = false;
+
+ if ((cflags & CF_USE_ICOUNT) ||
+ (!(cflags & CF_NOIRQ) && !defer_exit_check(cflags))) {
count = tcg_temp_new_i32();
tcg_gen_ld_i32(count, tcg_env,
offsetof(CPUState, neg.icount_decr.u32) -
@@ -76,6 +99,9 @@ static TCGOp *gen_tb_start(DisasContextBase *db, uint32_t cflags)
*/
if (cflags & CF_NOIRQ) {
tcg_ctx->exitreq_label = NULL;
+ } else if (defer_exit_check(cflags)) {
+ /* Emitted retroactively by gen_tb_end(), if this TB emits a goto_tb. */
+ tcg_ctx->exitreq_label = gen_new_label();
} else {
tcg_ctx->exitreq_label = gen_new_label();
tcg_gen_brcondi_i32(TCG_COND_LT, count, 0, tcg_ctx->exitreq_label);
@@ -91,7 +117,8 @@ static TCGOp *gen_tb_start(DisasContextBase *db, uint32_t cflags)
}
static void gen_tb_end(const TranslationBlock *tb, uint32_t cflags,
- TCGOp *icount_start_insn, int num_insns)
+ TCGOp *icount_start_insn, int num_insns,
+ TCGOp *first_insn_start)
{
if (cflags & CF_USE_ICOUNT) {
/*
@@ -102,6 +129,23 @@ static void gen_tb_end(const TranslationBlock *tb, uint32_t cflags,
tcgv_i32_arg(tcg_constant_i32(num_insns)));
}
+ if (tcg_ctx->exitreq_label && defer_exit_check(cflags) &&
+ !(cflags & CF_NOIRQ)) {
+ if (tcg_ctx->exit_check_needed) {
+ TCGv_i32 count = tcg_temp_new_i32();
+ TCGOp *save = tcg_ctx->emit_before_op;
+
+ tcg_ctx->emit_before_op = first_insn_start;
+ tcg_gen_ld_i32(count, tcg_env,
+ offsetof(CPUState, neg.icount_decr.u32) -
+ sizeof(CPUState));
+ tcg_gen_brcondi_i32(TCG_COND_LT, count, 0, tcg_ctx->exitreq_label);
+ tcg_ctx->emit_before_op = save;
+ } else {
+ tcg_ctx->exitreq_label = NULL;
+ }
+ }
+
if (tcg_ctx->exitreq_label) {
gen_set_label(tcg_ctx->exitreq_label);
tcg_gen_exit_tb(tb, TB_EXIT_REQUESTED);
@@ -238,7 +282,8 @@ void translator_loop(CPUState *cpu, TranslationBlock *tb, int *max_insns,
/* Emit code to exit the TB, as indicated by db->is_jmp. */
ops->tb_stop(db, cpu);
- gen_tb_end(tb, cflags, icount_start_insn, db->num_insns);
+ gen_tb_end(tb, cflags, icount_start_insn, db->num_insns,
+ first_insn_start);
/*
* Manage can_do_io for the translation block: set to false before
diff --git ./include/hw/core/cpu.h ./include/hw/core/cpu.h
index bd2cdd2a0b..4272740303 100644
--- ./include/hw/core/cpu.h
+++ ./include/hw/core/cpu.h
@@ -523,9 +523,11 @@ struct CPUState {
* @tb_jmp_cache_probe: base the inline jump cache probe reads.
*
* Normally @tb_jmp_cache. Pointed at a shared page of zeroes to force
- * every inline dispatch to miss and fall back to helper_lookup_tb_ptr();
- * see tcg_cpu_sync_jmp_cache(). NULL before tcg_exec_realizefn() and
- * after tcg_exec_unrealizefn().
+ * every inline dispatch to miss and fall back to helper_lookup_tb_ptr(),
+ * either because a breakpoint is set or because an exit is pending; see
+ * tcg_cpu_sync_jmp_cache(). Only generated code and the accessors in
+ * cpu-exec.c may touch it. NULL before tcg_exec_realizefn() and after
+ * tcg_exec_unrealizefn().
*/
struct CPUJumpCache *tb_jmp_cache_probe;
diff --git ./include/tcg/tcg.h ./include/tcg/tcg.h
index 7669dc1c2d..df08c10544 100644
--- ./include/tcg/tcg.h
+++ ./include/tcg/tcg.h
@@ -389,6 +389,8 @@ struct TCGContext {
struct TCGLabelPoolData *pool_labels;
TCGLabel *exitreq_label;
+ /* Set by goto_tb emission: this TB chains without reaching a check. */
+ bool exit_check_needed;
#ifdef CONFIG_PLUGIN
/*
diff --git ./tcg/tcg-op.c ./tcg/tcg-op.c
index cf7b6882d8..d384325a2e 100644
--- ./tcg/tcg-op.c
+++ ./tcg/tcg-op.c
@@ -2713,6 +2713,13 @@ void tcg_gen_goto_tb(unsigned idx)
tcg_debug_assert((tcg_ctx->goto_tb_issue_mask & (1 << idx)) == 0);
tcg_ctx->goto_tb_issue_mask |= 1 << idx;
#endif
+ /*
+ * A goto_tb chains straight into the destination, with nothing in between
+ * that looks at icount_decr, so this TB has to poll on entry. See
+ * defer_exit_check().
+ */
+ tcg_ctx->exit_check_needed = true;
+
plugin_gen_disable_mem_helpers();
tcg_gen_op1i(INDEX_op_goto_tb, 0, idx);
}
@@ -2834,6 +2841,12 @@ void tcg_gen_lookup_and_goto_ptr_tmp(TCGTemp *pc, const TranslationBlock *tb)
plugin_gen_disable_mem_helpers();
+ /*
+ * Neither path below needs an icount_decr poll. The helper returns to
+ * the main loop while an exit is pending, and a pending exit poisons
+ * tb_jmp_cache_probe, so the inline probe finds a NULL tb and falls into
+ * that same helper.
+ */
if (pc) {
TCGv_i64 pc64;
diff --git ./tests/tcg/multiarch/test-indirect-irq.c ./tests/tcg/multiarch/test-indirect-irq.c
new file mode 100644
index 0000000000..a672faf641
--- /dev/null
+++ ./tests/tcg/multiarch/test-indirect-irq.c
@@ -0,0 +1,62 @@
+/*
+ * A loop whose only back edge is an indirect branch must still be
+ * interruptible.
+ *
+ * Blocks that dispatch indirectly do not emit the icount_decr poll; a pending
+ * exit instead poisons the inline jump cache probe so that the dispatch falls
+ * into helper_lookup_tb_ptr(), which returns to the main loop. If that
+ * mechanism breaks, this program never leaves the loop and the test times
+ * out rather than failing an assertion.
+ *
+ * A computed goto is used deliberately: a plain while(1) would end the block
+ * with a direct backward branch, that is a goto_tb, and a block that emits a
+ * goto_tb still polls -- so it would not exercise the path under test.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+#include <assert.h>
+#include <signal.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+/* Written by the handler, read by the loop, so it must not be cached. */
+static volatile sig_atomic_t fired;
+/* Read after the loop, so the loop must not optimize the increment away. */
+static volatile unsigned long iterations;
+
+static void handler(int sig)
+{
+ fired = 1;
+}
+
+int main(void)
+{
+ /*
+ * Indexing a table with a volatile index, rather than jumping through a
+ * volatile pointer: gcc happily proves a single-valued pointer constant
+ * and emits a direct branch, which is the case this test is not about.
+ */
+ volatile int idx = 0;
+ void *target[2];
+ struct sigaction sa;
+
+ memset(&sa, 0, sizeof(sa));
+ sa.sa_handler = handler;
+ sigemptyset(&sa.sa_mask);
+ assert(sigaction(SIGALRM, &sa, NULL) == 0);
+ alarm(1);
+
+ target[0] = &&spin;
+ target[1] = &&out;
+spin:
+ iterations++;
+ if (!fired) {
+ goto *target[idx];
+ }
+out:
+
+ printf("interrupted after %lu iterations\n", iterations);
+ return 0;
+}
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH v4 9/9] RFC: tcg: fold a guest displacement into the host addressing mode
2026-08-22 19:08 [PATCH v3 0/7] accel/tcg: cut per-block dispatch overhead Matt Turner
` (15 preceding siblings ...)
2026-08-27 5:02 ` [PATCH v4 8/9] RFC: accel/tcg: poison the jump cache instead of polling for indirect exits Matt Turner
@ 2026-08-27 5:02 ` Matt Turner
16 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-08-27 5:02 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
Nothing in the TCG frontend interface can express a based memory access.
tcg_gen_qemu_ld/st take an address and nothing else, so a target with a
displacement in its load and store encodings -- which is most of them --
has to materialize the address first:
ldq a1,8(a0) -> mov 0x80(%rbp),%rbx reload a0
lea 0x8(%rbx),%r12 address
mov (%r12),%r12 the load
mov %r12,0x88(%rbp) spill a1
The lea is pure loss on a host whose addressing mode has a displacement
field sitting empty. It also needs a register, at the point in a block
where pressure is highest.
Fold it. After optimization, look for an add of a constant immediately
before a guest access, defining that access's address operand, and move the
constant into a new second constant argument on the op. The add is left for
liveness to remove, so nothing breaks if its result has another use. Only
the immediately preceding op is examined: that is what the frontends emit,
and a window of one op means the pass does not have to reason about what
could have happened in between. The one thing it does check is that the add
did not clobber the base it read, since the access now reads that base
directly.
Targets opt in with TCG_TARGET_HAS_ldst_disp and an out_disp member on
TCGOutOpQemuLdSt. Without it the pass does not run, the displacement stays
zero and the existing out member is called exactly as before, so no other
backend changes behavior or needs touching.
The fold is refused unless the access has no slow path at all, since the
slow path hands addr_reg to the helper and that register no longer holds
the full guest address. That is decided generically: user-only, because
softmmu compares the unadjusted address against the TLB; a 64-bit address
type, because a 32-bit one wraps where a host displacement would not; and
no alignment test on the access. For x86_64 the displacement goes in the
disp32 that prepare_host_addr() already fills in for guest_base, so all the
backend has left to check is that guest_base plus the displacement still
fits there.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the
SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, LTO build,
on top of the preceding patches, against a control measured in the same
session:
before: 868,811,832,620 instructions, 79.85s
after: 819,262,147,022 instructions, 77.30s
-5.70% instructions, -3.20% wall
Emitted code shrinks from 50.55MB to 48.80MB over the run, 167.4 to 161.6
bytes per block. Per Alpha opcode, the host bytes emitted for an access
fall as expected and nothing else moves:
ldq 18.3 -> 15.4 ldah 20.9 -> 20.9
ldl 16.6 -> 14.1 lda 12.9 -> 12.9
stq 12.8 -> 9.7 mov 9.8 -> 9.8
The emulated compiler produces byte-identical output and the alpha tests
still pass, including with a non-zero guest_base forced via -B.
RFC because:
- Only wired up for x86_64, and only for qemu_ld and qemu_st; the i128
qemu_ld2 and qemu_st2 pairs are left alone.
- Requiring that no slow path exists is stricter than necessary. The fast
path test can stay on the base register as long as the displacement is
itself a multiple of the required alignment, which it is for anything a
frontend emits for a struct or stack access. Recording the displacement
in TCGLabelQemuLdst and emitting one lea on the slow path would then
cover alignment-checked accesses too, at no fast path cost.
- Softmmu wants the displacement folded into the TLB comparison as well,
which is a bigger change than this one.
- A one op window catches everything the frontends emit today but is
trivially defeated by anything scheduled in between.
v4:
- Hoisted the compilation mode tests -- tcg_use_softmmu and the 64-bit
address type -- out of the backend hook and into fold_ldst_disp(), next
to the TCG_TARGET_HAS_ldst_disp test, so the loop is not entered at all
when the mode rules the fold out.
- Pass MemOp rather than MemOpIdx to the backend hook; nothing about the
mmu_idx is relevant to it.
- Moved the alignment test into generic code as ldst_disp_needs_align(),
so a backend does not have to repeat the atom_and_align_for_opc() call.
The exact answer depends on the host's atomicity capabilities, which the
generic pass does not know, so it answers for the most restrictive host.
That is the same answer for everything the frontends actually emit --
MO_ATOM_IFALIGN is the default -- and conservative for the handful of
MO_ATOM_WITHIN16 and MO_ATOM_SUBALIGN accesses, which lose the fold on a
host that could have taken it.
- What is left of the x86_64 hook is the guest_base test, so it now lives
beside x86_guest_base under the CONFIG_USER_ONLY that declares it.
- Refuse a displacement that does not fit in an int32_t, which is what
out_disp() takes. Not reachable with any real guest_base, but the pass
should not offer the backend something the interface cannot carry.
- The numbers above are unchanged from v3: they have not been re-measured
on the restructured patch, which is not expected to move them since the
accesses in this workload are all MO_ATOM_IFALIGN.
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
include/tcg/tcg-opc.h | 9 ++-
tcg/tcg-op-ldst.c | 3 +-
tcg/tcg.c | 132 +++++++++++++++++++++++++++++++++++-
tcg/x86_64/tcg-target.c.inc | 41 +++++++++++
tcg/x86_64/tcg-target.h | 3 +
5 files changed, 184 insertions(+), 4 deletions(-)
diff --git ./include/tcg/tcg-opc.h ./include/tcg/tcg-opc.h
index f3a81d5d7f..92fd34d3e3 100644
--- ./include/tcg/tcg-opc.h
+++ ./include/tcg/tcg-opc.h
@@ -125,8 +125,13 @@ DEF(goto_ptr, 0, 1, 0, TCG_OPF_BB_EXIT | TCG_OPF_BB_END)
DEF(plugin_cb, 0, 0, 1, TCG_OPF_NOT_PRESENT)
DEF(plugin_mem_cb, 0, 1, 1, TCG_OPF_NOT_PRESENT)
-DEF(qemu_ld, 1, 1, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT)
-DEF(qemu_st, 0, 2, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT)
+/*
+ * The second constant argument is a displacement to add to the address,
+ * zero unless a target advertises TCG_TARGET_HAS_ldst_disp and the fold in
+ * fold_ldst_disp() applied.
+ */
+DEF(qemu_ld, 1, 1, 2, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT)
+DEF(qemu_st, 0, 2, 2, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT)
DEF(qemu_ld2, 2, 1, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT)
DEF(qemu_st2, 0, 3, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT)
diff --git ./tcg/tcg-op-ldst.c ./tcg/tcg-op-ldst.c
index 22211ccb45..ffc5e651a6 100644
--- ./tcg/tcg-op-ldst.c
+++ ./tcg/tcg-op-ldst.c
@@ -92,7 +92,8 @@ static MemOp tcg_canonicalize_memop(MemOp op, bool is64, bool st)
static void gen_ldst1(TCGOpcode opc, TCGType type, TCGTemp *v,
TCGTemp *addr, MemOpIdx oi)
{
- TCGOp *op = tcg_gen_op3(opc, type, temp_arg(v), temp_arg(addr), oi);
+ /* The trailing zero is the address displacement; see fold_ldst_disp(). */
+ TCGOp *op = tcg_gen_op4(opc, type, temp_arg(v), temp_arg(addr), oi, 0);
TCGOP_FLAGS(op) = get_memop(oi) & MO_SIZE;
}
diff --git ./tcg/tcg.c ./tcg/tcg.c
index 489df0e738..466604eb97 100644
--- ./tcg/tcg.c
+++ ./tcg/tcg.c
@@ -1058,6 +1058,13 @@ typedef struct TCGOutOpQemuLdSt {
TCGOutOp base;
void (*out)(TCGContext *s, TCGType type, TCGReg dest,
TCGReg addr, MemOpIdx oi);
+ /*
+ * As out(), for an access at addr + disp. Only required of targets that
+ * define TCG_TARGET_HAS_ldst_disp; for everyone else fold_ldst_disp()
+ * never runs and the displacement is always zero.
+ */
+ void (*out_disp)(TCGContext *s, TCGType type, TCGReg dest,
+ TCGReg addr, MemOpIdx oi, int32_t disp);
} TCGOutOpQemuLdSt;
typedef struct TCGOutOpQemuLdSt2 {
@@ -3574,6 +3581,123 @@ static void move_label_uses(TCGLabel *to, TCGLabel *from)
QSIMPLEQ_CONCAT(&to->branches, &from->branches);
}
+#ifndef TCG_TARGET_HAS_ldst_disp
+#define TCG_TARGET_HAS_ldst_disp 0
+#define tcg_target_ldst_disp_ok(s, opc, disp) false
+#endif
+
+/*
+ * Return true if @opc needs an alignment test in the fast path.
+ *
+ * atom_and_align_for_opc() gives the exact answer, but only once the host's
+ * atomicity capabilities are known, and those belong to the backend. Answer
+ * instead for the most restrictive host, which is valid for all of them.
+ */
+static bool ldst_disp_needs_align(MemOp opc)
+{
+ MemOp size = opc & MO_SIZE;
+
+ if (memop_alignment_bits(opc)) {
+ return true;
+ }
+ switch (opc & MO_ATOM_MASK) {
+ case MO_ATOM_NONE:
+ case MO_ATOM_IFALIGN:
+ case MO_ATOM_IFALIGN_PAIR:
+ return false;
+ case MO_ATOM_WITHIN16:
+ /* Misalignment implies !within16, and therefore no atomicity. */
+ return size != MO_128;
+ case MO_ATOM_WITHIN16_PAIR:
+ case MO_ATOM_SUBALIGN:
+ return size != MO_8;
+ default:
+ g_assert_not_reached();
+ }
+}
+
+/*
+ * Fold "add addr, base, $disp" into the guest access that follows it, so
+ * that the displacement becomes part of the host addressing mode instead of
+ * a separate instruction. Frontends have no way to express this: there is
+ * no displacement operand on tcg_gen_qemu_ld/st, so a based access always
+ * costs an extra add, and an extra register to hold its result.
+ *
+ * Only an add in the op immediately before the access is recognized. That
+ * is what the frontends emit, and a window of one op means no analysis is
+ * needed of what might have happened in between. The add is left in place;
+ * liveness removes it if its result has no other use.
+ */
+static void __attribute__((noinline))
+fold_ldst_disp(TCGContext *s)
+{
+ TCGOp *op;
+
+ /*
+ * The fold requires that the access have no slow path, because the slow
+ * path hands the address operand to the helper and that register no
+ * longer holds the complete guest address. That means user-only, since
+ * softmmu compares the unadjusted address against the TLB. It also
+ * requires a 64-bit address type: for a 32-bit one the add wraps and a
+ * host displacement would not.
+ */
+ if (!TCG_TARGET_HAS_ldst_disp || tcg_use_softmmu ||
+ s->addr_type != TCG_TYPE_I64) {
+ return;
+ }
+
+ QTAILQ_FOREACH(op, &s->ops, link) {
+ TCGOp *prev;
+ TCGTemp *cts;
+ int64_t disp;
+ MemOp opc;
+
+ switch (op->opc) {
+ case INDEX_op_qemu_ld:
+ case INDEX_op_qemu_st:
+ break;
+ default:
+ continue;
+ }
+
+ opc = get_memop(op->args[2]);
+ if (ldst_disp_needs_align(opc)) {
+ continue;
+ }
+
+ prev = QTAILQ_PREV(op, link);
+ if (prev == NULL || prev->opc != INDEX_op_add ||
+ TCGOP_TYPE(prev) != s->addr_type) {
+ continue;
+ }
+
+ /*
+ * The add must define the address operand, and must not have
+ * clobbered the base it read: after the fold the access reads the
+ * base directly, so the base has to still hold its original value.
+ */
+ if (prev->args[0] != op->args[1] || prev->args[0] == prev->args[1]) {
+ continue;
+ }
+
+ cts = arg_temp(prev->args[2]);
+ if (cts->kind != TEMP_CONST) {
+ continue;
+ }
+ /* out_disp() takes an int32_t, so anything wider cannot be passed. */
+ disp = cts->val;
+ if (disp != (int32_t)disp) {
+ continue;
+ }
+ if (disp == 0 || !tcg_target_ldst_disp_ok(s, opc, disp)) {
+ continue;
+ }
+
+ op->args[1] = prev->args[1];
+ op->args[3] = disp;
+ }
+}
+
/* Reachable analysis : remove unreachable code. */
static void __attribute__((noinline))
reachable_code_pass(TCGContext *s)
@@ -5728,7 +5852,12 @@ static void tcg_reg_alloc_op(TCGContext *s, const TCGOp *op)
const TCGOutOpQemuLdSt *out =
container_of(all_outop[op->opc], TCGOutOpQemuLdSt, base);
- out->out(s, type, new_args[0], new_args[1], new_args[2]);
+ if (new_args[3]) {
+ out->out_disp(s, type, new_args[0], new_args[1],
+ new_args[2], new_args[3]);
+ } else {
+ out->out(s, type, new_args[0], new_args[1], new_args[2]);
+ }
}
break;
@@ -6611,6 +6740,7 @@ int tcg_gen_code(TCGContext *s, TranslationBlock *tb, uint64_t pc_start)
tcg_temp_ebb_reset_freed(s);
tcg_optimize(s);
+ fold_ldst_disp(s);
reachable_code_pass(s);
liveness_pass_0(s);
diff --git ./tcg/x86_64/tcg-target.c.inc ./tcg/x86_64/tcg-target.c.inc
index 2c8f1f3e58..9b177d3475 100644
--- ./tcg/x86_64/tcg-target.c.inc
+++ ./tcg/x86_64/tcg-target.c.inc
@@ -1892,6 +1892,18 @@ static HostAddress x86_guest_base = {
.index = -1
};
+/*
+ * Whether the displacement of a guest access can be folded into the host
+ * addressing mode rather than materialized by a separate lea. The generic
+ * pass has already established that the access has no slow path, so all
+ * that is left is guest_base, which shares the disp32 field.
+ */
+static bool tcg_target_ldst_disp_ok(TCGContext *s, MemOp opc, int32_t disp)
+{
+ int64_t ofs = (int64_t)x86_guest_base.ofs + disp;
+ return ofs == (int32_t)ofs;
+}
+
#if defined(__linux__)
# include <asm/prctl.h>
# include <sys/prctl.h>
@@ -1917,6 +1929,7 @@ static inline int setup_guest_base_seg(void)
#endif
#else
# define x86_guest_base (*(HostAddress *)({ qemu_build_not_reached(); NULL; }))
+# define tcg_target_ldst_disp_ok(s, opc, disp) false
#endif /* CONFIG_USER_ONLY */
#ifndef setup_guest_base_seg
# define setup_guest_base_seg() 0
@@ -2183,9 +2196,23 @@ static void tgen_qemu_ld(TCGContext *s, TCGType type, TCGReg data,
}
}
+static void tgen_qemu_ld_disp(TCGContext *s, TCGType type, TCGReg data,
+ TCGReg addr, MemOpIdx oi, int32_t disp)
+{
+ TCGLabelQemuLdst *ldst;
+ HostAddress h;
+
+ ldst = prepare_host_addr(s, &h, addr, oi, true);
+ /* tcg_target_ldst_disp_ok() has ruled out every slow path. */
+ tcg_debug_assert(ldst == NULL);
+ h.ofs += disp;
+ tcg_out_qemu_ld_direct(s, data, -1, h, type, get_memop(oi));
+}
+
static const TCGOutOpQemuLdSt outop_qemu_ld = {
.base.static_constraint = C_O1_I1(r, L),
.out = tgen_qemu_ld,
+ .out_disp = tgen_qemu_ld_disp,
};
static void tgen_qemu_ld2(TCGContext *s, TCGType type, TCGReg datalo,
@@ -2321,9 +2348,23 @@ static void tgen_qemu_st(TCGContext *s, TCGType type, TCGReg data,
}
}
+static void tgen_qemu_st_disp(TCGContext *s, TCGType type, TCGReg data,
+ TCGReg addr, MemOpIdx oi, int32_t disp)
+{
+ TCGLabelQemuLdst *ldst;
+ HostAddress h;
+
+ ldst = prepare_host_addr(s, &h, addr, oi, false);
+ /* tcg_target_ldst_disp_ok() has ruled out every slow path. */
+ tcg_debug_assert(ldst == NULL);
+ h.ofs += disp;
+ tcg_out_qemu_st_direct(s, data, -1, h, get_memop(oi));
+}
+
static const TCGOutOpQemuLdSt outop_qemu_st = {
.base.static_constraint = C_O0_I2(L, L),
.out = tgen_qemu_st,
+ .out_disp = tgen_qemu_st_disp,
};
static void tgen_qemu_st2(TCGContext *s, TCGType type, TCGReg datalo,
diff --git ./tcg/x86_64/tcg-target.h ./tcg/x86_64/tcg-target.h
index 7ebae56a7d..8f2315c15e 100644
--- ./tcg/x86_64/tcg-target.h
+++ ./tcg/x86_64/tcg-target.h
@@ -30,6 +30,9 @@
#define TCG_TARGET_NB_REGS 32
#define MAX_CODE_GEN_BUFFER_SIZE (2 * GiB)
+/* A guest displacement can go in the disp32 of the addressing mode. */
+#define TCG_TARGET_HAS_ldst_disp 1
+
typedef enum {
TCG_REG_EAX = 0,
TCG_REG_ECX,
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* Re: [PATCH v4 1/9] accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags
2026-08-27 5:02 ` [PATCH v4 1/9] accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags Matt Turner
@ 2026-08-27 18:51 ` Richard Henderson
0 siblings, 0 replies; 47+ messages in thread
From: Richard Henderson @ 2026-08-27 18:51 UTC (permalink / raw)
To: Matt Turner, qemu-devel; +Cc: pbonzini, philmd, alex.bennee, zhao1.liu
On 8/26/26 22:02, Matt Turner wrote:
> curr_cflags() is called once per TB dispatch, from helper_lookup_tb_ptr()
> and from the cpu_exec() loop. It recomputes the same value every time:
>
> uint32_t cflags = cpu->tcg_cflags;
> if (unlikely(cpu_single_stepping(cpu))) { ... }
> else if (qatomic_read(&one_insn_per_tb)) { ... }
> else if (qemu_loglevel_mask(CPU_LOG_TB_NOCHAIN)) { ... }
>
> That is three loads and three branches on the hottest path in the
> interpreter, for state that changes only when gdb enables single-step,
> when one-insn-per-tb is toggled, or when the log mask changes.
>
> None of the three has to be sampled at dispatch time. Fold each into
> CPUState::tcg_cflags where it changes and curr_cflags() becomes a single
> load of a field that TB lookup has to read anyway.
>
> The derived bits -- CF_COUNT_MASK, CF_NO_GOTO_TB, CF_NO_GOTO_PTR and
> CF_SINGLE_STEP -- are never set by tcg_cflags_set(), so tcg_update_cflags()
> can recompute them in place without disturbing the rest, and conversely
> tcg_cflags_set() ORs in its bits without disturbing them.
>
> There are three places to call it:
>
> - tcg_exec_realizefn(), so that a CPU created after the command line has
> been parsed starts out with the right value. This covers user-only,
> where tcg_cpu_init_cflags() is not reached. linux-user's cpu_copy()
> copies tcg_cflags wholesale, so a cloned thread inherits it.
>
> - cpu_single_step(), which changes one CPU. gdb is the only caller that
> matters; in system mode it runs with the vCPUs stopped, and in user mode
> gdb_continue_partial() can reach a thread that is still running, because
> gdb_handlesig() stops only the thread that trapped. That is exactly the
> plain cross-thread store to another CPU's CPUState that
> cpu->singlestep_flags already was, read back by that CPU through
> cpu_single_stepping() in curr_cflags(). This patch changes which field
> carries it, not who writes it or how.
>
> - hmp_one_insn_per_tb() and hmp_log(), which change every CPU while the
> vCPUs are running, so the update is queued with async_run_on_cpu() and
> each CPU writes its own cflags from its own thread. The command line
> spellings of those two settings need nothing: they are parsed before
> any CPU is realized, so tcg_exec_realizefn() picks them up.
>
> Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling
> the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, in a
> build configured with --enable-lto:
>
> before: 1,646,994,254,249 instructions
> after: 1,562,204,796,597 instructions -5.15%
>
> That workload issues 8.4 billion dispatches, so the per-call saving is
> small but the aggregate is not. The emulated compiler produces
> byte-identical output before and after.
>
> Wall clock does not move: 133.19s to 132.58s, a 0.46% difference against a
> run-to-run spread larger than that. The removed work is a few predictable
> loads and branches that the host executes largely in parallel with the
> surrounding dispatch, so this patch is worth taking for the instruction
> count and for what it enables, not for a time saving that can be measured
> on its own.
>
> v4: Update the cflags from the HMP handlers for 'log' and 'one-insn-per-tb'
> rather than from qemu_set_log_internal() and the accelerator property
> setter. Those are the paths that reach a running vCPU, and the monitor
> is the only thing that does. Suggested by Richard Henderson.
>
> v4: Queue the per-CPU update with async_run_on_cpu() rather than
> async_safe_run_on_cpu(). Halting the other vCPUs buys nothing: the
> queued work already runs on the owning CPU's own thread. Suggested by
> Alex Bennee, who also asked whether there are cross-vCPU updates of
> tcg_cflags at all. With this change the monitor path has none: the
> only remaining writer from another thread is cpu_single_step(), above,
> which is neither new nor made worse here.
>
> v4: Move the stub to accel/stubs/, which is where the other accelerator
> stubs live.
>
> Signed-off-by: Matt Turner<mattst88@gmail.com>
> ---
> accel/stubs/meson.build | 1 +
> accel/stubs/tcg-stub.c | 16 ++++++++++++++++
> accel/tcg/cpu-exec-common.c | 33 ++++++++++++++++++++++++++++++---
> accel/tcg/cpu-exec.c | 3 +++
> accel/tcg/internal-common.h | 11 +++++++++--
> cpu-target.c | 3 +++
> include/system/tcg.h | 12 ++++++++++++
> monitor/hmp-cmds.c | 5 +++++
> system/runstate-hmp-cmds.c | 4 ++++
> 9 files changed, 83 insertions(+), 5 deletions(-)
> create mode 100644 accel/stubs/tcg-stub.c
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
r~
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v4 5/9] accel/tcg: give the TB jump cache a second base pointer for generated code
2026-08-27 5:02 ` [PATCH v4 5/9] accel/tcg: give the TB jump cache a second base pointer for generated code Matt Turner
@ 2026-08-27 20:03 ` Richard Henderson
2026-09-01 2:55 ` Matt Turner
0 siblings, 1 reply; 47+ messages in thread
From: Richard Henderson @ 2026-08-27 20:03 UTC (permalink / raw)
To: Matt Turner, qemu-devel; +Cc: pbonzini, philmd, alex.bennee, zhao1.liu
On 8/26/26 22:02, Matt Turner wrote:
> +/*
> + * The inline jump cache probe reads cpu->tb_jmp_cache_probe and takes the
> + * slow path when the entry it finds has a NULL tb. Pointing the probe at a
> + * region that is all zeroes therefore forces every indirect dispatch into
> + * helper_lookup_tb_ptr(), which does the full lookup the inline probe only
> + * approximates. The real jump cache is untouched, so no contents are lost
> + * and recovery is a single store.
> + *
> + * Only ever read from, and only one entry per dispatch, so one shared
> + * zero-filled cache is enough for every CPU. Not const: that would put a
> + * megabyte of zeroes in .rodata and so in the binary, where .bss costs
> + * nothing on disk and only faults in the handful of pages a poisoned run
> + * happens to probe.
> + */
> +static CPUJumpCache tb_jmp_cache_poison;
Hmm. Maybe we should mmap it at startup then, because while we're not
supposed to be writing into this, it would be nice to enforce that.
> +/*
> + * Poison @cpu's probe, from any thread. Called when a breakpoint is
> + * inserted, which is what makes the poison take effect at the dispatch
> + * after the insert rather than whenever @cpu next reaches its main loop:
> + * a vCPU chaining indirectly need never reach it, and would run past a
> + * breakpoint another thread had just set.
> + *
> + * A plain store is enough. The value only ever costs a slow path that is
> + * correct on its own, and the generated code re-reads the base on every
> + * dispatch. Un-poisoning is tcg_cpu_sync_jmp_cache()'s job.
> + */
> +void tcg_cpu_poison_jmp_cache(CPUState *cpu)
> +{
> + if (qatomic_read(&cpu->tb_jmp_cache_probe) != NULL) {
> + qatomic_set(&cpu->tb_jmp_cache_probe, &tb_jmp_cache_poison);
> + }
> +}
Why do we need to check for NULL?
> +
> +/*
> + * Called from the main loop, which is the only context that can establish
> + * that no reason to be poisoned is left. Cheap enough to call every time
> + * round: the common case is a load, a compare and no store at all.
> + */
> +void tcg_cpu_sync_jmp_cache(CPUState *cpu)
> +{
> + CPUJumpCache *want;
> +
> + if (qatomic_read(&cpu->tb_jmp_cache_probe) == NULL) {
> + return; /* not realized, or already unrealized */
> + }
If this function is only called by the main loop, we shouldn't have to
deal with either unrealized state.
> +
> + want = tcg_cpu_may_dispatch(cpu)
> + ? cpu->tb_jmp_cache
> + : &tb_jmp_cache_poison;
> +
> + if (qatomic_read(&cpu->tb_jmp_cache_probe) != want) {
> + qatomic_set(&cpu->tb_jmp_cache_probe, want);
> +
> + if (want == cpu->tb_jmp_cache) {
> + /*
> + * Un-poisoning races a concurrent tcg_cpu_poison_jmp_cache():
> + * the reason may have appeared after tcg_cpu_may_dispatch() read
> + * it, and the poison may have landed before the store above.
> + * Order that store against the re-read below, so that the race
> + * is lost in the safe direction.
> + */
> + smp_mb();
> + if (!tcg_cpu_may_dispatch(cpu)) {
> + tcg_cpu_poison_jmp_cache(cpu);
Why do we need to check for breakpoints twice?
I don't think I understand this race.
I'm not familiar with how gdbstub interacts with user threads, but this
feels overly complicated. Up to and including needing to poison the
jump cache just for adding a breakpoint.
I suspect what we need is to add a CF_NO_GOTO_JC flag that suppresses
the inline jump cache, which is set whenever any breakpoint exists,
which falls back to the helper, which checks for breakpoints.
Conveniently, you've already shown how to adjust tcg_cflags from
gdbstub. :-)
r~
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v4 4/9] tcg: pass the destination to tcg_gen_lookup_and_goto_ptr()
2026-08-27 5:02 ` [PATCH v4 4/9] tcg: pass the destination to tcg_gen_lookup_and_goto_ptr() Matt Turner
@ 2026-08-27 23:12 ` Richard Henderson
2026-09-01 2:55 ` Matt Turner
0 siblings, 1 reply; 47+ messages in thread
From: Richard Henderson @ 2026-08-27 23:12 UTC (permalink / raw)
To: Matt Turner, qemu-devel; +Cc: pbonzini, philmd, alex.bennee, zhao1.liu
On 8/26/26 22:02, Matt Turner wrote:
> tcg_gen_lookup_and_goto_ptr() takes no arguments and emits a call to
> helper_lookup_tb_ptr(), which recovers the destination PC from env by
> calling back into the target through TCGCPUOps::get_tb_cpu_state(). At
> translation time the caller already has the destination PC in a temp, and
> knows the flags, cflags and cs_base any destination it may reach has to
> match, because they are the ones the block being generated was translated
> with.
>
> Pass both, so that a later patch can use them to look the destination up
> inline. Nothing reads them yet and the generated code does not change.
>
> The contract on @pc is the whole of the interface: it must hold exactly
> what get_tb_cpu_state() reports as the pc for the destination block. Five
> targets keep their PC in a temp whose value is that pc by construction and
> so can pass it: alpha, loongarch, mips, ppc and s390x. Everything else
> passes NULL and keeps today's behavior.
>
> For six of those the TB pc is derived and passing the PC temp would be
> wrong: avr's TB pc is the word address doubled, i386's is eip before
> segmentation, riscv masks it to 32 bits when xl is MXL_RV32, hppa derives
> it from the IAQ, hexagon adjusts it inside a hardware loop, and sparc puts
> npc in cs_base. The remaining seven -- arm, m68k, microblaze, or1k, rx, sh4
> and tricore -- look like they could pass it, but I have not convinced
> myself of the contract for them and have nothing to test them with. Each is
> a one-line change for whoever wants it.
>
> The common entry point takes a TCGTemp rather than a TCGv and reads the
> width from it, because the translators that are built for both values of
> TARGET_LONG_BITS -- arm, s390x, microblaze -- cannot include tcg-op.h.
> tcg-op.h wraps it for everyone else. This is the same split as
> tcg_gen_qemu_ld_*_chk().
>
> v4: Split out of "tcg: probe the TB jump cache inline instead of calling a
> helper", which did the API change and the inline probe in one patch.
> Requested by Richard Henderson.
>
> Signed-off-by: Matt Turner<mattst88@gmail.com>
> ---
Ok, I was a bit surprised at your claim that only 5 targets qualify, as
there are plenty that have a simple PC.
However! The subtlety of the interface, that we are asserting that the
current TranslationBlock flags are still valid, now leads me to think
that it's a mistake to adjust the current interface.
We need to introduce a new interface to which targets may be migrated.
This won't be difficult, but it's not entirely trivial.
For instance, target/arm/ has
case DISAS_UPDATE_NOCHAIN:
gen_update_pc(dc, curr_insn_len(dc));
/* fall through */
case DISAS_JUMP:
gen_goto_ptr();
break;
where DISAS_UPDATE_NOCHAIN requires the helper because of state change
and DISAS_JUMP does not.
This is subtle enough that we probably want to verify that the flags are
unchanged with --enable-debug-tcg.
Perhaps tcg_gen_goto_jc_{i32,i64,tl)?
BTW:
> --- ./include/tcg/tcg-op.h
> +++ ./include/tcg/tcg-op.h
> @@ -49,6 +49,18 @@ typedef TCGv_i64 TCGv;
> #error Unhandled TARGET_LONG_BITS value
> #endif
>
> +/*
> + * See tcg_gen_lookup_and_goto_ptr_tmp(). @pc may be NULL, for a target
> + * whose guest PC is not directly the key a destination block is found by.
> + * A translator that is built for more than one value of TARGET_LONG_BITS,
> + * and so cannot include this header, calls the _tmp() form directly.
> + */
> +static inline void
> +tcg_gen_lookup_and_goto_ptr(TCGv pc, const TranslationBlock *tb)
> +{
> + tcg_gen_lookup_and_goto_ptr_tmp(pc ? tcgv_tl_temp(pc) : NULL, tb);
> +}
> +
This needs adjustment. As we migrate binaries to single-binary, we
start building bits of code once and stop relying on TARGET_LONG_BITS.
Notice where we include "tcg-op-common.h" instead of "tcg-op.h".
The simplest solution, IMO is to define functions for _i32 and _i64,
as for most everything else, and to have a _tl alias in tcg-op.h.
r~
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v4 6/9] RFC: tcg: probe the TB jump cache inline instead of calling a helper
2026-08-27 5:02 ` [PATCH v4 6/9] RFC: tcg: probe the TB jump cache inline instead of calling a helper Matt Turner
@ 2026-08-27 23:34 ` Richard Henderson
2026-09-01 2:55 ` Matt Turner
0 siblings, 1 reply; 47+ messages in thread
From: Richard Henderson @ 2026-08-27 23:34 UTC (permalink / raw)
To: Matt Turner, qemu-devel; +Cc: pbonzini, philmd, alex.bennee, zhao1.liu
On 8/26/26 22:02, Matt Turner wrote:
> +static void gen_jmp_cache_probe(TCGv_i64 pc, const TranslationBlock *tb)
> +{
> + TCGv_ptr jc, ent, tbp, ptr;
> + TCGv_i64 h, tmp;
> + TCGLabel *slow;
> + uint64_t fpair;
> +
> + QEMU_BUILD_BUG_ON(sizeof(((CPUJumpCache *)0)->array[0]) != 16);
> + QEMU_BUILD_BUG_ON(offsetof(CPUJumpCache, array[0].pc) % 8 != 0);
> + /* One 64-bit load has to cover both, so they must be adjacent... */
> + QEMU_BUILD_BUG_ON(offsetof(TranslationBlock, cflags) !=
> + offsetof(TranslationBlock, flags) + 4);
> + /* ...and aligned, which nothing else currently relies on. */
> + QEMU_BUILD_BUG_ON(offsetof(TranslationBlock, flags) % 8 != 0);
> +
> + jc = tcg_temp_ebb_new_ptr();
> + ent = tcg_temp_ebb_new_ptr();
> + tbp = tcg_temp_ebb_new_ptr();
> + ptr = tcg_temp_ebb_new_ptr();
> + h = tcg_temp_ebb_new_i64();
> + tmp = tcg_temp_ebb_new_i64();
> + slow = gen_new_label();
> +
> + /* ent = &jc->array[tb_jmp_cache_hash_func(pc)] */
> + gen_jmp_cache_hash(h, pc);
> + tcg_gen_shli_i64(h, h, 4);
> +
> + /*
> + * Not cpu->tb_jmp_cache: the probe reads its own base so that the main
> + * loop can poison it, which is how conditions the probe cannot test for
> + * itself force every dispatch back into the helper. See
> + * tcg_cpu_sync_jmp_cache().
> + */
> + tcg_gen_ld_ptr(jc, tcg_env,
> + offsetof(CPUState, tb_jmp_cache_probe) - sizeof(CPUState));
> + tcg_gen_trunc_i64_ptr(ent, h);
> + tcg_gen_add_ptr(ent, jc, ent);
> +
> + /*
> + * The pc first: on a hash miss it is the field most likely to differ,
> + * and an entry whose tb is NULL has a zero pc that only pc 0 matches.
> + */
> + tcg_gen_ld_i64(tmp, ent, offsetof(CPUJumpCache, array[0].pc));
> + tcg_gen_brcond_i64(TCG_COND_NE, tmp, pc, slow);
> +
> + tcg_gen_ld_ptr(tbp, ent, offsetof(CPUJumpCache, array[0].tb));
> + tcg_gen_brcondi_ptr(TCG_COND_EQ, tbp, 0, slow);
> +
> + /*
> + * flags and cflags are adjacent uint32_t, so one aligned 64-bit load
> + * and compare covers both.
> + */
> +#if HOST_BIG_ENDIAN
> + fpair = ((uint64_t)tb->flags << 32) | tb->cflags;
> +#else
> + fpair = ((uint64_t)tb->cflags << 32) | tb->flags;
> +#endif
I'd use
fpair = (HOST_BIG_ENDIAN
? deposit64(tb->cflags, 32, 32, tb->flags)
: deposit64(tb->flags, 32, 32, tb->cflags));
Whenever possible, avoid ifdefs to make sure all paths compile on every
host.
> + tcg_gen_ld_i64(tmp, tbp, offsetof(TranslationBlock, flags));
> + tcg_gen_brcondi_i64(TCG_COND_NE, tmp, fpair, slow);
> +
> + /*
> + * cs_base is a second word of target-specific flags despite the name,
> + * and the pc alone does not imply it on a target that uses it.
> + */
> + tcg_gen_ld_i64(tmp, tbp, offsetof(TranslationBlock, cs_base));
> + tcg_gen_brcondi_i64(TCG_COND_NE, tmp, tb->cs_base, slow);
> +
> + tcg_gen_ld_ptr(ptr, tbp, offsetof(TranslationBlock, tc.ptr));
> + tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr));
> +
> + /*
> + * Emit a second goto_ptr rather than branching to a shared one: a temp
> + * live across the label would be spilled and reloaded on every dispatch.
> + */
> + gen_set_label(slow);
> + ptr = tcg_temp_ebb_new_ptr();
> + gen_helper_lookup_tb_ptr(ptr, tcg_env);
> + tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr));
> +}
This is pretty good. As a follow-up I suspect we'll want a tcg backend
expansion of this. For instance:
- x86_64 and s390x can use memory-operand comparisons.
- aarch64
- use shift-add insn for env_plus_off + h * 16.
- use ldp to load (tb, pc) and (cs_base, flags) in one insn.
- use ccmp to halve the number of branches.
etc.
r~
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v4 5/9] accel/tcg: give the TB jump cache a second base pointer for generated code
2026-08-27 20:03 ` Richard Henderson
@ 2026-09-01 2:55 ` Matt Turner
0 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-09-01 2:55 UTC (permalink / raw)
To: Richard Henderson; +Cc: qemu-devel, pbonzini, philmd, alex.bennee, zhao1.liu
On Thu, Aug 27, 2026 at 4:03 PM Richard Henderson
<richard.henderson@linaro.org> wrote:
>
> On 8/26/26 22:02, Matt Turner wrote:
> > +/*
> > + * The inline jump cache probe reads cpu->tb_jmp_cache_probe and takes the
> > + * slow path when the entry it finds has a NULL tb. Pointing the probe at a
> > + * region that is all zeroes therefore forces every indirect dispatch into
> > + * helper_lookup_tb_ptr(), which does the full lookup the inline probe only
> > + * approximates. The real jump cache is untouched, so no contents are lost
> > + * and recovery is a single store.
> > + *
> > + * Only ever read from, and only one entry per dispatch, so one shared
> > + * zero-filled cache is enough for every CPU. Not const: that would put a
> > + * megabyte of zeroes in .rodata and so in the binary, where .bss costs
> > + * nothing on disk and only faults in the handful of pages a poisoned run
> > + * happens to probe.
> > + */
> > +static CPUJumpCache tb_jmp_cache_poison;
>
> Hmm. Maybe we should mmap it at startup then, because while we're not
> supposed to be writing into this, it would be nice to enforce that.
Done, also in 8/9. It's a page-aligned qemu_memalign() + mprotect at
startup instead of a .bss object, which also removes the "1MB never
written" note from that patch's RFC list. I added qemu_mprotect_ro()
next to the existing _rw/_rwx/_none forms for it. A stray store into
a page every vCPU dispatches through is worth trapping rather than
debugging.
> > +/*
> > + * Poison @cpu's probe, from any thread. Called when a breakpoint is
> > + * inserted, which is what makes the poison take effect at the dispatch
> > + * after the insert rather than whenever @cpu next reaches its main loop:
> > + * a vCPU chaining indirectly need never reach it, and would run past a
> > + * breakpoint another thread had just set.
> > + *
> > + * A plain store is enough. The value only ever costs a slow path that is
> > + * correct on its own, and the generated code re-reads the base on every
> > + * dispatch. Un-poisoning is tcg_cpu_sync_jmp_cache()'s job.
> > + */
> > +void tcg_cpu_poison_jmp_cache(CPUState *cpu)
> > +{
> > + if (qatomic_read(&cpu->tb_jmp_cache_probe) != NULL) {
> > + qatomic_set(&cpu->tb_jmp_cache_probe, &tb_jmp_cache_poison);
> > + }
> > +}
>
> Why do we need to check for NULL?
Fixed, see below.
> > +
> > +/*
> > + * Called from the main loop, which is the only context that can establish
> > + * that no reason to be poisoned is left. Cheap enough to call every time
> > + * round: the common case is a load, a compare and no store at all.
> > + */
> > +void tcg_cpu_sync_jmp_cache(CPUState *cpu)
> > +{
> > + CPUJumpCache *want;
> > +
> > + if (qatomic_read(&cpu->tb_jmp_cache_probe) == NULL) {
> > + return; /* not realized, or already unrealized */
> > + }
>
> If this function is only called by the main loop, we shouldn't have to
> deal with either unrealized state.
>
> > +
> > + want = tcg_cpu_may_dispatch(cpu)
> > + ? cpu->tb_jmp_cache
> > + : &tb_jmp_cache_poison;
> > +
> > + if (qatomic_read(&cpu->tb_jmp_cache_probe) != want) {
> > + qatomic_set(&cpu->tb_jmp_cache_probe, want);
> > +
> > + if (want == cpu->tb_jmp_cache) {
> > + /*
> > + * Un-poisoning races a concurrent tcg_cpu_poison_jmp_cache():
> > + * the reason may have appeared after tcg_cpu_may_dispatch() read
> > + * it, and the poison may have landed before the store above.
> > + * Order that store against the re-read below, so that the race
> > + * is lost in the safe direction.
> > + */
> > + smp_mb();
> > + if (!tcg_cpu_may_dispatch(cpu)) {
> > + tcg_cpu_poison_jmp_cache(cpu);
>
> Why do we need to check for breakpoints twice?
> I don't think I understand this race.
Fixed, see below.
> I'm not familiar with how gdbstub interacts with user threads, but this
> feels overly complicated. Up to and including needing to poison the
> jump cache just for adding a breakpoint.
>
> I suspect what we need is to add a CF_NO_GOTO_JC flag that suppresses
> the inline jump cache, which is set whenever any breakpoint exists,
> which falls back to the helper, which checks for breakpoints.
You're right, and this is much better. v5 replaces this patch with
"accel/tcg: add CF_NO_GOTO_JC, set while a breakpoint is present".
What makes the cflag sufficient rather than merely convenient: the
inline probe already compares the destination TB's cflags against the
constant cflags of the block doing the dispatching, and only takes the
destination when they're equal. So a block translated while a
breakpoint exists neither dispatches inline itself nor can be reached
by a block that does. tcg_update_cflags() sets the flag while
cpu->breakpoints is non-empty, and
cpu_breakpoint_insert()/cpu_breakpoint_remove_by_ref() call it -- the
only two places the list changes, both of which already run on the
CPU's own thread or with it stopped, or reach another CPU exactly the
way cpu_single_step() does.
That deletes the entire mechanism you were objecting to. The
cross-thread poison from breakpoint insertion, the un-poison, and the
double breakpoint check are all gone, so the questions above mostly
answer themselves.
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v4 6/9] RFC: tcg: probe the TB jump cache inline instead of calling a helper
2026-08-27 23:34 ` Richard Henderson
@ 2026-09-01 2:55 ` Matt Turner
0 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-09-01 2:55 UTC (permalink / raw)
To: Richard Henderson; +Cc: qemu-devel, pbonzini, philmd, alex.bennee, zhao1.liu
On Thu, Aug 27, 2026 at 7:34 PM Richard Henderson
<richard.henderson@linaro.org> wrote:
>
> On 8/26/26 22:02, Matt Turner wrote:
> > +static void gen_jmp_cache_probe(TCGv_i64 pc, const TranslationBlock *tb)
> > +{
> > + TCGv_ptr jc, ent, tbp, ptr;
> > + TCGv_i64 h, tmp;
> > + TCGLabel *slow;
> > + uint64_t fpair;
> > +
> > + QEMU_BUILD_BUG_ON(sizeof(((CPUJumpCache *)0)->array[0]) != 16);
> > + QEMU_BUILD_BUG_ON(offsetof(CPUJumpCache, array[0].pc) % 8 != 0);
> > + /* One 64-bit load has to cover both, so they must be adjacent... */
> > + QEMU_BUILD_BUG_ON(offsetof(TranslationBlock, cflags) !=
> > + offsetof(TranslationBlock, flags) + 4);
> > + /* ...and aligned, which nothing else currently relies on. */
> > + QEMU_BUILD_BUG_ON(offsetof(TranslationBlock, flags) % 8 != 0);
> > +
> > + jc = tcg_temp_ebb_new_ptr();
> > + ent = tcg_temp_ebb_new_ptr();
> > + tbp = tcg_temp_ebb_new_ptr();
> > + ptr = tcg_temp_ebb_new_ptr();
> > + h = tcg_temp_ebb_new_i64();
> > + tmp = tcg_temp_ebb_new_i64();
> > + slow = gen_new_label();
> > +
> > + /* ent = &jc->array[tb_jmp_cache_hash_func(pc)] */
> > + gen_jmp_cache_hash(h, pc);
> > + tcg_gen_shli_i64(h, h, 4);
> > +
> > + /*
> > + * Not cpu->tb_jmp_cache: the probe reads its own base so that the main
> > + * loop can poison it, which is how conditions the probe cannot test for
> > + * itself force every dispatch back into the helper. See
> > + * tcg_cpu_sync_jmp_cache().
> > + */
> > + tcg_gen_ld_ptr(jc, tcg_env,
> > + offsetof(CPUState, tb_jmp_cache_probe) - sizeof(CPUState));
> > + tcg_gen_trunc_i64_ptr(ent, h);
> > + tcg_gen_add_ptr(ent, jc, ent);
> > +
> > + /*
> > + * The pc first: on a hash miss it is the field most likely to differ,
> > + * and an entry whose tb is NULL has a zero pc that only pc 0 matches.
> > + */
> > + tcg_gen_ld_i64(tmp, ent, offsetof(CPUJumpCache, array[0].pc));
> > + tcg_gen_brcond_i64(TCG_COND_NE, tmp, pc, slow);
> > +
> > + tcg_gen_ld_ptr(tbp, ent, offsetof(CPUJumpCache, array[0].tb));
> > + tcg_gen_brcondi_ptr(TCG_COND_EQ, tbp, 0, slow);
> > +
> > + /*
> > + * flags and cflags are adjacent uint32_t, so one aligned 64-bit load
> > + * and compare covers both.
> > + */
> > +#if HOST_BIG_ENDIAN
> > + fpair = ((uint64_t)tb->flags << 32) | tb->cflags;
> > +#else
> > + fpair = ((uint64_t)tb->cflags << 32) | tb->flags;
> > +#endif
>
> I'd use
>
> fpair = (HOST_BIG_ENDIAN
> ? deposit64(tb->cflags, 32, 32, tb->flags)
> : deposit64(tb->flags, 32, 32, tb->cflags));
>
> Whenever possible, avoid ifdefs to make sure all paths compile on every
> host.
Good call. Fixed.
> > + tcg_gen_ld_i64(tmp, tbp, offsetof(TranslationBlock, flags));
> > + tcg_gen_brcondi_i64(TCG_COND_NE, tmp, fpair, slow);
> > +
> > + /*
> > + * cs_base is a second word of target-specific flags despite the name,
> > + * and the pc alone does not imply it on a target that uses it.
> > + */
> > + tcg_gen_ld_i64(tmp, tbp, offsetof(TranslationBlock, cs_base));
> > + tcg_gen_brcondi_i64(TCG_COND_NE, tmp, tb->cs_base, slow);
> > +
> > + tcg_gen_ld_ptr(ptr, tbp, offsetof(TranslationBlock, tc.ptr));
> > + tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr));
> > +
> > + /*
> > + * Emit a second goto_ptr rather than branching to a shared one: a temp
> > + * live across the label would be spilled and reloaded on every dispatch.
> > + */
> > + gen_set_label(slow);
> > + ptr = tcg_temp_ebb_new_ptr();
> > + gen_helper_lookup_tb_ptr(ptr, tcg_env);
> > + tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr));
> > +}
> This is pretty good. As a follow-up I suspect we'll want a tcg backend
> expansion of this. For instance:
>
> - x86_64 and s390x can use memory-operand comparisons.
>
> - aarch64
> - use shift-add insn for env_plus_off + h * 16.
> - use ldp to load (tb, pc) and (cs_base, flags) in one insn.
> - use ccmp to halve the number of branches.
>
> etc.
Yeah, that sounds good. I'd like to look into that.
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH v4 4/9] tcg: pass the destination to tcg_gen_lookup_and_goto_ptr()
2026-08-27 23:12 ` Richard Henderson
@ 2026-09-01 2:55 ` Matt Turner
0 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-09-01 2:55 UTC (permalink / raw)
To: Richard Henderson; +Cc: qemu-devel, pbonzini, philmd, alex.bennee, zhao1.liu
On Thu, Aug 27, 2026 at 7:13 PM Richard Henderson
<richard.henderson@linaro.org> wrote:
>
> On 8/26/26 22:02, Matt Turner wrote:
> > tcg_gen_lookup_and_goto_ptr() takes no arguments and emits a call to
> > helper_lookup_tb_ptr(), which recovers the destination PC from env by
> > calling back into the target through TCGCPUOps::get_tb_cpu_state(). At
> > translation time the caller already has the destination PC in a temp, and
> > knows the flags, cflags and cs_base any destination it may reach has to
> > match, because they are the ones the block being generated was translated
> > with.
> >
> > Pass both, so that a later patch can use them to look the destination up
> > inline. Nothing reads them yet and the generated code does not change.
> >
> > The contract on @pc is the whole of the interface: it must hold exactly
> > what get_tb_cpu_state() reports as the pc for the destination block. Five
> > targets keep their PC in a temp whose value is that pc by construction and
> > so can pass it: alpha, loongarch, mips, ppc and s390x. Everything else
> > passes NULL and keeps today's behavior.
> >
> > For six of those the TB pc is derived and passing the PC temp would be
> > wrong: avr's TB pc is the word address doubled, i386's is eip before
> > segmentation, riscv masks it to 32 bits when xl is MXL_RV32, hppa derives
> > it from the IAQ, hexagon adjusts it inside a hardware loop, and sparc puts
> > npc in cs_base. The remaining seven -- arm, m68k, microblaze, or1k, rx, sh4
> > and tricore -- look like they could pass it, but I have not convinced
> > myself of the contract for them and have nothing to test them with. Each is
> > a one-line change for whoever wants it.
> >
> > The common entry point takes a TCGTemp rather than a TCGv and reads the
> > width from it, because the translators that are built for both values of
> > TARGET_LONG_BITS -- arm, s390x, microblaze -- cannot include tcg-op.h.
> > tcg-op.h wraps it for everyone else. This is the same split as
> > tcg_gen_qemu_ld_*_chk().
> >
> > v4: Split out of "tcg: probe the TB jump cache inline instead of calling a
> > helper", which did the API change and the inline probe in one patch.
> > Requested by Richard Henderson.
> >
> > Signed-off-by: Matt Turner<mattst88@gmail.com>
> > ---
>
> Ok, I was a bit surprised at your claim that only 5 targets qualify, as
> there are plenty that have a simple PC.
>
> However! The subtlety of the interface, that we are asserting that the
> current TranslationBlock flags are still valid, now leads me to think
> that it's a mistake to adjust the current interface.
>
> We need to introduce a new interface to which targets may be migrated.
> This won't be difficult, but it's not entirely trivial.
>
> For instance, target/arm/ has
>
> case DISAS_UPDATE_NOCHAIN:
> gen_update_pc(dc, curr_insn_len(dc));
> /* fall through */
> case DISAS_JUMP:
> gen_goto_ptr();
> break;
>
> where DISAS_UPDATE_NOCHAIN requires the helper because of state change
> and DISAS_JUMP does not.
>
> This is subtle enough that we probably want to verify that the flags are
> unchanged with --enable-debug-tcg.
Agreed, and done. tcg_gen_lookup_and_goto_ptr(void) is unchanged in
v5. Nothing that isn't migrated sees any difference.
> Perhaps tcg_gen_goto_jc_{i32,i64,tl)?
Sounds good. Done.
Replying to bits out of order:
> The simplest solution, IMO is to define functions for _i32 and _i64, as
> for most everything else, and to have a _tl alias in tcg-op.h.
Also done -- the _tmp()/TCGTemp form is gone. tcg_gen_goto_jc_i32()
and tcg_gen_goto_jc_i64() are declared in tcg-op-common.h, and
tcg-op.h has the _tl alias in each TARGET_LONG_BITS arm alongside the
qemu_ld/st ones. The _i32 entry point zero-extends into a temp and
shares the body, so there is one implementation.
s390x is the one migrated target that cannot include tcg-op.h, so it
calls tcg_gen_goto_jc_i64(psw_addr) directly; that reads fine.
> the subtlety is asserting that the current TB flags are still valid ...
> target/arm: DISAS_UPDATE_NOCHAIN requires the helper because of the
> state change, DISAS_JUMP does not.
Right, and that distinction is exactly what the new interface makes
the caller state. The contract is now written dow the goto_jc runs,
the CPU state must already be the destination's -- @pc must be what
get_tb_cpu_state() would report, it would report must be the ones this
block was translated with. A translator that hasn't finished updating
state lookup key (avr's word address, i386's eip before segmentation),
keeps using tcg_gen_lookup_and_goto_ptr().
> we probably want to verify that the flags ar
> --enable-debug-tcg
Done. Under CONFIG_DEBUG_TCG the goto_jc emits a call to a new
helper_goto_jc_check(env, pc, flags, cs_base), get_tb_cpu_state() and
asserts all three against what the block was translated with. It
costs nothing in a normal contract being stated rather than assumed
while I migrated the targets.
Five targets are migrated in v5: alpha, loongarch, mips, ppc and s390x
-- the ones whose lookup_and_goto_ptr sites are u already the
destination's, and the pc register is the key". arm, i386 and the
rest are untouched and can be migrated point of it being a separate
interface.
All five build and run their indirect-dispatch paths (computed goto,
function pointers, returns, and a longjmp out completion under
--enable-debug-tcg, so the assert has actually been exercised on each.
^ permalink raw reply [flat|nested] 47+ messages in thread
* [PATCH v5 0/9] accel/tcg: cut per-block dispatch overhead
2026-08-27 5:02 ` [PATCH v4 0/9] accel/tcg: cut per-block dispatch overhead Matt Turner
@ 2026-09-01 3:47 ` Matt Turner
2026-09-01 3:48 ` [PATCH v5 1/9] accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags Matt Turner
` (8 more replies)
0 siblings, 9 replies; 47+ messages in thread
From: Matt Turner @ 2026-09-01 3:47 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
For guests running large amounts of code, most of what TCG executes is not
translated guest work but the fixed overhead around it. Blocks are short and
there are a great many of them, so the constant cost at each end of a block
(the interrupt poll and the can_do_io stores on entry, the dispatch on exit)
ends up dominating everything else.
The workload throughout is qemu-alpha running an emulated alpha gcc 16.2.0
compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64
host, in a --static --enable-lto --target-list=alpha-linux-user build. It
executes 34.2 billion TBs at 6.04 guest instructions each, and 24.6% of its
TB exits cannot use goto_tb. That is a representative shape for any guest
whose text is much larger than a page: indirect calls and returns
everywhere, plus direct branches that merely crossed a page boundary.
The first three patches are ordinary cleanups that stand on their own; patch
1 picked up a review tag in v4 and patch 3 in v2. Patches 4 and 5 are
preparation and move nothing on their own. The remaining four are marked RFC
individually and are where the interesting questions are.
1 accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags
curr_cflags() recomputed three unlikely tests on every one of the run's
8.4 billion dispatches, from state that changes only when gdb enables
single-step, when one-insn-per-tb is toggled, or when the log mask
moves. Fold each into tcg_cflags where it changes. -5.15%
2 accel/tcg: enlarge the TB jump cache to 64K entries
4096 entries is too small for a guest running a large program;
tb_htable_lookup() is 6.10% of samples. 16 bits is the knee of the
sizing curve, at 1 MiB per CPUState. -5.92%, -8.71% wall
3 accel/tcg: skip the can_do_io stores in user-only builds
Two stores per TB that nothing in a user-only build reads: 68 billion
of them over the run. -4.57%, -4.53% wall
4 tcg: add tcg_gen_goto_jc_{i32,i64,tl}()
Preparation. A second dispatch interface alongside
tcg_gen_lookup_and_goto_ptr(), which is unchanged. The caller passes
the destination PC and thereby states that the CPU state is already
the destination's, which is what an inline lookup needs and what the
existing interface cannot promise. --enable-debug-tcg checks that
claim at runtime. Five targets are migrated; every other target and
every unmigrated call site is untouched.
5 accel/tcg: add CF_NO_GOTO_JC, set while a breakpoint is present
Preparation. The one thing an inline jump cache probe cannot check is
breakpoints, and it does not have to: the probe compares cflags, so a
cflag set while cpu->breakpoints is non-empty keeps such blocks both
from dispatching inline and from being reached by a block that does.
Nothing reads it yet.
6 RFC: tcg: probe the TB jump cache inline instead of calling a helper
95.8% of those 8.4 billion helper_lookup_tb_ptr() calls hit the jump
cache. Emit the probe inline (hash, four guarded loads, goto_ptr) and
call the helper only on a miss. -34.67%, -25.94% wall
7 RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds
translator_use_goto_tb() refuses to chain across a page. In user-only
builds the invalidation path already covers what that was protecting
against: every mmap/mprotect/munmap reaches page_set_flags(), which
invalidates and unlinks. Lift it there for runs that can never acquire
a breakpoint, keep it for system mode. -2.75%, -4.84% wall
8 RFC: accel/tcg: poison the jump cache instead of polling for indirect
exits
A block only needs the icount_decr poll if it can leave by goto_tb;
every other exit already passes through a dispatch. Give generated
code its own jump cache base pointer and point it at a read-only page
of zeroes when an exit is requested, so every dispatch misses into the
helper, which returns the epilogue. Emit the poll only in blocks that
emitted a goto_tb. -2.52%, -1.96% wall
9 RFC: tcg: fold a guest displacement into the host addressing mode
tcg_gen_qemu_ld/st cannot express a based access, so a target with a
displacement in its encodings materializes the address with an lea
that the host addressing mode would have done for free. Fold a
preceding constant add into a new argument on the op, opt-in per
backend, wired up for x86_64 user-only. -5.70%, -3.20% wall
Each percentage is against the patch before it. Every stage was measured in
one session on the same host, so end to end, from an unmodified LTO build of
the same base to the full series:
instructions retired: 1,646,994,254,249 -> 819,262,147,022 -50.26%
wall clock: 133.19s -> 77.30s -41.96%
Those are still the v3 measurements, unchanged and not re-run; the machine
they were taken on is busy. Nothing in v4 or v5 is expected to move them --
both revisions are reorganization, and the emulated compiler's output is
still byte-identical -- but they are not a measurement of this posting and
should not be read as one.
The two figures do not track each other, and that is the interesting part:
what the series removes is cheap, well-predicted, highly pipelined work, so
it retires far more instructions than it saves time. Patch 6 also cuts L1
icache load misses by 39.0%, because a dispatch no longer jumps into qemu's
.text and evicts translated code; qemu's own .text falls from 38.8% to 5.3%
of profile samples.
Every step builds and runs on its own, so the series bisects, and the
emulated compiler produces byte-identical assembly output at every step,
which is the correctness check these patches most need. Three tests under
tests/tcg/multiarch cover the hazards the series creates: test-xpage-chain.c
and gdbstub/xpage-bp.py (patch 7) and test-indirect-irq.c (patch 8). Each
fails or hangs if the mechanism it covers is removed, which is what makes
them tests of the new behavior rather than of the old.
Changes since v4
================
The structural change is that v4's patches 4 and 5 are gone, replaced by
different patches with the same job.
v4's patch 4 added an argument to tcg_gen_lookup_and_goto_ptr() and made
every caller pass NULL. Richard's example is target/arm, where
DISAS_UPDATE_NOCHAIN needs the helper because the state has changed while
DISAS_JUMP does not; that subtlety, he pointed out, means the existing
interface should not be adjusted at all, and that targets should migrate to
a new one instead. So v5 leaves tcg_gen_lookup_and_goto_ptr(void) as it was
and adds tcg_gen_goto_jc_{i32,i64,tl}() beside it. Only the five migrated
targets are touched, rather than all twenty; the diffstat is a good summary
of the difference.
v4's patch 5 gave generated code a second jump cache base pointer and
poisoned it when a breakpoint was inserted, which needed a cross-thread
poison, an un-poison, and a double check of the breakpoint list. Richard
suggested a cflag instead, which is both simpler and sufficient: the probe
already compares cflags, so CF_NO_GOTO_JC alone keeps both the block itself
and anything chaining to it off the inline path. The whole poison mechanism
leaves this patch. The base pointer moves down to patch 8, where a pending
exit is the only reason left to want one, that being a per-execution
condition no cflag can express.
1 Reviewed-by: Richard Henderson.
2 Unchanged.
3 Unchanged.
4 Replaces "tcg: pass the destination to tcg_gen_lookup_and_goto_ptr()".
New interface rather than a changed one; _i32 and _i64 entry points
with a _tl alias in tcg-op.h rather than one entry point taking a
TCGTemp, since single-binary targets build once and stop relying on
TARGET_LONG_BITS; and a new helper_goto_jc_check() that asserts pc,
flags and cs_base against get_tb_cpu_state() under CONFIG_DEBUG_TCG.
(All Richard.) The contract is now written on the declaration rather
than left to be inferred.
5 Replaces "accel/tcg: give the TB jump cache a second base pointer for
generated code" (Richard, as above). That leaves a residual window,
in which blocks translated before the insert keep chaining on their
old cflags until the vCPU reaches its main loop. It is the same window
goto_tb chaining already has, and in system mode gdb inserts
breakpoints with the vCPUs stopped, so there is none. The commit
message says so rather than leaving it implicit.
6 Build the folded flags/cflags constant with deposit64() rather than
under #if HOST_BIG_ENDIAN, so both arms compile on every host
(Richard). Read cpu->tb_jmp_cache directly and honor CF_NO_GOTO_JC,
following patch 5. Commit message notes the backend expansion this
wants as a follow-up, which is Richard's list: x86_64 and s390x can
compare against a memory operand, and aarch64 has shift-add for the
entry address, ldp to load (tb, pc) and (cs_base, flags), and ccmp to
halve the branches. Not attempted here: the probe as posted is correct
on every backend, and the expansions are strictly additive and deserve
their own numbers, particularly the aarch64 one, which changes the
shape enough that it should be measured on aarch64 hardware.
7 Unchanged.
8 Gains CPUState::tb_jmp_cache_probe from v4's patch 5. With breakpoints
handled by a cflag, a pending exit is the only reason left to poison,
so there is one condition rather than two, no cross-thread poison from
cpu_breakpoint_insert(), and no unrealized or NULL state for either
helper to consider: the probe is initialized alongside tb_jmp_cache
and unrealize leaves it pointing at the poison (Richard). The poison
is now a page-aligned allocation mapped read-only at startup rather
than a writable .bss object (Richard); qemu_mprotect_ro() is added for
it beside the existing _rw, _rwx and _none forms. That also settles
v4's own note about a 1 MiB object that is never written.
9 Unchanged.
Testing
=======
alpha, loongarch64, mips, mipsel, mips64, ppc, ppc64 and s390x all build and
run an indirect-dispatch exerciser (computed-goto back edges, function
pointer calls, returns and a longjmp out of a SIGALRM handler) to
completion under --enable-debug-tcg, so helper_goto_jc_check()'s assertions
have actually been exercised on every migrated target rather than only on
alpha. The gdbstub path (insert, hit, backtrace, delete, continue) and
test-indirect-irq still pass, and the emulated compiler's output is still
byte-identical.
What I would most like reviewed
===============================
- Patch 7 reverses a deliberate decision made in d3a2a1d803 on the
strength of an argument about the user-only invalidation paths, plus a
gate on whether gdb can ever attach.
- Patch 8's un-poison in the main loop races a concurrent poison from
another thread. I believe the existing barrier around
icount_decr.u16.high covers it, but my testing was single-threaded user
mode.
- Patch 5's argument that a cflag is sufficient rests on the probe
comparing cflags and on the residual window being one goto_tb chaining
already accepts. Both seem clearly true to me, which is why they are
worth a second reader.
- Patch 6 treats cpu flags, cflags and cs_base as translation-time
constants in its guards, reads a jump cache entry without qatomic_read(),
and leaves one_insn_per_tb and -d nochain toggles visible only at the
next non-inline exit.
- Patch 9 only examines the immediately preceding op, refuses any access
with a slow path (so user-only, and no alignment check), and leaves the
i128 pairs alone. Richard asked whether the alignment test could stay on
the base register when the displacement is itself aligned; it can, and
the reason the fold is still refused there is the slow path handing
addr_reg to the helper. Recording the displacement in TCGLabelQemuLdst
and emitting one lea on the slow path would cover alignment-checked
accesses too, at no fast path cost. Not attempted here.
- Patch 2's 1 MiB per CPUState is easy to justify for a single-threaded
linux-user process and less obvious for system emulation with many
vCPUs, or for a heavily threaded guest. It may want to be sized per
target or made tunable rather than raised unconditionally.
Patches 6 and 9 are wired up for alpha and x86_64 respectively; everything
else is target-independent, and no other backend changes behavior or needs
touching.
v4: https://lore.kernel.org/qemu-devel/20260827050241.3713332-1-mattst88@gmail.com/
v3: https://lore.kernel.org/qemu-devel/20260822190818.1829249-1-mattst88@gmail.com/
v2: https://lore.kernel.org/qemu-devel/20260817190038.580257-1-mattst88@gmail.com/
Matt Turner (9):
accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags
accel/tcg: enlarge the TB jump cache to 64K entries
accel/tcg: skip the can_do_io stores in user-only builds
tcg: add tcg_gen_goto_jc_{i32,i64,tl}()
accel/tcg: add CF_NO_GOTO_JC, set while a breakpoint is present
RFC: tcg: probe the TB jump cache inline instead of calling a helper
RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds
RFC: accel/tcg: poison the jump cache instead of polling for indirect
exits
RFC: tcg: fold a guest displacement into the host addressing mode
accel/stubs/meson.build | 1 +
accel/stubs/tcg-stub.c | 16 +
accel/tcg/cpu-exec-common.c | 44 ++-
accel/tcg/cpu-exec.c | 121 +++++++
accel/tcg/internal-common.h | 20 +-
accel/tcg/tb-jmp-cache.h | 2 +-
accel/tcg/tcg-accel-ops.c | 2 +
accel/tcg/tcg-runtime.h | 4 +
accel/tcg/translator.c | 94 ++++-
cpu-common.c | 7 +
cpu-target.c | 3 +
gdbstub/user.c | 14 +
include/exec/translation-block.h | 1 +
include/gdbstub/user.h | 11 +
include/hw/core/cpu.h | 9 +
include/qemu/mprotect.h | 1 +
include/system/tcg.h | 12 +
include/tcg/tcg-op-common.h | 21 ++
include/tcg/tcg-op.h | 2 +
include/tcg/tcg-opc.h | 9 +-
include/tcg/tcg.h | 2 +
monitor/hmp-cmds.c | 5 +
system/runstate-hmp-cmds.c | 4 +
target/alpha/translate.c | 4 +-
.../tcg/insn_trans/trans_branch.c.inc | 2 +-
target/loongarch/tcg/translate.c | 4 +-
target/mips/tcg/nanomips_translate.c.inc | 2 +-
target/mips/tcg/translate.c | 6 +-
target/ppc/translate.c | 4 +-
target/s390x/tcg/translate.c | 4 +-
tcg/tcg-op-ldst.c | 3 +-
tcg/tcg-op.c | 182 +++++++++-
tcg/tcg.c | 132 ++++++-
tcg/x86_64/tcg-target.c.inc | 41 +++
tcg/x86_64/tcg-target.h | 3 +
tests/tcg/multiarch/Makefile.target | 12 +-
tests/tcg/multiarch/gdbstub/xpage-bp.py | 37 ++
tests/tcg/multiarch/test-indirect-irq.c | 62 ++++
tests/tcg/multiarch/test-xpage-chain.c | 336 ++++++++++++++++++
util/osdep.c | 9 +
40 files changed, 1216 insertions(+), 32 deletions(-)
create mode 100644 accel/stubs/tcg-stub.c
create mode 100644 tests/tcg/multiarch/gdbstub/xpage-bp.py
create mode 100644 tests/tcg/multiarch/test-indirect-irq.c
create mode 100644 tests/tcg/multiarch/test-xpage-chain.c
base-commit: eea8fe61b8be8f3016e522e6af24924a0266ca95
--
2.54.0
^ permalink raw reply [flat|nested] 47+ messages in thread
* [PATCH v5 1/9] accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags
2026-09-01 3:47 ` [PATCH v5 " Matt Turner
@ 2026-09-01 3:48 ` Matt Turner
2026-09-01 3:48 ` [PATCH v5 2/9] accel/tcg: enlarge the TB jump cache to 64K entries Matt Turner
` (7 subsequent siblings)
8 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-09-01 3:48 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
curr_cflags() is called once per TB dispatch, from helper_lookup_tb_ptr()
and from the cpu_exec() loop. It recomputes the same value every time:
uint32_t cflags = cpu->tcg_cflags;
if (unlikely(cpu_single_stepping(cpu))) { ... }
else if (qatomic_read(&one_insn_per_tb)) { ... }
else if (qemu_loglevel_mask(CPU_LOG_TB_NOCHAIN)) { ... }
That is three loads and three branches on the hottest path in the
interpreter, for state that changes only when gdb enables single-step,
when one-insn-per-tb is toggled, or when the log mask changes.
None of the three has to be sampled at dispatch time. Fold each into
CPUState::tcg_cflags where it changes and curr_cflags() becomes a single
load of a field that TB lookup has to read anyway.
The derived bits -- CF_COUNT_MASK, CF_NO_GOTO_TB, CF_NO_GOTO_PTR and
CF_SINGLE_STEP -- are never set by tcg_cflags_set(), so tcg_update_cflags()
can recompute them in place without disturbing the rest, and conversely
tcg_cflags_set() ORs in its bits without disturbing them.
There are three places to call it:
- tcg_exec_realizefn(), so that a CPU created after the command line has
been parsed starts out with the right value. This covers user-only,
where tcg_cpu_init_cflags() is not reached. linux-user's cpu_copy()
copies tcg_cflags wholesale, so a cloned thread inherits it.
- cpu_single_step(), which changes one CPU. gdb is the only caller that
matters; in system mode it runs with the vCPUs stopped, and in user mode
gdb_continue_partial() can reach a thread that is still running, because
gdb_handlesig() stops only the thread that trapped. That is exactly the
plain cross-thread store to another CPU's CPUState that
cpu->singlestep_flags already was, read back by that CPU through
cpu_single_stepping() in curr_cflags(). This patch changes which field
carries it, not who writes it or how.
- hmp_one_insn_per_tb() and hmp_log(), which change every CPU while the
vCPUs are running, so the update is queued with async_run_on_cpu() and
each CPU writes its own cflags from its own thread. The command line
spellings of those two settings need nothing: they are parsed before
any CPU is realized, so tcg_exec_realizefn() picks them up.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling
the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, in a
build configured with --enable-lto:
before: 1,646,994,254,249 instructions
after: 1,562,204,796,597 instructions -5.15%
That workload issues 8.4 billion dispatches, so the per-call saving is
small but the aggregate is not. The emulated compiler produces
byte-identical output before and after.
Wall clock does not move: 133.19s to 132.58s, a 0.46% difference against a
run-to-run spread larger than that. The removed work is a few predictable
loads and branches that the host executes largely in parallel with the
surrounding dispatch, so this patch is worth taking for the instruction
count and for what it enables, not for a time saving that can be measured
on its own.
v4: Update the cflags from the HMP handlers for 'log' and 'one-insn-per-tb'
rather than from qemu_set_log_internal() and the accelerator property
setter. Those are the paths that reach a running vCPU, and the monitor
is the only thing that does. Suggested by Richard Henderson.
v4: Queue the per-CPU update with async_run_on_cpu() rather than
async_safe_run_on_cpu(). Halting the other vCPUs buys nothing: the
queued work already runs on the owning CPU's own thread. Suggested by
Alex Bennee, who also asked whether there are cross-vCPU updates of
tcg_cflags at all. With this change the monitor path has none: the
only remaining writer from another thread is cpu_single_step(), above,
which is neither new nor made worse here.
v4: Move the stub to accel/stubs/, which is where the other accelerator
stubs live.
Signed-off-by: Matt Turner <mattst88@gmail.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
---
accel/stubs/meson.build | 1 +
accel/stubs/tcg-stub.c | 16 ++++++++++++++++
accel/tcg/cpu-exec-common.c | 33 ++++++++++++++++++++++++++++++---
accel/tcg/cpu-exec.c | 3 +++
accel/tcg/internal-common.h | 11 +++++++++--
cpu-target.c | 3 +++
include/system/tcg.h | 12 ++++++++++++
monitor/hmp-cmds.c | 5 +++++
system/runstate-hmp-cmds.c | 4 ++++
9 files changed, 83 insertions(+), 5 deletions(-)
create mode 100644 accel/stubs/tcg-stub.c
diff --git ./accel/stubs/meson.build ./accel/stubs/meson.build
index 7c6d7ad943..ccad583e64 100644
--- ./accel/stubs/meson.build
+++ ./accel/stubs/meson.build
@@ -4,6 +4,7 @@ stub_ss.add(files(
'nitro-stub.c',
'mshv-stub.c',
'nvmm-stub.c',
+ 'tcg-stub.c',
'whpx-stub.c',
'xen-stub.c',
))
diff --git ./accel/stubs/tcg-stub.c ./accel/stubs/tcg-stub.c
new file mode 100644
index 0000000000..f9e1bd22d6
--- /dev/null
+++ ./accel/stubs/tcg-stub.c
@@ -0,0 +1,16 @@
+/*
+ * Stubs for the TCG entry points in system/tcg.h, for binaries that link
+ * cpu-target.c or the HMP command handlers but not TCG.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+#include "qemu/osdep.h"
+#include "system/tcg.h"
+
+void tcg_update_cflags(CPUState *cpu)
+{
+}
+
+void tcg_update_all_cflags(void)
+{
+}
diff --git ./accel/tcg/cpu-exec-common.c ./accel/tcg/cpu-exec-common.c
index 44e84344f3..9f3517f36b 100644
--- ./accel/tcg/cpu-exec-common.c
+++ ./accel/tcg/cpu-exec-common.c
@@ -36,9 +36,16 @@ void tcg_cflags_set(CPUState *cpu, uint32_t flags)
cpu->tcg_cflags |= flags;
}
-uint32_t curr_cflags(CPUState *cpu)
+/*
+ * The bits of CPUState::tcg_cflags that tcg_cflags_set() never sets, because
+ * they are derived from gdb single-step, one-insn-per-tb and -d nochain.
+ */
+#define CF_DERIVED (CF_COUNT_MASK | CF_NO_GOTO_TB | CF_NO_GOTO_PTR | \
+ CF_SINGLE_STEP)
+
+void tcg_update_cflags(CPUState *cpu)
{
- uint32_t cflags = cpu->tcg_cflags;
+ uint32_t cflags = cpu->tcg_cflags & ~CF_DERIVED;
/*
* Record gdb single-step. We should be exiting the TB by raising
@@ -55,7 +62,27 @@ uint32_t curr_cflags(CPUState *cpu)
cflags |= CF_NO_GOTO_TB;
}
- return cflags;
+ cpu->tcg_cflags = cflags;
+}
+
+static void tcg_update_cflags_work(CPUState *cpu, run_on_cpu_data data)
+{
+ tcg_update_cflags(cpu);
+}
+
+void tcg_update_all_cflags(void)
+{
+ CPUState *cpu;
+
+ /*
+ * one-insn-per-tb and -d nochain can both be changed from the monitor
+ * while the vCPUs are running. Queue the update onto each CPU rather
+ * than writing tcg_cflags from here, so that the field is only ever
+ * written by the CPU that owns it.
+ */
+ CPU_FOREACH(cpu) {
+ async_run_on_cpu(cpu, tcg_update_cflags_work, RUN_ON_CPU_NULL);
+ }
}
/* exit the current TB, but without causing any exception to be raised */
diff --git ./accel/tcg/cpu-exec.c ./accel/tcg/cpu-exec.c
index 257211235d..148e0f583e 100644
--- ./accel/tcg/cpu-exec.c
+++ ./accel/tcg/cpu-exec.c
@@ -1068,6 +1068,9 @@ bool tcg_exec_realizefn(CPUState *cpu, Error **errp)
tcg_target_initialized = true;
}
+ /* Pick up one-insn-per-tb and -d nochain from the command line. */
+ tcg_update_cflags(cpu);
+
cpu->tb_jmp_cache = g_new0(CPUJumpCache, 1);
tlb_init(cpu);
#ifndef CONFIG_USER_ONLY
diff --git ./accel/tcg/internal-common.h ./accel/tcg/internal-common.h
index 9e7be2d78d..853d1b51ee 100644
--- ./accel/tcg/internal-common.h
+++ ./accel/tcg/internal-common.h
@@ -69,8 +69,15 @@ void tlb_destroy(CPUState *cpu);
bool tcg_exec_realizefn(CPUState *cpu, Error **errp);
void tcg_exec_unrealizefn(CPUState *cpu);
-/* current cflags for hashing/comparison */
-uint32_t curr_cflags(CPUState *cpu);
+/*
+ * Current cflags for hashing/comparison. Everything that feeds into the
+ * value is folded into CPUState::tcg_cflags when it changes, by
+ * tcg_update_cflags(), so that TB dispatch only has to load it.
+ */
+static inline uint32_t curr_cflags(CPUState *cpu)
+{
+ return cpu->tcg_cflags;
+}
void tb_check_watchpoint(CPUState *cpu, uintptr_t retaddr);
diff --git ./cpu-target.c ./cpu-target.c
index 4783845c9b..50be591acf 100644
--- ./cpu-target.c
+++ ./cpu-target.c
@@ -24,6 +24,7 @@
#include "exec/replay-core.h"
#include "exec/log.h"
#include "hw/core/cpu.h"
+#include "system/tcg.h"
#include "trace/trace-root.h"
/* enable or disable single step mode. EXCP_DEBUG is returned by the
@@ -35,6 +36,8 @@ void cpu_single_step(CPUState *cpu, unsigned flags)
cpu->singlestep_flags, flags);
cpu->singlestep_flags = flags;
+ tcg_update_cflags(cpu);
+
#if !defined(CONFIG_USER_ONLY)
const AccelOpsClass *ops = cpus_get_accel();
if (ops->update_guest_debug) {
diff --git ./include/system/tcg.h ./include/system/tcg.h
index 7622dcea30..2c2dbc753b 100644
--- ./include/system/tcg.h
+++ ./include/system/tcg.h
@@ -17,6 +17,18 @@ extern bool tcg_allowed;
#define tcg_enabled() 0
#endif
+/*
+ * Recompute the parts of CPUState::tcg_cflags that TB dispatch consumes but
+ * tcg_cflags_set() does not provide: gdb single-step, one-insn-per-tb and
+ * the CPU_LOG_TB_NOCHAIN log flag. Call whenever one of those changes.
+ *
+ * tcg_update_cflags() updates one CPU and must be called from that CPU's
+ * thread, or with it stopped. tcg_update_all_cflags() updates every CPU
+ * and is safe to call from the monitor while the vCPUs run.
+ */
+void tcg_update_cflags(CPUState *cpu);
+void tcg_update_all_cflags(void);
+
/**
* qemu_tcg_mttcg_enabled:
* Check whether we are running MultiThread TCG or not.
diff --git ./monitor/hmp-cmds.c ./monitor/hmp-cmds.c
index 4e8d996dbb..b83551ea54 100644
--- ./monitor/hmp-cmds.c
+++ ./monitor/hmp-cmds.c
@@ -39,6 +39,7 @@
#include "system/hw_accel.h"
#include "system/memory.h"
#include "system/system.h"
+#include "system/tcg.h"
#include "disas/disas.h"
/* Please update hmp-commands.hx when adding or changing commands */
@@ -335,7 +336,11 @@ void hmp_log(Monitor *mon, const QDict *qdict)
if (!qemu_set_log(mask, &err)) {
error_report_err(err);
+ return;
}
+
+ /* CPU_LOG_TB_NOCHAIN feeds into the per-CPU cflags. */
+ tcg_update_all_cflags();
}
void hmp_gdbserver(Monitor *mon, const QDict *qdict)
diff --git ./system/runstate-hmp-cmds.c ./system/runstate-hmp-cmds.c
index 02d1d42bf3..86754a37f8 100644
--- ./system/runstate-hmp-cmds.c
+++ ./system/runstate-hmp-cmds.c
@@ -22,6 +22,7 @@
#include "qapi/qapi-commands-run-state.h"
#include "qobject/qdict.h"
#include "qemu/accel.h"
+#include "system/tcg.h"
void hmp_info_status(Monitor *mon, const QDict *qdict)
{
@@ -64,6 +65,9 @@ void hmp_one_insn_per_tb(Monitor *mon, const QDict *qdict)
/* If the property exists then setting it can never fail */
object_property_set_bool(OBJECT(accel), "one-insn-per-tb",
newval, &error_abort);
+
+ /* one-insn-per-tb feeds into the per-CPU cflags. */
+ tcg_update_all_cflags();
}
void hmp_watchdog_action(Monitor *mon, const QDict *qdict)
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH v5 2/9] accel/tcg: enlarge the TB jump cache to 64K entries
2026-09-01 3:47 ` [PATCH v5 " Matt Turner
2026-09-01 3:48 ` [PATCH v5 1/9] accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags Matt Turner
@ 2026-09-01 3:48 ` Matt Turner
2026-09-01 3:48 ` [PATCH v5 3/9] accel/tcg: skip the can_do_io stores in user-only builds Matt Turner
` (6 subsequent siblings)
8 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-09-01 3:48 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
The per-CPU TB jump cache has held 4096 entries since it was introduced.
That is too small for guests running large programs: an emulated compiler
misses often enough that the fallback qht lookup shows up prominently in
a profile.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling
the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host. The
compile performs 34.2 billion TB executions, of which 8.4 billion take
the indirect dispatch path.
Sizing curve, on top of the preceding patch, instructions retired and
wall clock:
12 bits ( 64 KiB): 1,562,204,796,597 132.58s
14 bits ( 256 KiB): 1,493,318,515,396 -4.41% 124.67s -5.97%
16 bits ( 1 MiB): 1,469,772,951,575 -5.92% 121.04s -8.71%
18 bits ( 4 MiB): 1,462,309,832,762 -6.39% 119.82s -9.62%
16 bits is the knee. 18 buys another 0.47% of instructions for four times
the memory. It does show a further 1.01% of wall clock, which is outside
the 0.70% run-to-run spread at 16 bits, so the effect is probably real --
but paying four times the memory for it is a poor trade, and instructions
retired does not account for the data cache pressure of a 4 MiB table.
In a perf profile the mechanism is visible directly: tb_htable_lookup(),
which is where qht_lookup_custom() lands once it is inlined in an LTO
build, falls from 6.10% of samples to 1.66%.
The cost is memory: the cache grows from 64 KiB to 1 MiB, once per
CPUState. In linux-user that is per guest thread rather than per process,
so a threaded guest pays it as many times as it has threads, exactly as
system emulation pays it per vCPU. The allocation is g_new0(), so the
pages are faulted in as the cache is touched and a thread that runs a
small amount of code touches a small part of it, but the address space is
committed either way.
So this may still want to be tunable, or scaled from the number of CPUs,
rather than raised unconditionally. I do not have a threaded workload where
the smaller cache is the better trade, and would welcome one.
v4: Fix the claim that a linux-user process is a single vCPU. The cache is
per CPUState, and linux-user creates one per guest thread. Pointed out
by Richard Henderson.
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
accel/tcg/tb-jmp-cache.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git ./accel/tcg/tb-jmp-cache.h ./accel/tcg/tb-jmp-cache.h
index c3a505e394..268dacd7ba 100644
--- ./accel/tcg/tb-jmp-cache.h
+++ ./accel/tcg/tb-jmp-cache.h
@@ -12,7 +12,7 @@
#include "qemu/rcu.h"
#include "exec/cpu-common.h"
-#define TB_JMP_CACHE_BITS 12
+#define TB_JMP_CACHE_BITS 16
#define TB_JMP_CACHE_SIZE (1 << TB_JMP_CACHE_BITS)
/*
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH v5 3/9] accel/tcg: skip the can_do_io stores in user-only builds
2026-09-01 3:47 ` [PATCH v5 " Matt Turner
2026-09-01 3:48 ` [PATCH v5 1/9] accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags Matt Turner
2026-09-01 3:48 ` [PATCH v5 2/9] accel/tcg: enlarge the TB jump cache to 64K entries Matt Turner
@ 2026-09-01 3:48 ` Matt Turner
2026-09-01 3:48 ` [PATCH v5 4/9] tcg: add tcg_gen_goto_jc_{i32,i64,tl}() Matt Turner
` (5 subsequent siblings)
8 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-09-01 3:48 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
Every translation block stores to cpu->neg.can_do_io twice: false before
the first instruction, true before the last one. Nothing reads it in a
user-only build. There is no memory-mapped I/O in linux-user, and every
reader is in system_ss: cputlb.c, watchpoint.c, icount-common.c and
tcg-accel-ops-icount.c.
Two stores per TB is not much on its own, but TBs are short. An emulated
alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines,
-O2) executes 34.2 billion TBs at 6.04 guest instructions each, so this is
68 billion stores for nothing.
Measured on an x86-64 host, LTO build, on top of the preceding two
patches:
before: 1,469,772,951,575 instructions
after: 1,402,667,803,616 instructions -4.57%
before: 121.04s wall clock
after: 115.56s wall clock -4.53%
The emulated compiler produces byte-identical output.
v3: Use #ifndef CONFIG_USER_ONLY again rather than
if (IS_ENABLED(CONFIG_USER_ONLY)). QEMU's IS_ENABLED() is IS_EMPTY(),
which is only true for a symbol Meson defines empty; CONFIG_USER_ONLY
is defined as 1, so the test was always false and v2 emitted the two
stores after all. The measurements above are from the working form.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
accel/tcg/translator.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git ./accel/tcg/translator.c ./accel/tcg/translator.c
index 57daded60f..6c8fcd7a20 100644
--- ./accel/tcg/translator.c
+++ ./accel/tcg/translator.c
@@ -21,12 +21,14 @@
#include "disas/disas.h"
#include "tb-internal.h"
+#ifndef CONFIG_USER_ONLY
static void set_can_do_io(DisasContextBase *db, bool val)
{
QEMU_BUILD_BUG_ON(sizeof_field(CPUState, neg.can_do_io) != 1);
tcg_gen_st8_i32(tcg_constant_i32(val), tcg_env,
offsetof(CPUState, neg.can_do_io) - sizeof(CPUState));
}
+#endif
bool translator_io_start(DisasContextBase *db)
{
@@ -210,17 +212,25 @@ void translator_loop(CPUState *cpu, TranslationBlock *tb, int *max_insns,
/*
* Manage can_do_io for the translation block: set to false before
* the first insn and set to true before the last insn.
+ *
+ * Nothing reads can_do_io in user-only builds. There is no MMIO
+ * there, and every reader (cputlb.c, watchpoint.c, icount) is in
+ * system_ss, so skip the two stores per TB entirely.
*/
if (db->num_insns == 1) {
tcg_debug_assert(first_insn_start == db->insn_start);
} else {
tcg_debug_assert(first_insn_start != db->insn_start);
+#ifndef CONFIG_USER_ONLY
tcg_ctx->emit_before_op = first_insn_start;
set_can_do_io(db, false);
+#endif
}
+#ifndef CONFIG_USER_ONLY
tcg_ctx->emit_before_op = db->insn_start;
set_can_do_io(db, true);
tcg_ctx->emit_before_op = NULL;
+#endif
/* May be used by disas_log or plugin callbacks. */
tb->size = db->pc_next - db->pc_first;
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH v5 4/9] tcg: add tcg_gen_goto_jc_{i32,i64,tl}()
2026-09-01 3:47 ` [PATCH v5 " Matt Turner
` (2 preceding siblings ...)
2026-09-01 3:48 ` [PATCH v5 3/9] accel/tcg: skip the can_do_io stores in user-only builds Matt Turner
@ 2026-09-01 3:48 ` Matt Turner
2026-09-01 3:48 ` [PATCH v5 5/9] accel/tcg: add CF_NO_GOTO_JC, set while a breakpoint is present Matt Turner
` (4 subsequent siblings)
8 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-09-01 3:48 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
tcg_gen_lookup_and_goto_ptr() takes no arguments and emits a call to
helper_lookup_tb_ptr(), which recovers the destination PC from env by
calling back into the target through TCGCPUOps::get_tb_cpu_state(). At
translation time the caller often already has the destination PC in a temp,
and knows the flags, cflags and cs_base any destination it may reach has to
match, because they are the ones the block being generated was translated
with. A later patch uses that to look the destination up inline.
But it is not something every caller of tcg_gen_lookup_and_goto_ptr() can
promise, and the promise is subtle. target/arm has
case DISAS_UPDATE_NOCHAIN:
gen_update_pc(dc, curr_insn_len(dc));
/* fall through */
case DISAS_JUMP:
gen_goto_ptr();
break;
where DISAS_JUMP could make the promise and DISAS_UPDATE_NOCHAIN could not,
because it is there precisely because the state changed. Two call sites,
one line apart, on opposite sides of the contract.
So add a second entry point rather than growing an argument on the first.
tcg_gen_lookup_and_goto_ptr() keeps today's meaning and today's signature:
dispatch, and let the helper work out where. tcg_gen_goto_jc_*() means
dispatch to the destination that env already describes, and takes the pc as
proof that the caller knows which one that is. Targets migrate one call
site at a time, and a call site that cannot promise simply does not move.
The contract is:
- @pc holds exactly what get_tb_cpu_state() reports as the destination pc.
- The flags and cs_base it reports are the ones this block was translated
with, which is what lets them be constants in the generated code.
--enable-debug-tcg checks all three against get_tb_cpu_state() at run time,
via a new helper_goto_jc_check(). That turns a mistake into an assertion at
the offending call site instead of a block that runs with someone else's
flags.
Five targets have a call site whose pc temp is that key by construction, and
are migrated here: alpha, loongarch, mips, ppc and s390x. Nothing else
changes; the generated code does not change either, since goto_jc still
emits the same helper call for now.
For six targets the TB pc is derived and passing the pc temp would be wrong:
avr's TB pc is the word address doubled, i386's is eip before segmentation,
riscv masks it to 32 bits when xl is MXL_RV32, hppa derives it from the IAQ,
hexagon adjusts it inside a hardware loop, and sparc puts npc in cs_base.
The remaining seven -- arm, m68k, microblaze, or1k, rx, sh4 and tricore --
have call sites that look like they could move, but I have not convinced
myself of the contract for them and have nothing to test them with. Each is
a one-line change for whoever wants it, and debug-tcg will say if it is
wrong.
The i32 and i64 forms are separate functions, with a _tl alias in tcg-op.h,
as for most everything else. A translator built for more than one value of
TARGET_LONG_BITS cannot include tcg-op.h and calls the sized form directly,
which is what s390x does here.
Neither form takes the TranslationBlock: tcg_ctx->gen_tb is the block being
generated, the same one tcg_gen_goto_tb() and tcg_gen_lookup_and_goto_ptr()
already read, so there is no way for a caller to pass the wrong one.
v4: Split out of "tcg: probe the TB jump cache inline instead of calling a
helper", which did the API change and the inline probe in one patch.
Requested by Richard Henderson.
v5: Add a new interface rather than growing an argument on
tcg_gen_lookup_and_goto_ptr(), and check the contract under
--enable-debug-tcg. Requested by Richard Henderson, who named it
tcg_gen_goto_jc_*(); the DISAS_UPDATE_NOCHAIN example above is his.
v5: Define _i32 and _i64 entry points with a _tl alias in tcg-op.h, rather
than one entry point taking a TCGTemp. Requested by Richard Henderson:
as targets migrate to single-binary, code is built once and stops
relying on TARGET_LONG_BITS, so the TCGTemp split was the wrong shape.
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
accel/tcg/cpu-exec.c | 27 +++++++++
accel/tcg/tcg-runtime.h | 4 ++
include/tcg/tcg-op-common.h | 20 +++++++
include/tcg/tcg-op.h | 2 +
target/alpha/translate.c | 4 +-
.../tcg/insn_trans/trans_branch.c.inc | 2 +-
target/loongarch/tcg/translate.c | 4 +-
target/mips/tcg/nanomips_translate.c.inc | 2 +-
target/mips/tcg/translate.c | 6 +-
target/ppc/translate.c | 4 +-
target/s390x/tcg/translate.c | 4 +-
tcg/tcg-op.c | 59 +++++++++++++++++--
12 files changed, 119 insertions(+), 19 deletions(-)
diff --git ./accel/tcg/cpu-exec.c ./accel/tcg/cpu-exec.c
index 148e0f583e..ca90a77a7b 100644
--- ./accel/tcg/cpu-exec.c
+++ ./accel/tcg/cpu-exec.c
@@ -407,6 +407,33 @@ const void *HELPER(lookup_tb_ptr)(CPUArchState *env)
return tb->tc.ptr;
}
+#ifdef CONFIG_DEBUG_TCG
+/**
+ * helper_goto_jc_check: check the contract of tcg_gen_goto_jc_*()
+ * @env: current cpu state
+ * @pc: the destination pc the caller passed at translation time
+ * @flags: the flags the dispatching block was translated with
+ * @cs_base: the cs_base the dispatching block was translated with
+ *
+ * A goto_jc looks the destination up on the caller's @pc with the flags and
+ * cs_base of the block doing the dispatching, so all three have to be what
+ * get_tb_cpu_state() reports by the time the dispatch runs. That is a
+ * property of the translator, not of the generated code, so check it here
+ * rather than leaving a target that gets it wrong to be debugged as a block
+ * running with someone else's flags.
+ */
+void HELPER(goto_jc_check)(CPUArchState *env, uint64_t pc, uint64_t flags,
+ uint64_t cs_base)
+{
+ CPUState *cpu = env_cpu(env);
+ TCGTBCPUState s = cpu->cc->tcg_ops->get_tb_cpu_state(cpu);
+
+ assert(s.pc == pc);
+ assert(s.flags == flags);
+ assert(s.cs_base == cs_base);
+}
+#endif
+
/* Return the current PC from CPU, which may be cached in TB. */
static vaddr log_pc(CPUState *cpu, const TranslationBlock *tb)
{
diff --git ./accel/tcg/tcg-runtime.h ./accel/tcg/tcg-runtime.h
index 0b832176b3..ec99170698 100644
--- ./accel/tcg/tcg-runtime.h
+++ ./accel/tcg/tcg-runtime.h
@@ -22,6 +22,10 @@ DEF_HELPER_FLAGS_1(ctpop_i64, TCG_CALL_NO_RWG_SE, i64, i64)
DEF_HELPER_FLAGS_1(lookup_tb_ptr, TCG_CALL_NO_WG_SE, cptr, env)
+#ifdef CONFIG_DEBUG_TCG
+DEF_HELPER_FLAGS_4(goto_jc_check, TCG_CALL_NO_WG_SE, void, env, i64, i64, i64)
+#endif
+
DEF_HELPER_FLAGS_1(exit_atomic, TCG_CALL_NO_WG, noreturn, env)
#ifndef IN_HELPER_PROTO
diff --git ./include/tcg/tcg-op-common.h ./include/tcg/tcg-op-common.h
index 9b321f959c..4f334faaaa 100644
--- ./include/tcg/tcg-op-common.h
+++ ./include/tcg/tcg-op-common.h
@@ -85,6 +85,26 @@ void tcg_gen_goto_tb(unsigned idx);
*/
void tcg_gen_lookup_and_goto_ptr(void);
+/**
+ * tcg_gen_goto_jc_i32() - dispatch to the destination TB via the jump cache
+ * tcg_gen_goto_jc_i64() - dispatch to the destination TB via the jump cache
+ * @pc: temp holding the destination guest PC
+ *
+ * As tcg_gen_lookup_and_goto_ptr(), but the caller states where the
+ * dispatch is going, which allows the lookup to be done inline.
+ *
+ * The contract is that when this runs, the CPU state must already be
+ * exactly the destination's: @pc must hold what get_tb_cpu_state() would
+ * report as the destination pc, and the flags and cs_base it would report
+ * must be the ones the block being generated was translated with. A
+ * translator that has not finished updating the state, or whose pc is
+ * derived rather than being the lookup key -- avr's word address, i386's
+ * eip before segmentation -- must use tcg_gen_lookup_and_goto_ptr()
+ * instead. --enable-debug-tcg checks the contract at runtime.
+ */
+void tcg_gen_goto_jc_i32(TCGv_i32 pc);
+void tcg_gen_goto_jc_i64(TCGv_i64 pc);
+
void tcg_gen_plugin_cb(unsigned from);
void tcg_gen_plugin_mem_cb(TCGv_i64 addr, unsigned meminfo);
diff --git ./include/tcg/tcg-op.h ./include/tcg/tcg-op.h
index 3721164236..cd4794d745 100644
--- ./include/tcg/tcg-op.h
+++ ./include/tcg/tcg-op.h
@@ -38,6 +38,7 @@ typedef TCGv_i32 TCGv;
#define tcgv_tl_temp tcgv_i32_temp
#define tcg_gen_qemu_ld_tl tcg_gen_qemu_ld_i32
#define tcg_gen_qemu_st_tl tcg_gen_qemu_st_i32
+#define tcg_gen_goto_jc_tl tcg_gen_goto_jc_i32
#elif TARGET_LONG_BITS == 64
typedef TCGv_i64 TCGv;
#define tcg_temp_new() tcg_temp_new_i64()
@@ -45,6 +46,7 @@ typedef TCGv_i64 TCGv;
#define tcgv_tl_temp tcgv_i64_temp
#define tcg_gen_qemu_ld_tl tcg_gen_qemu_ld_i64
#define tcg_gen_qemu_st_tl tcg_gen_qemu_st_i64
+#define tcg_gen_goto_jc_tl tcg_gen_goto_jc_i64
#else
#error Unhandled TARGET_LONG_BITS value
#endif
diff --git ./target/alpha/translate.c ./target/alpha/translate.c
index c66e3f9c14..8318487cd8 100644
--- ./target/alpha/translate.c
+++ ./target/alpha/translate.c
@@ -449,7 +449,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_idx, int32_t disp)
tcg_gen_exit_tb(ctx->base.tb, tb_slot_idx);
} else {
gen_pc_disp(ctx, cpu_pc, disp);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_goto_jc_tl(cpu_pc);
}
}
@@ -2917,7 +2917,7 @@ static void alpha_tr_tb_stop(DisasContextBase *dcbase, CPUState *cpu)
gen_pc_disp(ctx, cpu_pc, 0);
/* FALLTHRU */
case DISAS_PC_UPDATED:
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_goto_jc_tl(cpu_pc);
break;
case DISAS_PC_UPDATED_NOCHAIN:
tcg_gen_exit_tb(NULL, 0);
diff --git ./target/loongarch/tcg/insn_trans/trans_branch.c.inc ./target/loongarch/tcg/insn_trans/trans_branch.c.inc
index da07778658..d4318dfa43 100644
--- ./target/loongarch/tcg/insn_trans/trans_branch.c.inc
+++ ./target/loongarch/tcg/insn_trans/trans_branch.c.inc
@@ -27,7 +27,7 @@ static bool trans_jirl(DisasContext *ctx, arg_jirl *a)
tcg_gen_mov_tl(cpu_pc, addr);
tcg_gen_movi_tl(dest, make_address_pc(ctx, ctx->base.pc_next + 4));
gen_set_gpr(a->rd, dest, EXT_NONE);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_goto_jc_tl(cpu_pc);
ctx->base.is_jmp = DISAS_NORETURN;
return true;
}
diff --git ./target/loongarch/tcg/translate.c ./target/loongarch/tcg/translate.c
index 124dce6269..6ac0c3773a 100644
--- ./target/loongarch/tcg/translate.c
+++ ./target/loongarch/tcg/translate.c
@@ -111,7 +111,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_idx, vaddr dest)
tcg_gen_exit_tb(ctx->base.tb, tb_slot_idx);
} else {
tcg_gen_movi_tl(cpu_pc, dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_goto_jc_tl(cpu_pc);
}
}
@@ -311,7 +311,7 @@ static void loongarch_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
switch (ctx->base.is_jmp) {
case DISAS_STOP:
tcg_gen_movi_tl(cpu_pc, ctx->base.pc_next);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_goto_jc_tl(cpu_pc);
break;
case DISAS_TOO_MANY:
gen_goto_tb(ctx, 0, ctx->base.pc_next);
diff --git ./target/mips/tcg/nanomips_translate.c.inc ./target/mips/tcg/nanomips_translate.c.inc
index 4b0b01ba37..106f49990d 100644
--- ./target/mips/tcg/nanomips_translate.c.inc
+++ ./target/mips/tcg/nanomips_translate.c.inc
@@ -2406,7 +2406,7 @@ static void gen_compute_nanomips_pbalrsc_branch(DisasContext *ctx, int rs,
/* unconditional branch to register */
tcg_gen_mov_tl(cpu_PC, btarget);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_goto_jc_tl(cpu_PC);
}
/* nanoMIPS Branches */
diff --git ./target/mips/tcg/translate.c ./target/mips/tcg/translate.c
index e3467d1525..dea1ba4c1e 100644
--- ./target/mips/tcg/translate.c
+++ ./target/mips/tcg/translate.c
@@ -4374,7 +4374,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_idx,
tcg_gen_exit_tb(ctx->base.tb, tb_slot_idx);
} else {
gen_save_pc(dest);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_goto_jc_tl(cpu_PC);
}
}
@@ -11014,7 +11014,7 @@ static void gen_branch(DisasContext *ctx, int insn_bytes)
} else {
tcg_gen_mov_tl(cpu_PC, btarget);
}
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_goto_jc_tl(cpu_PC);
break;
default:
LOG_DISAS("unknown branch 0x%x\n", proc_hflags);
@@ -15244,7 +15244,7 @@ static void mips_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
switch (ctx->base.is_jmp) {
case DISAS_STOP:
gen_save_pc(ctx->base.pc_next);
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_goto_jc_tl(cpu_PC);
break;
case DISAS_NEXT:
case DISAS_TOO_MANY:
diff --git ./target/ppc/translate.c ./target/ppc/translate.c
index 06ed2adf10..21e21102fc 100644
--- ./target/ppc/translate.c
+++ ./target/ppc/translate.c
@@ -3664,7 +3664,7 @@ static void gen_lookup_and_goto_ptr(DisasContext *ctx)
pmu_count_insns(ctx);
}
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_goto_jc_tl(cpu_nip);
}
}
@@ -6690,7 +6690,7 @@ static void ppc_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
pmu_count_insns(ctx);
}
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_goto_jc_tl(cpu_nip);
break;
case DISAS_EXIT_UPDATE:
diff --git ./target/s390x/tcg/translate.c ./target/s390x/tcg/translate.c
index 1b6023168b..906951c7df 100644
--- ./target/s390x/tcg/translate.c
+++ ./target/s390x/tcg/translate.c
@@ -1162,7 +1162,7 @@ static DisasJumpType help_branch(DisasContext *s, DisasCompare *c,
tcg_gen_goto_tb(0);
tcg_gen_exit_tb(s->base.tb, 0);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_goto_jc_i64(psw_addr);
}
gen_set_label(lab);
@@ -6477,7 +6477,7 @@ static void s390x_tr_tb_stop(DisasContextBase *dcbase, CPUState *cs)
if (dc->exit_to_mainloop) {
tcg_gen_exit_tb(NULL, 0);
} else {
- tcg_gen_lookup_and_goto_ptr();
+ tcg_gen_goto_jc_i64(psw_addr);
}
break;
default:
diff --git ./tcg/tcg-op.c ./tcg/tcg-op.c
index 28d3b2a847..a2f35359fe 100644
--- ./tcg/tcg-op.c
+++ ./tcg/tcg-op.c
@@ -2715,18 +2715,65 @@ void tcg_gen_goto_tb(unsigned idx)
tcg_gen_op1i(INDEX_op_goto_tb, 0, idx);
}
+static void gen_lookup_tb_ptr_and_goto(void)
+{
+ TCGv_ptr ptr = tcg_temp_ebb_new_ptr();
+
+ gen_helper_lookup_tb_ptr(ptr, tcg_env);
+ tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr));
+ tcg_temp_free_ptr(ptr);
+}
+
void tcg_gen_lookup_and_goto_ptr(void)
{
- TCGv_ptr ptr;
-
if (tcg_ctx->gen_tb->cflags & CF_NO_GOTO_PTR) {
tcg_gen_exit_tb(NULL, 0);
return;
}
plugin_gen_disable_mem_helpers();
- ptr = tcg_temp_ebb_new_ptr();
- gen_helper_lookup_tb_ptr(ptr, tcg_env);
- tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr));
- tcg_temp_free_ptr(ptr);
+ gen_lookup_tb_ptr_and_goto();
+}
+
+/*
+ * The common half of tcg_gen_goto_jc_i32() and tcg_gen_goto_jc_i64(). @pc
+ * is widened to i64 because the jump cache is keyed on a vaddr; for a
+ * 32-bit guest PC that is its zero extension.
+ */
+static void gen_goto_jc(TCGv_i64 pc)
+{
+ const TranslationBlock *tb = tcg_ctx->gen_tb;
+
+ if (tb->cflags & CF_NO_GOTO_PTR) {
+ tcg_gen_exit_tb(NULL, 0);
+ return;
+ }
+
+ plugin_gen_disable_mem_helpers();
+
+#ifdef CONFIG_DEBUG_TCG
+ /*
+ * The caller has asserted that env already describes the destination.
+ * Check it, rather than leaving a target that gets it wrong to be
+ * debugged as a block that runs with someone else's flags.
+ */
+ gen_helper_goto_jc_check(tcg_env, pc, tcg_constant_i64(tb->flags),
+ tcg_constant_i64(tb->cs_base));
+#endif
+
+ gen_lookup_tb_ptr_and_goto();
+}
+
+void tcg_gen_goto_jc_i64(TCGv_i64 pc)
+{
+ gen_goto_jc(pc);
+}
+
+void tcg_gen_goto_jc_i32(TCGv_i32 pc)
+{
+ TCGv_i64 pc64 = tcg_temp_ebb_new_i64();
+
+ tcg_gen_extu_i32_i64(pc64, pc);
+ gen_goto_jc(pc64);
+ tcg_temp_free_i64(pc64);
}
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH v5 5/9] accel/tcg: add CF_NO_GOTO_JC, set while a breakpoint is present
2026-09-01 3:47 ` [PATCH v5 " Matt Turner
` (3 preceding siblings ...)
2026-09-01 3:48 ` [PATCH v5 4/9] tcg: add tcg_gen_goto_jc_{i32,i64,tl}() Matt Turner
@ 2026-09-01 3:48 ` Matt Turner
2026-09-01 3:48 ` [PATCH v5 6/9] RFC: tcg: probe the TB jump cache inline instead of calling a helper Matt Turner
` (3 subsequent siblings)
8 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-09-01 3:48 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
The next patch dispatches a goto_jc by probing the TB jump cache from
generated code. That probe cannot check everything helper_lookup_tb_ptr()
checks, and the one that matters is breakpoints: check_for_breakpoints()
raises EXCP_DEBUG on an exact pc match and selects CF_BP_PAGE cflags for the
rest of the page, and inserting a breakpoint deliberately invalidates no TB.
The probe does compare the destination's cflags against the cflags of the
block doing the dispatching, and only takes the destination when they are
equal. So a cflag is all that is needed. Add CF_NO_GOTO_JC, set it in
CPUState::tcg_cflags while cpu->breakpoints is non-empty, and blocks
translated from then on both decline to dispatch inline themselves -- the
next patch makes them emit the plain helper call -- and are unreachable from
blocks that do, because their cflags no longer match.
The two ends of the flag are cpu_breakpoint_insert() and
cpu_breakpoint_remove_by_ref(), which are the only places the list changes.
Both already run either on the CPU's own thread or with it stopped, or reach
another CPU exactly as cpu_single_step() does, which is where the previous
patch put the same kind of update.
That leaves blocks translated before the breakpoint was inserted, which are
still live and still chain to each other. They do so on the old cflags, so
inline dispatch among them keeps working until the vCPU reaches its main
loop, which then looks up with the new cflags and translates afresh. In
system mode gdb inserts breakpoints with the vCPUs stopped, so there is no
window at all. In user mode the window is the one goto_tb chaining already
has: a chained direct jump consults nothing either, and is not broken by
inserting a breakpoint.
Nothing reads CF_NO_GOTO_JC yet; the next patch does.
v5: New patch, replacing "accel/tcg: give the TB jump cache a second base
pointer for generated code", which forced the same fallback by pointing
generated code at a zero-filled jump cache when a breakpoint was
inserted, and needed a cross-thread poison and an un-poison race to do
it. Richard Henderson suggested a cflag instead, and pointed out that
the previous patch had already shown how to update tcg_cflags from
gdbstub. The base pointer comes back later in the series, for pending
exits, which a cflag cannot express.
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
accel/tcg/cpu-exec-common.c | 13 ++++++++++++-
cpu-common.c | 7 +++++++
include/exec/translation-block.h | 1 +
3 files changed, 20 insertions(+), 1 deletion(-)
diff --git ./accel/tcg/cpu-exec-common.c ./accel/tcg/cpu-exec-common.c
index 9f3517f36b..a3148bbf8f 100644
--- ./accel/tcg/cpu-exec-common.c
+++ ./accel/tcg/cpu-exec-common.c
@@ -41,7 +41,7 @@ void tcg_cflags_set(CPUState *cpu, uint32_t flags)
* they are derived from gdb single-step, one-insn-per-tb and -d nochain.
*/
#define CF_DERIVED (CF_COUNT_MASK | CF_NO_GOTO_TB | CF_NO_GOTO_PTR | \
- CF_SINGLE_STEP)
+ CF_SINGLE_STEP | CF_NO_GOTO_JC)
void tcg_update_cflags(CPUState *cpu)
{
@@ -62,6 +62,17 @@ void tcg_update_cflags(CPUState *cpu)
cflags |= CF_NO_GOTO_TB;
}
+ /*
+ * A block that dispatches through the jump cache inline does not consult
+ * cpu->breakpoints, and inserting a breakpoint deliberately invalidates
+ * nothing. Give blocks translated while one is set a distinct cflags, so
+ * that they neither dispatch inline themselves nor are reached by a block
+ * that does, and check_for_breakpoints() gets to run on every dispatch.
+ */
+ if (unlikely(!QTAILQ_EMPTY(&cpu->breakpoints))) {
+ cflags |= CF_NO_GOTO_JC;
+ }
+
cpu->tcg_cflags = cflags;
}
diff --git ./cpu-common.c ./cpu-common.c
index adb76b3a78..3178601987 100644
--- ./cpu-common.c
+++ ./cpu-common.c
@@ -22,6 +22,7 @@
#include "exec/cpu-common.h"
#include "hw/core/cpu.h"
#include "qemu/lockable.h"
+#include "system/tcg.h"
#include "trace/trace-root.h"
QemuMutex qemu_cpu_list_lock;
@@ -429,6 +430,9 @@ int cpu_breakpoint_insert(CPUState *cpu, vaddr pc, int flags,
*breakpoint = bp;
}
+ /* The first breakpoint takes the CPU off the inline dispatch path. */
+ tcg_update_cflags(cpu);
+
trace_breakpoint_insert(cpu->cpu_index, pc, flags);
return 0;
}
@@ -456,6 +460,9 @@ void cpu_breakpoint_remove_by_ref(CPUState *cpu, CPUBreakpoint *bp)
{
QTAILQ_REMOVE(&cpu->breakpoints, bp, entry);
+ /* The last breakpoint puts the CPU back on it. */
+ tcg_update_cflags(cpu);
+
trace_breakpoint_remove(cpu->cpu_index, bp->pc, bp->flags);
g_free(bp);
}
diff --git ./include/exec/translation-block.h ./include/exec/translation-block.h
index 40cc699031..8c4778c681 100644
--- ./include/exec/translation-block.h
+++ ./include/exec/translation-block.h
@@ -84,6 +84,7 @@ struct TranslationBlock {
#define CF_NOIRQ 0x00010000 /* Generate an uninterruptible TB */
#define CF_PCREL 0x00020000 /* Opcodes in TB are PC-relative */
#define CF_BP_PAGE 0x00040000 /* Breakpoint present in code page */
+#define CF_NO_GOTO_JC 0x00080000 /* Do not dispatch via the inline probe */
#define CF_CLUSTER_MASK 0xff000000 /* Top 8 bits are cluster ID */
#define CF_CLUSTER_SHIFT 24
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH v5 6/9] RFC: tcg: probe the TB jump cache inline instead of calling a helper
2026-09-01 3:47 ` [PATCH v5 " Matt Turner
` (4 preceding siblings ...)
2026-09-01 3:48 ` [PATCH v5 5/9] accel/tcg: add CF_NO_GOTO_JC, set while a breakpoint is present Matt Turner
@ 2026-09-01 3:48 ` Matt Turner
2026-09-01 3:48 ` [PATCH v5 7/9] RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds Matt Turner
` (2 subsequent siblings)
8 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-09-01 3:48 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
Every indirect branch that cannot use goto_tb ends in a dispatch that calls
helper_lookup_tb_ptr(). For an emulated compiler that is 8.4 billion helper
calls in a single translation unit: 24.6% of all TB exits take this path,
because jsr/ret/jmp have a register destination and because goto_tb is
restricted to same-page targets.
The helper itself is already tight, but each call pays for a call frame,
the can_do_io store, the get_tb_cpu_state() indirect call through
TCGCPUOps, curr_cflags(), and a breakpoint check, before it gets to the
jump cache probe that almost always hits (95.8% for this workload).
Emit the probe inline instead, for the callers that have migrated to
tcg_gen_goto_jc_*(). Those supply what it needs: the destination PC is in a
TCG temp, and the flags, cflags and cs_base the destination must match are
constants at translation time. The fast path is therefore a hash, four
guarded loads and a goto_ptr. Only a miss calls the helper, which still owns
filling the cache.
Two details matter for the generated code. The flags and cflags guards are
folded into a single aligned 64-bit load and compare, since the fields are
adjacent. And each path emits its own goto_ptr rather than branching to a
shared one: a temp live across the label is spilled and reloaded on every
dispatch, which cost 6.3% on its own.
The flags and cflags constants are safe against the other things that can
change them. CF_PARALLEL is only ever set by begin_parallel_context(),
which flushes first, so no block predating it survives to dispatch. gdb
single-step is only turned on with the CPU stopped, and a block translated
without CF_SINGLE_STEP can only be re-entered through tb_lookup(), which
from then on demands the new cflags -- so a stale-cflags block is never the
one running. Breakpoints are handled by CF_NO_GOTO_JC, added by the previous
patch: while one is set, blocks are translated with a cflags that both keeps
them off the inline path and keeps them unreachable from blocks already on
it. What is left is one_insn_per_tb and -d nochain; see below.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling
the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, LTO
build, on top of the preceding patches:
before: 1,402,667,803,616 instructions
after: 916,415,123,244 instructions -34.67%
before: 115.56s wall clock
after: 85.59s wall clock -25.94%
The gap between the two is the point at which this stops being a
straight-line win: the helper call was highly predictable work that the
host pipelined well, so removing it retires far fewer instructions than it
saves time. IPC falls from 2.48 to 2.17 across this patch for that reason.
Despite emitting more code, this also reduces instruction cache pressure,
because a dispatch no longer jumps into qemu's .text and evicts translated
code:
before: 11,735,141,703 L1-icache-load-misses
after: 7,154,863,292 L1-icache-load-misses -39.0%
The mechanism is visible directly in a profile: helper_lookup_tb_ptr()
falls from 31.01% of samples to 0.35%, and qemu's own .text falls from
38.8% to 5.3%, with the balance moving into generated code.
Combined with the preceding patches, against an unmodified LTO build,
1,646,994,254,249 instructions fall to 916,415,123,244, or -44.36%. The
emulated compiler produces byte-identical output throughout.
A follow-up worth having: the probe is emitted entirely out of generic TCG
ops, and several backends can do much better than the result. x86_64 and
s390x have memory-operand comparisons; aarch64 can form env + off + h * 16
with a shift-add, load (tb, pc) and (cs_base, flags) with two ldp, and halve
the branches with ccmp. That wants a backend expansion of a dedicated
opcode, which is a separate series.
Open issues, hence RFC:
- one_insn_per_tb and CPU_LOG_TB_NOCHAIN can be toggled from the monitor
while a vCPU is inside a block that was translated without them. The
block keeps dispatching inline on the old cflags until it exits for some
other reason. This is the same window goto_tb chaining already has, since
a chained direct jump consults nothing either, but it is worth saying out
loud.
- The jump cache entry is read without qatomic_read(); entries are
invalidated concurrently by setting tb to NULL.
- Only alpha has been measured. The other four targets that use goto_jc are
built and boot-tested only.
v4: Split out of the patch that also changed the
tcg_gen_lookup_and_goto_ptr() API and introduced tb_jmp_cache_probe,
which are now the two preceding patches. Requested by Richard
Henderson.
v4: Emit the softmmu form of tb_jmp_cache_hash_func() under
CONFIG_SOFTMMU rather than the user-only form everywhere. v3 emitted
the user-only hash unconditionally, which was wrong for system mode
and was only not a correctness bug because a wrong index simply
misses. Caught by Richard Henderson. tcg-op.c is compiled once per
build rather than once per target, but CONFIG_SOFTMMU is set for it,
and TARGET_PAGE_BITS -- a load from target_page here -- is fixed long
before any translation happens.
v4: Compare the pc before testing tb for NULL. On a hash miss the pc is
the field most likely to differ, and an unused entry has a zero pc
that only pc 0 can match, so the tb test buys nothing ahead of it.
Suggested by Richard Henderson.
v4: Assert that offsetof(TranslationBlock, flags) is 8-byte aligned, since
folding the flags and cflags guards into one 64-bit load relies on it
and nothing else does. Requested by Richard Henderson.
v4: Zero-extend a 32-bit guest PC instead of falling back to the helper.
Suggested by Richard Henderson. The high half then folds to a compare
against zero.
v4: Describe cs_base in the probe as a second word of target-specific
flags rather than by name. Suggested by Richard Henderson.
v5: Build the folded flags/cflags constant with deposit64() rather than
under #if HOST_BIG_ENDIAN, so both arms compile on every host.
Requested by Richard Henderson.
v5: Read cpu->tb_jmp_cache directly, and honor CF_NO_GOTO_JC rather than a
poisoned base pointer, which is no longer how breakpoints are handled.
A separate base pointer comes back later in the series for pending
exits.
v5: Note the backend expansion this wants as a follow-up. Suggested by
Richard Henderson, whose list it is.
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
include/tcg/tcg-op-common.h | 3 +-
tcg/tcg-op.c | 110 +++++++++++++++++++++++++++++++++++-
2 files changed, 111 insertions(+), 2 deletions(-)
diff --git ./include/tcg/tcg-op-common.h ./include/tcg/tcg-op-common.h
index 4f334faaaa..f41f3ee58f 100644
--- ./include/tcg/tcg-op-common.h
+++ ./include/tcg/tcg-op-common.h
@@ -91,7 +91,8 @@ void tcg_gen_lookup_and_goto_ptr(void);
* @pc: temp holding the destination guest PC
*
* As tcg_gen_lookup_and_goto_ptr(), but the caller states where the
- * dispatch is going, which allows the lookup to be done inline.
+ * dispatch is going, so the TB jump cache is probed inline and only a miss
+ * reaches helper_lookup_tb_ptr().
*
* The contract is that when this runs, the CPU state must already be
* exactly the destination's: @pc must hold what get_tb_cpu_state() would
diff --git ./tcg/tcg-op.c ./tcg/tcg-op.c
index a2f35359fe..b10b2d66d5 100644
--- ./tcg/tcg-op.c
+++ ./tcg/tcg-op.c
@@ -28,6 +28,8 @@
#include "tcg/tcg-op-common.h"
#include "exec/translation-block.h"
#include "exec/plugin-gen.h"
+#include "hw/core/cpu.h"
+#include "../accel/tcg/tb-hash.h"
#include "tcg-internal.h"
#include "tcg-has.h"
@@ -2735,6 +2737,102 @@ void tcg_gen_lookup_and_goto_ptr(void)
gen_lookup_tb_ptr_and_goto();
}
+static void gen_jmp_cache_hash(TCGv_i64 h, TCGv_i64 pc)
+{
+#ifdef CONFIG_SOFTMMU
+ /*
+ * tb_jmp_cache_hash_func(), softmmu form. TARGET_PAGE_BITS is a load
+ * from target_page in this translation unit, but it is decided long
+ * before any translation happens, so it is a constant here.
+ */
+ int shift = TARGET_PAGE_BITS - TB_JMP_PAGE_BITS;
+ TCGv_i64 tmp = tcg_temp_ebb_new_i64();
+
+ tcg_gen_shri_i64(tmp, pc, shift);
+ tcg_gen_xor_i64(tmp, tmp, pc);
+ tcg_gen_shri_i64(h, tmp, shift);
+ tcg_gen_andi_i64(h, h, TB_JMP_PAGE_MASK);
+ tcg_gen_andi_i64(tmp, tmp, TB_JMP_ADDR_MASK);
+ tcg_gen_or_i64(h, h, tmp);
+ tcg_temp_free_i64(tmp);
+#else
+ /* tb_jmp_cache_hash_func(), user-only form. */
+ tcg_gen_shri_i64(h, pc, TB_JMP_CACHE_BITS);
+ tcg_gen_xor_i64(h, h, pc);
+ tcg_gen_andi_i64(h, h, TB_JMP_CACHE_SIZE - 1);
+#endif
+}
+
+static void gen_jmp_cache_probe(TCGv_i64 pc, const TranslationBlock *tb)
+{
+ TCGv_ptr jc, ent, tbp, ptr;
+ TCGv_i64 h, tmp;
+ TCGLabel *slow;
+ uint64_t fpair;
+
+ QEMU_BUILD_BUG_ON(sizeof(((CPUJumpCache *)0)->array[0]) != 16);
+ QEMU_BUILD_BUG_ON(offsetof(CPUJumpCache, array[0].pc) % 8 != 0);
+ /* One 64-bit load has to cover both, so they must be adjacent... */
+ QEMU_BUILD_BUG_ON(offsetof(TranslationBlock, cflags) !=
+ offsetof(TranslationBlock, flags) + 4);
+ /* ...and aligned, which nothing else currently relies on. */
+ QEMU_BUILD_BUG_ON(offsetof(TranslationBlock, flags) % 8 != 0);
+
+ jc = tcg_temp_ebb_new_ptr();
+ ent = tcg_temp_ebb_new_ptr();
+ tbp = tcg_temp_ebb_new_ptr();
+ ptr = tcg_temp_ebb_new_ptr();
+ h = tcg_temp_ebb_new_i64();
+ tmp = tcg_temp_ebb_new_i64();
+ slow = gen_new_label();
+
+ /* ent = &jc->array[tb_jmp_cache_hash_func(pc)] */
+ gen_jmp_cache_hash(h, pc);
+ tcg_gen_shli_i64(h, h, 4);
+
+ tcg_gen_ld_ptr(jc, tcg_env,
+ offsetof(CPUState, tb_jmp_cache) - sizeof(CPUState));
+ tcg_gen_trunc_i64_ptr(ent, h);
+ tcg_gen_add_ptr(ent, jc, ent);
+
+ /*
+ * The pc first: on a hash miss it is the field most likely to differ,
+ * and an entry whose tb is NULL has a zero pc that only pc 0 matches.
+ */
+ tcg_gen_ld_i64(tmp, ent, offsetof(CPUJumpCache, array[0].pc));
+ tcg_gen_brcond_i64(TCG_COND_NE, tmp, pc, slow);
+
+ tcg_gen_ld_ptr(tbp, ent, offsetof(CPUJumpCache, array[0].tb));
+ tcg_gen_brcondi_ptr(TCG_COND_EQ, tbp, 0, slow);
+
+ /*
+ * flags and cflags are adjacent uint32_t, so one aligned 64-bit load
+ * and compare covers both.
+ */
+ fpair = (HOST_BIG_ENDIAN
+ ? deposit64(tb->cflags, 32, 32, tb->flags)
+ : deposit64(tb->flags, 32, 32, tb->cflags));
+ tcg_gen_ld_i64(tmp, tbp, offsetof(TranslationBlock, flags));
+ tcg_gen_brcondi_i64(TCG_COND_NE, tmp, fpair, slow);
+
+ /*
+ * cs_base is a second word of target-specific flags despite the name,
+ * and the pc alone does not imply it on a target that uses it.
+ */
+ tcg_gen_ld_i64(tmp, tbp, offsetof(TranslationBlock, cs_base));
+ tcg_gen_brcondi_i64(TCG_COND_NE, tmp, tb->cs_base, slow);
+
+ tcg_gen_ld_ptr(ptr, tbp, offsetof(TranslationBlock, tc.ptr));
+ tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr));
+
+ /*
+ * Emit a second goto_ptr rather than branching to a shared one: a temp
+ * live across the label would be spilled and reloaded on every dispatch.
+ */
+ gen_set_label(slow);
+ gen_lookup_tb_ptr_and_goto();
+}
+
/*
* The common half of tcg_gen_goto_jc_i32() and tcg_gen_goto_jc_i64(). @pc
* is widened to i64 because the jump cache is keyed on a vaddr; for a
@@ -2761,7 +2859,17 @@ static void gen_goto_jc(TCGv_i64 pc)
tcg_constant_i64(tb->cs_base));
#endif
- gen_lookup_tb_ptr_and_goto();
+ /*
+ * A breakpoint is the one thing the probe cannot check for itself, so
+ * while one is set the flag is set too and every dispatch takes the
+ * helper, which does check. See tcg_update_cflags().
+ */
+ if (tb->cflags & CF_NO_GOTO_JC) {
+ gen_lookup_tb_ptr_and_goto();
+ return;
+ }
+
+ gen_jmp_cache_probe(pc, tb);
}
void tcg_gen_goto_jc_i64(TCGv_i64 pc)
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH v5 7/9] RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds
2026-09-01 3:47 ` [PATCH v5 " Matt Turner
` (5 preceding siblings ...)
2026-09-01 3:48 ` [PATCH v5 6/9] RFC: tcg: probe the TB jump cache inline instead of calling a helper Matt Turner
@ 2026-09-01 3:48 ` Matt Turner
2026-09-01 3:48 ` [PATCH v5 8/9] RFC: accel/tcg: poison the jump cache instead of polling for indirect exits Matt Turner
2026-09-01 3:48 ` [PATCH v5 9/9] RFC: tcg: fold a guest displacement into the host addressing mode Matt Turner
8 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-09-01 3:48 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
translator_use_goto_tb() refuses to chain unless the destination is on the
same page as the start of the TB. For guests whose text is much larger than
a page this is expensive: an emulated alpha gcc compiling a 255k line
translation unit takes the indirect dispatch path for 8.4 billion of its
34.2 billion TB exits, and a large share of those are ordinary direct
branches that simply crossed an 8 KiB page boundary.
The restriction was made unconditional by d3a2a1d803 ("accel/tcg:
Introduce translator_use_goto_tb"), whose rationale was:
Various targets avoid the page crossing test for CONFIG_USER_ONLY,
but that is wrong: mmap and mprotect can change page permissions.
That is true, but in user-only builds the invalidation path already covers
it. There are no page tables: every mmap, mprotect and munmap reaches
page_set_flags(), which calls tb_invalidate_phys_range() whenever the flags
actually change, and tb_phys_invalidate() calls tb_jmp_unlink() to reset
incoming jumps. A chained cross-page jump is therefore broken whenever the
destination page's permissions change. This is not true in system mode,
where TBs are keyed by physical address and a page table change invalidates
nothing, so the restriction is kept there.
The rule protects one more thing, which the original rationale does not
mention: it guarantees that execution cannot enter a page without a TB
lookup, and so without check_for_breakpoints(). That is what makes a
breakpoint set after a block was translated take effect, since insertion
deliberately invalidates nothing. A link established before the breakpoint
was set would jump straight over it.
So the chaining is only enabled for a run that can never acquire a
breakpoint. In user-only mode every breakpoint comes from gdb -- BP_CPU is
g_assert_not_reached() there, and the guest cannot ask for one -- and gdb
has to be requested with -g before the first block is translated, even
though with suspend=n it may connect later. gdb_may_set_breakpoints()
reports whether it was, and is fixed for the lifetime of the process.
Add tests/tcg/multiarch/test-xpage-chain.c to cover both hazards directly.
It writes the last instruction of one page and the first of the next, so
that the fall-through between them is a cross-page goto_tb, runs it 200000
times so the chain is established, then checks that mprotect(PROT_NONE)
makes the next call fault, and that different code written into the page
once it is mapped back runs rather than a stale translation.
The two instructions -- set the return value register, and return -- are
all the architecture specific code there is; thirteen architectures supply
them and the rest skip.
The test detects the hazard it is meant to detect: with the
tb_invalidate_phys_range() call in page_set_flags() commented out, it fails
both phases, executing page B after PROT_NONE and returning the stale
result.
Run with -b, the same binary stops once the chain is established and lets
tests/tcg/multiarch/gdbstub/xpage-bp.py set a breakpoint on the far side of it,
which the next call has to stop on. With gdb_may_set_breakpoints() forced to
false so that the chaining stays on under gdb, that breakpoint is missed and
the test fails, which is what makes it a test of the gate rather than of
gdb.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the
SQLite 3.45.1 amalgamation on an x86-64 host, LTO build, on top of the
preceding patches:
before: 916,415,123,244 instructions
after: 891,254,240,071 instructions -2.75%
before: 85.59s wall clock
after: 81.45s wall clock -4.84%
Note that this is worth more in time than in instructions, the reverse of
the preceding patch: a chained jump replaces a cache probe whose loads can
miss, so the instructions it removes are more expensive than average.
Measured before the inline jump cache probe, when a missed chain cost a
helper call rather than an inline probe, the same change was worth -7.9%.
RFC because this reverses a deliberate decision and the reasoning above
wants review from someone who knows the invalidation paths better than I
do.
v3: Only take the shortcut when no gdbstub was requested. The same-page
rule also forces a lookup, and so a breakpoint check, on entry to every
page; without that, a chain established before a breakpoint was set runs
past it. Reported by Richard Henderson.
v3: Change translator_use_goto_tb() rather than translator_is_same_page().
i386, riscv and s390x call translator_is_same_page() for something else
-- enforcing that only a single-insn TB may cross a page -- and v2
changed their TB boundaries in user-only mode as a side effect. alpha
does not call it, so the numbers above are unaffected.
v3: Add the gdbstub half of the test.
v4: Move the test to tests/tcg/multiarch so that every *-user target runs
it, rather than only alpha. Requested by Alex Bennee. The direct branch
is gone with it: a fall-through off the end of a page is a cross-page
goto_tb just the same, and needs no per-architecture branch encoding or
displacement arithmetic, only "set the return value" and "return".
Built and run under qemu-user on aarch64, alpha, arm, hppa,
loongarch64, m68k, mips, ppc, ppc64le, riscv64, s390x, sh4, sparc64
and x86_64; ppc64 ELFv1 skips, because a function pointer there is a
descriptor rather than a code address.
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
accel/tcg/translator.c | 33 ++-
gdbstub/user.c | 14 +
include/gdbstub/user.h | 11 +
tests/tcg/multiarch/Makefile.target | 12 +-
tests/tcg/multiarch/gdbstub/xpage-bp.py | 37 +++
tests/tcg/multiarch/test-xpage-chain.c | 336 ++++++++++++++++++++++++
6 files changed, 441 insertions(+), 2 deletions(-)
create mode 100644 tests/tcg/multiarch/gdbstub/xpage-bp.py
create mode 100644 tests/tcg/multiarch/test-xpage-chain.c
diff --git ./accel/tcg/translator.c ./accel/tcg/translator.c
index 6c8fcd7a20..8879cd626f 100644
--- ./accel/tcg/translator.c
+++ ./accel/tcg/translator.c
@@ -15,6 +15,9 @@
#include "accel/tcg/cpu-mmu-index.h"
#include "exec/target_page.h"
#include "exec/translator.h"
+#ifdef CONFIG_USER_ONLY
+#include "gdbstub/user.h"
+#endif
#include "exec/plugin-gen.h"
#include "tcg/tcg-op-common.h"
#include "internal-common.h"
@@ -110,6 +113,34 @@ bool translator_is_same_page(const DisasContextBase *db, vaddr addr)
return ((addr ^ db->pc_first) & TARGET_PAGE_MASK) == 0;
}
+/*
+ * Whether a direct jump may be chained to a destination outside the page
+ * the TB started in.
+ *
+ * In user-only mode there are no page tables. Every mmap, mprotect and
+ * munmap goes through page_set_flags(), which calls tb_invalidate_phys_range()
+ * whenever the flags actually change, and tb_phys_invalidate() unlinks
+ * incoming jumps. A cross-page link is therefore broken whenever the
+ * destination page's permissions change.
+ *
+ * What the same-page rule also provides is that execution cannot enter a page
+ * without a TB lookup, and so without check_for_breakpoints(), which is what
+ * makes a breakpoint set after a block was translated take effect. Nothing
+ * invalidates on breakpoint insertion, so a link established beforehand would
+ * jump straight over it. In user-only mode breakpoints only ever come from
+ * gdb -- BP_CPU is g_assert_not_reached() there and the guest has no way to
+ * ask for one -- and gdb has to be requested with -g before the first block
+ * is translated, so a run that has no gdbstub can never acquire a breakpoint.
+ */
+static bool use_cross_page_goto_tb(void)
+{
+#ifdef CONFIG_USER_ONLY
+ return !gdb_may_set_breakpoints();
+#else
+ return false;
+#endif
+}
+
bool translator_use_goto_tb(DisasContextBase *db, vaddr dest)
{
/* Suppress goto_tb if requested. */
@@ -118,7 +149,7 @@ bool translator_use_goto_tb(DisasContextBase *db, vaddr dest)
}
/* Check for the dest on the same page as the start of the TB. */
- return translator_is_same_page(db, dest);
+ return use_cross_page_goto_tb() || translator_is_same_page(db, dest);
}
void translator_loop(CPUState *cpu, TranslationBlock *tb, int *max_insns,
diff --git ./gdbstub/user.c ./gdbstub/user.c
index 9e6f9a6f37..d810f0f38c 100644
--- ./gdbstub/user.c
+++ ./gdbstub/user.c
@@ -470,6 +470,18 @@ static void *gdbserver_accept_thread(void *arg)
#define USAGE "\nUsage: -g {port|path}[,suspend={y|n}]"
+/*
+ * Set before the guest runs and never cleared, so that code translated at
+ * any point can rely on it: with suspend=n gdb may connect long after
+ * startup, and once connected it can insert a breakpoint at any time.
+ */
+static bool gdbserver_requested;
+
+bool gdb_may_set_breakpoints(void)
+{
+ return gdbserver_requested;
+}
+
bool gdbserver_start(const char *args, Error **errp)
{
g_auto(GStrv) argv = g_strsplit(args, ",", 0);
@@ -513,6 +525,8 @@ bool gdbserver_start(const char *args, Error **errp)
return false;
}
+ gdbserver_requested = true;
+
if (suspend) {
if (gdbserver_accept(port, gdb_fd, port_or_path)) {
gdb_handlesig(first_cpu, 0, NULL, NULL, 0);
diff --git ./include/gdbstub/user.h ./include/gdbstub/user.h
index 654986d483..c091cd9758 100644
--- ./include/gdbstub/user.h
+++ ./include/gdbstub/user.h
@@ -11,6 +11,17 @@
#define MAX_SIGINFO_LENGTH 128
+/**
+ * gdb_may_set_breakpoints() - whether a breakpoint can ever be inserted
+ *
+ * In user-only mode every breakpoint comes from gdb, and gdb is only ever
+ * reachable if -g was given at startup, before the guest ran a single
+ * instruction. A run that has no gdbstub can therefore never acquire a
+ * breakpoint, which lets translation take shortcuts that a breakpoint
+ * would invalidate. Stays true once true, even if gdb detaches.
+ */
+bool gdb_may_set_breakpoints(void);
+
/**
* gdb_handlesig() - yield control to gdb
* @cpu: CPU
diff --git ./tests/tcg/multiarch/Makefile.target ./tests/tcg/multiarch/Makefile.target
index ab4bf9c5d5..f8a91fed2c 100644
--- ./tests/tcg/multiarch/Makefile.target
+++ ./tests/tcg/multiarch/Makefile.target
@@ -143,6 +143,15 @@ run-gdbstub-follow-fork-mode-parent: follow-fork-mode
--bin $< --test $(MULTIARCH_SRC)/gdbstub/follow-fork-mode-parent.py, \
following parents on fork)
+# The chaining this exercises is only enabled when no gdbstub was requested,
+# so what is under test here is that requesting one turns it back off.
+run-gdbstub-xpage-bp: test-xpage-chain
+ $(call run-test, $@, $(GDB_SCRIPT) \
+ --gdb $(GDB) \
+ --qemu $(QEMU) --qargs "$(QEMU_OPTS)" \
+ --bin "$< -b" --test $(MULTIARCH_SRC)/gdbstub/xpage-bp.py, \
+ breakpoint behind an established cross-page chain)
+
run-gdbstub-late-attach: late-attach
$(call run-test, $@, env LATE_ATTACH_PY=1 $(GDB_SCRIPT) \
--gdb $(GDB) \
@@ -159,7 +168,8 @@ EXTRA_RUNS += run-gdbstub-sha1 run-gdbstub-qxfer-auxv-read \
run-gdbstub-registers run-gdbstub-prot-none \
run-gdbstub-catch-syscalls run-gdbstub-follow-fork-mode-child \
run-gdbstub-follow-fork-mode-parent \
- run-gdbstub-qxfer-siginfo-read run-gdbstub-late-attach
+ run-gdbstub-qxfer-siginfo-read run-gdbstub-late-attach \
+ run-gdbstub-xpage-bp
# ARM Compatible Semi Hosting Tests
#
diff --git ./tests/tcg/multiarch/gdbstub/xpage-bp.py ./tests/tcg/multiarch/gdbstub/xpage-bp.py
new file mode 100644
index 0000000000..f40024f16d
--- /dev/null
+++ ./tests/tcg/multiarch/gdbstub/xpage-bp.py
@@ -0,0 +1,37 @@
+"""Test that a breakpoint set after a cross-page chain is established is hit.
+
+translator_use_goto_tb() lets a direct branch chain to another page in
+user-only builds, which is only safe because a run with no gdbstub can never
+acquire a breakpoint. This runs with one, so the chaining must be off and
+the breakpoint must still be reached.
+
+This runs as a sourced script (via -x, via run-test.py).
+
+SPDX-License-Identifier: GPL-2.0-or-later
+"""
+from test_gdbstub import main, report
+
+
+def run_test():
+ """Run through the tests one by one"""
+ gdb.Breakpoint("break_here")
+ gdb.execute("continue")
+
+ # The chain exists by now; put a breakpoint on the far side of it.
+ target = int(gdb.parse_and_eval("(unsigned long)page_b_entry"))
+ if target == 0:
+ report(True, "no code emitters for this architecture, skipped")
+ return
+ gdb.execute("break *{}".format(target))
+ gdb.execute("continue")
+
+ pc = int(gdb.parse_and_eval("(unsigned long)$pc"))
+ report(pc == target, "stopped at {:#x}, expected {:#x}".format(pc, target))
+
+ gdb.execute("delete")
+ gdb.execute("continue")
+ exitcode = int(gdb.parse_and_eval("$_exitcode"))
+ report(exitcode == 0, "{} == 0".format(exitcode))
+
+
+main(run_test)
diff --git ./tests/tcg/multiarch/test-xpage-chain.c ./tests/tcg/multiarch/test-xpage-chain.c
new file mode 100644
index 0000000000..a4e34149e7
--- /dev/null
+++ ./tests/tcg/multiarch/test-xpage-chain.c
@@ -0,0 +1,336 @@
+/*
+ * Cross-page TB chaining hazard test.
+ *
+ * Two adjacent pages of hand-written code. The last instruction of page A
+ * sets the return value and falls through into page B, which returns; a TB
+ * always ends at a page boundary, so page A reaches page B through a
+ * cross-page goto_tb.
+ *
+ * Phase 1: run it enough times that QEMU chains TB_A -> TB_B.
+ * Phase 2: mprotect page B away. Re-running must fault.
+ * Phase 3: map it back and write different code into it. Re-running must
+ * execute the NEW code, not a stale chained translation.
+ *
+ * With -b, phases 2 and 3 are replaced by a stop at break_here(), where the
+ * gdbstub test sets a breakpoint on page B -- after the chain exists -- and
+ * checks that re-running the chain still stops on it. See
+ * tests/tcg/multiarch/gdbstub/xpage-bp.py.
+ *
+ * The code the two pages hold is architecture specific, so each
+ * architecture supplies two emitters:
+ *
+ * emit_set_ret(p, val) - set the integer return value register to val
+ * emit_ret(p) - return to the caller
+ *
+ * both writing at @p and returning the number of bytes written. Neither
+ * may contain a branch: the fall-through from page A into page B is the
+ * whole point, and a delay slot must not straddle the boundary. An
+ * architecture that supplies neither skips the test.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+#include <stdbool.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <setjmp.h>
+#include <signal.h>
+#include <stdint.h>
+#include <sys/mman.h>
+#include <unistd.h>
+
+static inline size_t put32(void *p, uint32_t insn)
+{
+ memcpy(p, &insn, sizeof(insn));
+ return sizeof(insn);
+}
+
+static inline size_t put16(void *p, uint16_t insn)
+{
+ memcpy(p, &insn, sizeof(insn));
+ return sizeof(insn);
+}
+
+#if defined(__aarch64__)
+#define HAVE_EMITTERS
+/* movz w0, #val */
+static size_t emit_set_ret(void *p, int val)
+{
+ return put32(p, 0x52800000u | ((uint32_t)val << 5));
+}
+static size_t emit_ret(void *p)
+{
+ return put32(p, 0xd65f03c0u); /* ret */
+}
+#elif defined(__alpha__)
+#define HAVE_EMITTERS
+/* lda $0, val($31) */
+static size_t emit_set_ret(void *p, int val)
+{
+ return put32(p, 0x201f0000u | (uint16_t)val);
+}
+static size_t emit_ret(void *p)
+{
+ return put32(p, 0x6bfa8001u); /* ret */
+}
+#elif defined(__arm__)
+#define HAVE_EMITTERS
+/* mov r0, #val */
+static size_t emit_set_ret(void *p, int val)
+{
+ return put32(p, 0xe3a00000u | (uint8_t)val);
+}
+static size_t emit_ret(void *p)
+{
+ return put32(p, 0xe12fff1eu); /* bx lr */
+}
+#elif defined(__hppa__)
+#define HAVE_EMITTERS
+/* ldi val, %ret0 */
+static size_t emit_set_ret(void *p, int val)
+{
+ return put32(p, 0x341c0000u | ((uint32_t)val << 1));
+}
+static size_t emit_ret(void *p)
+{
+ size_t n = put32(p, 0xe840c000u); /* bv %r0(%rp) */
+ return n + put32((char *)p + n, 0x08000240u); /* nop (delay slot) */
+}
+#elif defined(__i386__) || defined(__x86_64__)
+#define HAVE_EMITTERS
+/* mov $val, %eax */
+static size_t emit_set_ret(void *p, int val)
+{
+ uint32_t imm = val;
+ *(unsigned char *)p = 0xb8;
+ return 1 + put32((char *)p + 1, imm);
+}
+static size_t emit_ret(void *p)
+{
+ *(unsigned char *)p = 0xc3; /* ret */
+ return 1;
+}
+#elif defined(__loongarch64)
+#define HAVE_EMITTERS
+/* ori $a0, $zero, val */
+static size_t emit_set_ret(void *p, int val)
+{
+ return put32(p, 0x03800004u | ((uint32_t)val << 10));
+}
+static size_t emit_ret(void *p)
+{
+ return put32(p, 0x4c000020u); /* jr $ra */
+}
+#elif defined(__m68k__)
+#define HAVE_EMITTERS
+/* moveq #val, %d0 */
+static size_t emit_set_ret(void *p, int val)
+{
+ return put16(p, 0x7000u | (uint8_t)val);
+}
+static size_t emit_ret(void *p)
+{
+ return put16(p, 0x4e75u); /* rts */
+}
+#elif defined(__mips__)
+#define HAVE_EMITTERS
+/* li $v0, val */
+static size_t emit_set_ret(void *p, int val)
+{
+ return put32(p, 0x24020000u | (uint16_t)val);
+}
+static size_t emit_ret(void *p)
+{
+ size_t n = put32(p, 0x03e00008u); /* jr $ra */
+ return n + put32((char *)p + n, 0x00000000u); /* nop (delay slot) */
+}
+/*
+ * ELFv1 function pointers are descriptors rather than code addresses, so
+ * there is nothing to call the raw code through.
+ */
+#elif defined(__powerpc__) && \
+ (!defined(__powerpc64__) || (defined(_CALL_ELF) && _CALL_ELF == 2))
+#define HAVE_EMITTERS
+/* li r3, val */
+static size_t emit_set_ret(void *p, int val)
+{
+ return put32(p, 0x38600000u | (uint16_t)val);
+}
+static size_t emit_ret(void *p)
+{
+ return put32(p, 0x4e800020u); /* blr */
+}
+#elif defined(__riscv)
+#define HAVE_EMITTERS
+/* addi a0, zero, val -- the 4 byte form, never c.li */
+static size_t emit_set_ret(void *p, int val)
+{
+ return put32(p, 0x00000513u | ((uint32_t)val << 20));
+}
+static size_t emit_ret(void *p)
+{
+ return put32(p, 0x00008067u); /* jalr zero, 0(ra) */
+}
+#elif defined(__s390x__)
+#define HAVE_EMITTERS
+/* lghi %r2, val */
+static size_t emit_set_ret(void *p, int val)
+{
+ size_t n = put16(p, 0xa729u);
+ return n + put16((char *)p + n, (uint16_t)val);
+}
+static size_t emit_ret(void *p)
+{
+ return put16(p, 0x07feu); /* br %r14 */
+}
+#elif defined(__sh__)
+#define HAVE_EMITTERS
+/* mov #val, r0 */
+static size_t emit_set_ret(void *p, int val)
+{
+ return put16(p, 0xe000u | (uint8_t)val);
+}
+static size_t emit_ret(void *p)
+{
+ size_t n = put16(p, 0x000bu); /* rts */
+ return n + put16((char *)p + n, 0x0009u); /* nop (delay slot) */
+}
+#elif defined(__sparc__)
+#define HAVE_EMITTERS
+/* mov val, %o0 */
+static size_t emit_set_ret(void *p, int val)
+{
+ return put32(p, 0x90102000u | (uint32_t)(val & 0x1fff));
+}
+static size_t emit_ret(void *p)
+{
+ size_t n = put32(p, 0x81c3e008u); /* retl */
+ return n + put32((char *)p + n, 0x01000000u); /* nop (delay slot) */
+}
+#endif
+
+/* Where the fall-through lands, for the gdbstub test to breakpoint on. */
+void *page_b_entry;
+
+/* Somewhere for the gdbstub test to stop once the chain is established. */
+void __attribute__((noinline)) break_here(void)
+{
+ asm volatile ("");
+}
+
+#ifdef HAVE_EMITTERS
+static sigjmp_buf jb;
+/*
+ * Written by the SIGSEGV handler and read by main(), so it must not be
+ * cached in a register across the faulting call.
+ */
+static volatile sig_atomic_t caught;
+
+static void segv(int sig)
+{
+ caught = 1;
+ siglongjmp(jb, 1);
+}
+#endif
+
+int main(int argc, char **argv)
+{
+ bool bp_mode = argc > 1 && strcmp(argv[1], "-b") == 0;
+#ifndef HAVE_EMITTERS
+ printf("SKIP: no code emitters for this architecture\n");
+ if (bp_mode) {
+ break_here();
+ }
+ return 0;
+#else
+ unsigned char tmp[16];
+ struct sigaction sa;
+ long (*fn)(void);
+ size_t setlen, n;
+ long ps = sysconf(_SC_PAGESIZE);
+ int rc = 0;
+ unsigned char *m = mmap(NULL, 2 * ps, PROT_READ | PROT_WRITE | PROT_EXEC,
+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ if (m == MAP_FAILED) {
+ perror("mmap");
+ return 2;
+ }
+
+ unsigned char *pb = m + ps;
+
+ /*
+ * Page A ends with the store to the return value register, so that the
+ * next instruction executed is the first one on page B.
+ */
+ setlen = emit_set_ret(tmp, 1);
+ memcpy(pb - setlen, tmp, setlen);
+ emit_ret(pb);
+ __builtin___clear_cache((char *)m, (char *)m + 2 * ps);
+
+ page_b_entry = pb;
+ fn = (long (*)(void))(pb - setlen);
+
+ for (int i = 0; i < 200000; i++) {
+ if (fn() != 1) {
+ printf("FAIL: phase 1 wrong result\n");
+ return 1;
+ }
+ }
+ printf("phase 1 ok (chained)\n");
+
+ if (bp_mode) {
+ /*
+ * The chain from page A to page B now exists. gdb puts a breakpoint
+ * on page_b_entry here; the call below has to stop on it rather than
+ * jump over it.
+ */
+ break_here();
+ if (fn() != 1) {
+ printf("FAIL: bp phase wrong result\n");
+ return 1;
+ }
+ printf("bp phase ok\n");
+ return 0;
+ }
+
+ memset(&sa, 0, sizeof(sa));
+ sa.sa_handler = segv;
+ sigemptyset(&sa.sa_mask);
+ if (sigaction(SIGSEGV, &sa, NULL) != 0) {
+ perror("sigaction");
+ return 2;
+ }
+ if (mprotect(pb, ps, PROT_NONE) != 0) {
+ perror("mprotect");
+ return 2;
+ }
+ if (sigsetjmp(jb, 1) == 0) {
+ fn();
+ printf("FAIL: phase 2 executed page B after mprotect(PROT_NONE)\n");
+ rc = 1;
+ } else if (!caught) {
+ printf("FAIL: phase 2 longjmp without entering the handler\n");
+ rc = 1;
+ } else {
+ printf("phase 2 ok (faulted)\n");
+ }
+
+ /* Phase 3: map back, overwrite, expect the new code to run. */
+ if (mprotect(pb, ps, PROT_READ | PROT_WRITE | PROT_EXEC) != 0) {
+ perror("mprotect back");
+ return 2;
+ }
+ n = emit_set_ret(pb, 2);
+ emit_ret(pb + n);
+ __builtin___clear_cache((char *)pb, (char *)pb + ps);
+
+ long r = fn();
+ if (r != 2) {
+ printf("FAIL: phase 3 returned %ld, expected 2 (stale chain)\n", r);
+ rc = 1;
+ } else {
+ printf("phase 3 ok (new code ran)\n");
+ }
+ return rc;
+#endif
+}
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH v5 8/9] RFC: accel/tcg: poison the jump cache instead of polling for indirect exits
2026-09-01 3:47 ` [PATCH v5 " Matt Turner
` (6 preceding siblings ...)
2026-09-01 3:48 ` [PATCH v5 7/9] RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds Matt Turner
@ 2026-09-01 3:48 ` Matt Turner
2026-09-01 3:48 ` [PATCH v5 9/9] RFC: tcg: fold a guest displacement into the host addressing mode Matt Turner
8 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-09-01 3:48 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
Every translation block begins by loading cpu->neg.icount_decr.u32, testing
it and branching to the exit path. That is three host instructions at the top
of every TB, and blocks are short: an emulated alpha gcc 16.2.0 compiling a
255k line translation unit executes 34.2 billion of them at 6.04 guest
instructions each.
A block does not need to poll if every way out of it already reaches a check.
A goto_tb does not: it chains straight into its destination, with nothing in
between that looks at icount_decr, so the destination has to poll on entry.
An indirect exit does. The out-of-line path calls helper_lookup_tb_ptr()
every time, so it only needs the helper to return the epilogue while an exit
is pending. The inline probe needs a way to be told, so give it one:
CPUState::tb_jmp_cache_probe, the base pointer it reads. Normally that is
cpu->tb_jmp_cache; pointed at a shared page of zeroes instead, every entry
the probe finds has a NULL tb, every dispatch misses, and a miss lands in the
same helper. The real jump cache is untouched, so no cache contents are lost,
and the fast path pays nothing: the base was a load from CPUState either way.
The two places that set icount_decr.u16.high poison the probe; the main loop
puts it back once cpu_handle_interrupt() has cleared the reason. The poison
is a single read-only mapping shared by every CPU, because nothing may ever
write to it and a stray store into a page every vCPU dispatches through is
worth trapping rather than debugging.
The poll is therefore emitted only in blocks that emit a goto_tb. Whether a
block does is not known until its last exit has been generated, so the
decision is deferred and the load and branch are emitted retroactively at the
head of the block in gen_tb_end(), using the same emit_before_op mechanism
the can_do_io stores use. icount opts out and keeps the counter
unconditionally.
Interrupt latency is bounded at one block, as before. It does not depend on
the shape of the guest's control flow graph: a block either polls on entry or
is checked on the way out, and no run of blocks can avoid both. What changes
is where the check sits, not how often one happens.
tests/tcg/multiarch/test-indirect-irq.c is added for this: a loop whose only
back edge is an indirect branch, under alarm(1). That loop's block emits no
goto_tb, so it no longer polls, and the test passes only because the dispatch
notices instead -- it hangs if the poison is removed, which is what makes it a
test of the new mechanism rather than of the old poll. Nothing in it is
architecture specific: the loop is a computed goto, which every target's
compiler supports, so it covers whichever targets go on to use the inline
probe. The other alpha tests still pass and the emulated compiler still
produces byte-identical output.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the
SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, LTO build, on
top of the preceding patches:
before: 891,254,240,071 instructions
after: 868,811,832,620 instructions -2.52%
before: 81.45s wall clock
after: 79.85s wall clock -1.96%
The emulated compiler produces byte-identical output.
RFC because:
- The un-poison in the main loop races a concurrent poison from another
thread. The existing barrier around icount_decr.u16.high covers it -- a
poison that lands after the sync also re-set the flag, and exit_request was
stored before it -- but this deserves more eyes than the single-threaded
user-mode testing I have given it.
- Only the inline probe needs the poison, and only alpha uses the inline
probe today. Targets on the out-of-line path are covered by the helper
check alone, but that has not been measured.
v3: Rebased onto the removal of "only poll for interrupts in blocks that can
close a cycle", which v2 sat on top of and which is dropped: it let a
straight-line run of arbitrary length go unchecked, since a block with no
backward edge polled nowhere (Richard).
The rule is now that a block polls iff it emits a goto_tb, rather than
iff it can close a control flow cycle. That keeps the bound at one block
without any analysis of the guest's control flow graph, so the objection
to the dropped patch does not carry over. The deferred-emission machinery
it needs moves here from that patch; DisasContextBase::needs_exit_check
and the hook in translator_use_goto_tb() are gone with it, and the flag
is now set by tcg_gen_goto_tb() rather than by goto_ptr emission.
All of v2's measurements were dropped: they were taken with the
cycle-analysis patch underneath, which changes both the baseline and
what is left to remove, so none of them described this patch. The
numbers above are a fresh measurement of the series as it now stands.
v4: Moved the test from tests/tcg/alpha/ to tests/tcg/multiarch/: the
mechanism is generic and nothing in the test is alpha specific (Alex).
The performance numbers above are the v3 measurements, not re-run: the
machine they were taken on is busy.
v5: CPUState::tb_jmp_cache_probe moves here from what was patch 5, which
used it for breakpoints too. Breakpoints are now a cflag, so a pending
exit is the only reason left to poison, and the machinery shrinks to
match: no NULL states to handle, no cross-thread poison from
cpu_breakpoint_insert(), and one condition rather than two.
v5: Map the poison read-only rather than leaving it a writable .bss object.
Requested by Richard Henderson. It costs a page-aligned 1MB allocation
at startup instead of nothing on disk, which the enforcement is worth.
qemu_mprotect_ro() is added for it, alongside the _rw, _rwx and _none
forms already there.
v5: Drop the NULL checks in the poison and sync helpers. Requested by
Richard Henderson: the sync is only ever called by the main loop, so it
cannot see an unrealized CPU, and unrealize now leaves the probe pointing
at the poison rather than at NULL, so neither has an unrealized state to
consider.
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
accel/tcg/cpu-exec.c | 91 +++++++++++++++++++++++++
accel/tcg/internal-common.h | 9 +++
accel/tcg/tcg-accel-ops.c | 2 +
accel/tcg/translator.c | 51 +++++++++++++-
include/hw/core/cpu.h | 9 +++
include/qemu/mprotect.h | 1 +
include/tcg/tcg.h | 2 +
tcg/tcg-op.c | 21 +++++-
tests/tcg/multiarch/test-indirect-irq.c | 62 +++++++++++++++++
util/osdep.c | 9 +++
10 files changed, 253 insertions(+), 4 deletions(-)
create mode 100644 tests/tcg/multiarch/test-indirect-irq.c
diff --git ./accel/tcg/cpu-exec.c ./accel/tcg/cpu-exec.c
index ca90a77a7b..7a371e6928 100644
--- ./accel/tcg/cpu-exec.c
+++ ./accel/tcg/cpu-exec.c
@@ -19,6 +19,9 @@
#include "qemu/osdep.h"
#include "qemu/qemu-print.h"
+#include "qemu/error-report.h"
+#include "qemu/memalign.h"
+#include "qemu/mprotect.h"
#include "qapi/error.h"
#include "qapi/type-helpers.h"
#include "hw/core/cpu.h"
@@ -388,6 +391,16 @@ const void *HELPER(lookup_tb_ptr)(CPUArchState *env)
*/
cpu->neg.can_do_io = true;
+ /*
+ * A block that dispatches indirectly does not emit the icount_decr poll,
+ * so this is where a pending exit is noticed for that path: either the
+ * probe was poisoned and every dispatch arrives here, or the target uses
+ * the out-of-line lookup and always did.
+ */
+ if (unlikely(cpu_loop_exit_requested(cpu))) {
+ return tcg_code_gen_epilogue;
+ }
+
TCGTBCPUState s = cpu->cc->tcg_ops->get_tb_cpu_state(cpu);
s.cflags = curr_cflags(cpu);
@@ -779,6 +792,70 @@ static inline bool cpu_handle_exception(CPUState *cpu, int *ret)
return false;
}
+/*
+ * The inline jump cache probe reads cpu->tb_jmp_cache_probe and takes the
+ * slow path when the entry it finds has a NULL tb. Pointing the probe at a
+ * region that is all zeroes therefore forces every indirect dispatch into
+ * helper_lookup_tb_ptr(), which returns the epilogue while an exit is
+ * pending. The real jump cache is untouched, so no contents are lost and
+ * recovery is a single store.
+ *
+ * Only ever read from, and only one entry per dispatch, so one shared
+ * zero-filled cache is enough for every CPU. Mapped read-only, since
+ * nothing may write to it and a stray store into a shared page every vCPU
+ * dispatches through is worth trapping rather than debugging.
+ */
+static CPUJumpCache *tb_jmp_cache_poison;
+
+static void tb_jmp_cache_poison_init(void)
+{
+ size_t align = qemu_real_host_page_size();
+ size_t size = ROUND_UP(sizeof(CPUJumpCache), align);
+ void *p = qemu_memalign(align, size);
+
+ memset(p, 0, size);
+ if (qemu_mprotect_ro(p, size) < 0) {
+ /* Only the enforcement is lost; the zeroes are what matter. */
+ warn_report("could not write-protect the jump cache poison");
+ }
+ tb_jmp_cache_poison = p;
+}
+
+/*
+ * Poison @cpu's probe, from any thread. A plain store is enough: the value
+ * only ever costs a slow path that is correct on its own, and the generated
+ * code re-reads the base on every dispatch.
+ */
+void tcg_cpu_poison_jmp_cache(CPUState *cpu)
+{
+ qatomic_set(&cpu->tb_jmp_cache_probe, tb_jmp_cache_poison);
+}
+
+/*
+ * Called from @cpu's own main loop, which is the only context that can
+ * establish that no reason to be poisoned is left.
+ */
+void tcg_cpu_sync_jmp_cache(CPUState *cpu)
+{
+ if (qatomic_read(&cpu->tb_jmp_cache_probe) == cpu->tb_jmp_cache) {
+ return;
+ }
+
+ qatomic_set(&cpu->tb_jmp_cache_probe, cpu->tb_jmp_cache);
+
+ /*
+ * Another thread may have set icount_decr.u16.high after the caller
+ * decided no exit was pending, and its poison may have landed before
+ * the store above. Order that store against the re-read, so the race
+ * is lost in the safe direction: an exit that is still pending here
+ * poisons again, and the dispatch after it returns to the main loop.
+ */
+ smp_mb();
+ if (unlikely(cpu_loop_exit_requested(cpu))) {
+ tcg_cpu_poison_jmp_cache(cpu);
+ }
+}
+
void tcg_kick_vcpu_thread(CPUState *cpu)
{
/*
@@ -791,6 +868,9 @@ void tcg_kick_vcpu_thread(CPUState *cpu)
/* Ensure cpu_exec will see the exit request after TCG has exited. */
qatomic_store_release(&cpu->neg.icount_decr.u16.high, -1);
+
+ /* Blocks that only dispatch indirectly do not poll; stop them chaining. */
+ tcg_cpu_poison_jmp_cache(cpu);
}
static inline bool icount_exit_request(CPUState *cpu)
@@ -991,6 +1071,13 @@ cpu_exec_loop(CPUState *cpu, SyncClocks *sc)
break;
}
+ /*
+ * cpu_handle_interrupt() has just cleared everything that would
+ * make a dispatch have to come back here, so this is where the
+ * probe is allowed to return after a poison.
+ */
+ tcg_cpu_sync_jmp_cache(cpu);
+
tb = tb_lookup(cpu, s);
if (tb == NULL) {
CPUJumpCache *jc;
@@ -1092,6 +1179,7 @@ bool tcg_exec_realizefn(CPUState *cpu, Error **errp)
assert(tcg_ops->get_tb_cpu_state);
assert(tcg_ops->mmu_index);
tcg_ops->initialize();
+ tb_jmp_cache_poison_init();
tcg_target_initialized = true;
}
@@ -1099,6 +1187,7 @@ bool tcg_exec_realizefn(CPUState *cpu, Error **errp)
tcg_update_cflags(cpu);
cpu->tb_jmp_cache = g_new0(CPUJumpCache, 1);
+ qatomic_set(&cpu->tb_jmp_cache_probe, cpu->tb_jmp_cache);
tlb_init(cpu);
#ifndef CONFIG_USER_ONLY
tcg_iommu_init_notifier_list(cpu);
@@ -1116,5 +1205,7 @@ void tcg_exec_unrealizefn(CPUState *cpu)
#endif /* !CONFIG_USER_ONLY */
tlb_destroy(cpu);
+ /* Not NULL: nothing then has to special-case an unrealized CPU. */
+ tcg_cpu_poison_jmp_cache(cpu);
g_free_rcu(cpu->tb_jmp_cache, rcu);
}
diff --git ./accel/tcg/internal-common.h ./accel/tcg/internal-common.h
index 853d1b51ee..6faa039850 100644
--- ./accel/tcg/internal-common.h
+++ ./accel/tcg/internal-common.h
@@ -144,6 +144,15 @@ void page_table_config_init(void);
G_NORETURN void cpu_io_recompile(CPUState *cpu, uintptr_t retaddr);
#endif /* CONFIG_USER_ONLY */
+/*
+ * Force @cpu's generated code back into helper_lookup_tb_ptr(), which
+ * re-checks everything the inline jump cache probe cannot. Safe to call
+ * from any thread. tcg_cpu_sync_jmp_cache() undoes it, and is for the
+ * owning CPU's main loop only.
+ */
+void tcg_cpu_poison_jmp_cache(CPUState *cpu);
+void tcg_cpu_sync_jmp_cache(CPUState *cpu);
+
void tb_phys_invalidate(TranslationBlock *tb, tb_page_addr_t page_addr);
void tb_set_jmp_target(TranslationBlock *tb, int n, uintptr_t addr);
diff --git ./accel/tcg/tcg-accel-ops.c ./accel/tcg/tcg-accel-ops.c
index 560fe2554b..63a15f1689 100644
--- ./accel/tcg/tcg-accel-ops.c
+++ ./accel/tcg/tcg-accel-ops.c
@@ -38,6 +38,7 @@
#include "exec/cputlb.h"
#include "exec/hwaddr.h"
#include "exec/tb-flush.h"
+#include "internal-common.h"
#include "exec/translation-block.h"
#include "exec/watchpoint.h"
#include "gdbstub/enums.h"
@@ -106,6 +107,7 @@ void tcg_handle_interrupt(CPUState *cpu, int mask)
qemu_cpu_kick(cpu);
} else {
qatomic_set(&cpu->neg.icount_decr.u16.high, -1);
+ tcg_cpu_poison_jmp_cache(cpu);
}
}
diff --git ./accel/tcg/translator.c ./accel/tcg/translator.c
index 8879cd626f..89d255bd04 100644
--- ./accel/tcg/translator.c
+++ ./accel/tcg/translator.c
@@ -45,12 +45,35 @@ bool translator_io_start(DisasContextBase *db)
return true;
}
+/*
+ * A block that ends in a goto_tb chains straight to its destination: nothing
+ * between the two looks at icount_decr, so the destination has to poll on
+ * entry. A block whose exits are all indirect does not, because the dispatch
+ * itself notices -- a pending exit poisons tb_jmp_cache_probe, so the probe
+ * misses into helper_lookup_tb_ptr(), which returns the epilogue. Every block
+ * therefore either polls on entry or is checked as it leaves, which bounds
+ * interrupt latency at one block without looking at the shape of the guest's
+ * control flow graph.
+ *
+ * Which kind a block is is not known until its last exit has been emitted, so
+ * defer the decision to gen_tb_end() and emit the poll retroactively.
+ *
+ * icount needs the counter unconditionally, so it opts out.
+ */
+static bool defer_exit_check(uint32_t cflags)
+{
+ return !(cflags & CF_USE_ICOUNT);
+}
+
static TCGOp *gen_tb_start(DisasContextBase *db, uint32_t cflags)
{
TCGv_i32 count = NULL;
TCGOp *icount_start_insn = NULL;
- if ((cflags & CF_USE_ICOUNT) || !(cflags & CF_NOIRQ)) {
+ tcg_ctx->exit_check_needed = false;
+
+ if ((cflags & CF_USE_ICOUNT) ||
+ (!(cflags & CF_NOIRQ) && !defer_exit_check(cflags))) {
count = tcg_temp_new_i32();
tcg_gen_ld_i32(count, tcg_env,
offsetof(CPUState, neg.icount_decr.u32) -
@@ -76,6 +99,9 @@ static TCGOp *gen_tb_start(DisasContextBase *db, uint32_t cflags)
*/
if (cflags & CF_NOIRQ) {
tcg_ctx->exitreq_label = NULL;
+ } else if (defer_exit_check(cflags)) {
+ /* Emitted retroactively by gen_tb_end(), if this TB emits a goto_tb. */
+ tcg_ctx->exitreq_label = gen_new_label();
} else {
tcg_ctx->exitreq_label = gen_new_label();
tcg_gen_brcondi_i32(TCG_COND_LT, count, 0, tcg_ctx->exitreq_label);
@@ -91,7 +117,8 @@ static TCGOp *gen_tb_start(DisasContextBase *db, uint32_t cflags)
}
static void gen_tb_end(const TranslationBlock *tb, uint32_t cflags,
- TCGOp *icount_start_insn, int num_insns)
+ TCGOp *icount_start_insn, int num_insns,
+ TCGOp *first_insn_start)
{
if (cflags & CF_USE_ICOUNT) {
/*
@@ -102,6 +129,23 @@ static void gen_tb_end(const TranslationBlock *tb, uint32_t cflags,
tcgv_i32_arg(tcg_constant_i32(num_insns)));
}
+ if (tcg_ctx->exitreq_label && defer_exit_check(cflags) &&
+ !(cflags & CF_NOIRQ)) {
+ if (tcg_ctx->exit_check_needed) {
+ TCGv_i32 count = tcg_temp_new_i32();
+ TCGOp *save = tcg_ctx->emit_before_op;
+
+ tcg_ctx->emit_before_op = first_insn_start;
+ tcg_gen_ld_i32(count, tcg_env,
+ offsetof(CPUState, neg.icount_decr.u32) -
+ sizeof(CPUState));
+ tcg_gen_brcondi_i32(TCG_COND_LT, count, 0, tcg_ctx->exitreq_label);
+ tcg_ctx->emit_before_op = save;
+ } else {
+ tcg_ctx->exitreq_label = NULL;
+ }
+ }
+
if (tcg_ctx->exitreq_label) {
gen_set_label(tcg_ctx->exitreq_label);
tcg_gen_exit_tb(tb, TB_EXIT_REQUESTED);
@@ -238,7 +282,8 @@ void translator_loop(CPUState *cpu, TranslationBlock *tb, int *max_insns,
/* Emit code to exit the TB, as indicated by db->is_jmp. */
ops->tb_stop(db, cpu);
- gen_tb_end(tb, cflags, icount_start_insn, db->num_insns);
+ gen_tb_end(tb, cflags, icount_start_insn, db->num_insns,
+ first_insn_start);
/*
* Manage can_do_io for the translation block: set to false before
diff --git ./include/hw/core/cpu.h ./include/hw/core/cpu.h
index 81af7b9ee1..c8669f2cad 100644
--- ./include/hw/core/cpu.h
+++ ./include/hw/core/cpu.h
@@ -519,6 +519,15 @@ struct CPUState {
MemoryRegion *memory;
struct CPUJumpCache *tb_jmp_cache;
+ /*
+ * @tb_jmp_cache_probe: the base the inline jump cache probe reads.
+ *
+ * Normally @tb_jmp_cache. Pointed at a shared read-only page of zeroes
+ * while an exit is pending, so that every inline dispatch misses and
+ * falls back to helper_lookup_tb_ptr(), which returns to the main loop.
+ * Only generated code and the accessors in cpu-exec.c may touch it.
+ */
+ struct CPUJumpCache *tb_jmp_cache_probe;
GArray *gdb_regs;
int gdb_num_regs;
diff --git ./include/qemu/mprotect.h ./include/qemu/mprotect.h
index 1e83d1433e..4fc13d79f6 100644
--- ./include/qemu/mprotect.h
+++ ./include/qemu/mprotect.h
@@ -8,6 +8,7 @@
#define QEMU_MPROTECT_H
int qemu_mprotect_rw(void *addr, size_t size);
+int qemu_mprotect_ro(void *addr, size_t size);
int qemu_mprotect_rwx(void *addr, size_t size);
int qemu_mprotect_none(void *addr, size_t size);
diff --git ./include/tcg/tcg.h ./include/tcg/tcg.h
index 7669dc1c2d..df08c10544 100644
--- ./include/tcg/tcg.h
+++ ./include/tcg/tcg.h
@@ -389,6 +389,8 @@ struct TCGContext {
struct TCGLabelPoolData *pool_labels;
TCGLabel *exitreq_label;
+ /* Set by goto_tb emission: this TB chains without reaching a check. */
+ bool exit_check_needed;
#ifdef CONFIG_PLUGIN
/*
diff --git ./tcg/tcg-op.c ./tcg/tcg-op.c
index b10b2d66d5..1c5c5ec1d3 100644
--- ./tcg/tcg-op.c
+++ ./tcg/tcg-op.c
@@ -2713,6 +2713,13 @@ void tcg_gen_goto_tb(unsigned idx)
tcg_debug_assert((tcg_ctx->goto_tb_issue_mask & (1 << idx)) == 0);
tcg_ctx->goto_tb_issue_mask |= 1 << idx;
#endif
+ /*
+ * A goto_tb chains straight into the destination, with nothing in between
+ * that looks at icount_decr, so this TB has to poll on entry. See
+ * defer_exit_check().
+ */
+ tcg_ctx->exit_check_needed = true;
+
plugin_gen_disable_mem_helpers();
tcg_gen_op1i(INDEX_op_goto_tb, 0, idx);
}
@@ -2790,8 +2797,13 @@ static void gen_jmp_cache_probe(TCGv_i64 pc, const TranslationBlock *tb)
gen_jmp_cache_hash(h, pc);
tcg_gen_shli_i64(h, h, 4);
+ /*
+ * Not cpu->tb_jmp_cache: the probe reads its own base so that the main
+ * loop can poison it, which is how a pending exit forces every dispatch
+ * back into the helper. See tcg_cpu_sync_jmp_cache().
+ */
tcg_gen_ld_ptr(jc, tcg_env,
- offsetof(CPUState, tb_jmp_cache) - sizeof(CPUState));
+ offsetof(CPUState, tb_jmp_cache_probe) - sizeof(CPUState));
tcg_gen_trunc_i64_ptr(ent, h);
tcg_gen_add_ptr(ent, jc, ent);
@@ -2849,6 +2861,13 @@ static void gen_goto_jc(TCGv_i64 pc)
plugin_gen_disable_mem_helpers();
+ /*
+ * Neither path below needs an icount_decr poll. The helper returns to
+ * the main loop while an exit is pending, and a pending exit poisons
+ * tb_jmp_cache_probe, so the inline probe finds a NULL tb and falls into
+ * that same helper.
+ */
+
#ifdef CONFIG_DEBUG_TCG
/*
* The caller has asserted that env already describes the destination.
diff --git ./tests/tcg/multiarch/test-indirect-irq.c ./tests/tcg/multiarch/test-indirect-irq.c
new file mode 100644
index 0000000000..a672faf641
--- /dev/null
+++ ./tests/tcg/multiarch/test-indirect-irq.c
@@ -0,0 +1,62 @@
+/*
+ * A loop whose only back edge is an indirect branch must still be
+ * interruptible.
+ *
+ * Blocks that dispatch indirectly do not emit the icount_decr poll; a pending
+ * exit instead poisons the inline jump cache probe so that the dispatch falls
+ * into helper_lookup_tb_ptr(), which returns to the main loop. If that
+ * mechanism breaks, this program never leaves the loop and the test times
+ * out rather than failing an assertion.
+ *
+ * A computed goto is used deliberately: a plain while(1) would end the block
+ * with a direct backward branch, that is a goto_tb, and a block that emits a
+ * goto_tb still polls -- so it would not exercise the path under test.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+#include <assert.h>
+#include <signal.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+/* Written by the handler, read by the loop, so it must not be cached. */
+static volatile sig_atomic_t fired;
+/* Read after the loop, so the loop must not optimize the increment away. */
+static volatile unsigned long iterations;
+
+static void handler(int sig)
+{
+ fired = 1;
+}
+
+int main(void)
+{
+ /*
+ * Indexing a table with a volatile index, rather than jumping through a
+ * volatile pointer: gcc happily proves a single-valued pointer constant
+ * and emits a direct branch, which is the case this test is not about.
+ */
+ volatile int idx = 0;
+ void *target[2];
+ struct sigaction sa;
+
+ memset(&sa, 0, sizeof(sa));
+ sa.sa_handler = handler;
+ sigemptyset(&sa.sa_mask);
+ assert(sigaction(SIGALRM, &sa, NULL) == 0);
+ alarm(1);
+
+ target[0] = &&spin;
+ target[1] = &&out;
+spin:
+ iterations++;
+ if (!fired) {
+ goto *target[idx];
+ }
+out:
+
+ printf("interrupted after %lu iterations\n", iterations);
+ return 0;
+}
diff --git ./util/osdep.c ./util/osdep.c
index 4a8b8b5a90..9c72a42b4f 100644
--- ./util/osdep.c
+++ ./util/osdep.c
@@ -99,6 +99,15 @@ int qemu_mprotect_rw(void *addr, size_t size)
#endif
}
+int qemu_mprotect_ro(void *addr, size_t size)
+{
+#ifdef _WIN32
+ return qemu_mprotect__osdep(addr, size, PAGE_READONLY);
+#else
+ return qemu_mprotect__osdep(addr, size, PROT_READ);
+#endif
+}
+
int qemu_mprotect_rwx(void *addr, size_t size)
{
#ifdef _WIN32
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH v5 9/9] RFC: tcg: fold a guest displacement into the host addressing mode
2026-09-01 3:47 ` [PATCH v5 " Matt Turner
` (7 preceding siblings ...)
2026-09-01 3:48 ` [PATCH v5 8/9] RFC: accel/tcg: poison the jump cache instead of polling for indirect exits Matt Turner
@ 2026-09-01 3:48 ` Matt Turner
8 siblings, 0 replies; 47+ messages in thread
From: Matt Turner @ 2026-09-01 3:48 UTC (permalink / raw)
To: qemu-devel
Cc: richard.henderson, pbonzini, philmd, alex.bennee, zhao1.liu,
Matt Turner
Nothing in the TCG frontend interface can express a based memory access.
tcg_gen_qemu_ld/st take an address and nothing else, so a target with a
displacement in its load and store encodings -- which is most of them --
has to materialize the address first:
ldq a1,8(a0) -> mov 0x80(%rbp),%rbx reload a0
lea 0x8(%rbx),%r12 address
mov (%r12),%r12 the load
mov %r12,0x88(%rbp) spill a1
The lea is pure loss on a host whose addressing mode has a displacement
field sitting empty. It also needs a register, at the point in a block
where pressure is highest.
Fold it. After optimization, look for an add of a constant immediately
before a guest access, defining that access's address operand, and move the
constant into a new second constant argument on the op. The add is left for
liveness to remove, so nothing breaks if its result has another use. Only
the immediately preceding op is examined: that is what the frontends emit,
and a window of one op means the pass does not have to reason about what
could have happened in between. The one thing it does check is that the add
did not clobber the base it read, since the access now reads that base
directly.
Targets opt in with TCG_TARGET_HAS_ldst_disp and an out_disp member on
TCGOutOpQemuLdSt. Without it the pass does not run, the displacement stays
zero and the existing out member is called exactly as before, so no other
backend changes behavior or needs touching.
The fold is refused unless the access has no slow path at all, since the
slow path hands addr_reg to the helper and that register no longer holds
the full guest address. That is decided generically: user-only, because
softmmu compares the unadjusted address against the TLB; a 64-bit address
type, because a 32-bit one wraps where a host displacement would not; and
no alignment test on the access. For x86_64 the displacement goes in the
disp32 that prepare_host_addr() already fills in for guest_base, so all the
backend has left to check is that guest_base plus the displacement still
fits there.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the
SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, LTO build,
on top of the preceding patches, against a control measured in the same
session:
before: 868,811,832,620 instructions, 79.85s
after: 819,262,147,022 instructions, 77.30s
-5.70% instructions, -3.20% wall
Emitted code shrinks from 50.55MB to 48.80MB over the run, 167.4 to 161.6
bytes per block. Per Alpha opcode, the host bytes emitted for an access
fall as expected and nothing else moves:
ldq 18.3 -> 15.4 ldah 20.9 -> 20.9
ldl 16.6 -> 14.1 lda 12.9 -> 12.9
stq 12.8 -> 9.7 mov 9.8 -> 9.8
The emulated compiler produces byte-identical output and the alpha tests
still pass, including with a non-zero guest_base forced via -B.
RFC because:
- Only wired up for x86_64, and only for qemu_ld and qemu_st; the i128
qemu_ld2 and qemu_st2 pairs are left alone.
- Requiring that no slow path exists is stricter than necessary. The fast
path test can stay on the base register as long as the displacement is
itself a multiple of the required alignment, which it is for anything a
frontend emits for a struct or stack access. Recording the displacement
in TCGLabelQemuLdst and emitting one lea on the slow path would then
cover alignment-checked accesses too, at no fast path cost.
- Softmmu wants the displacement folded into the TLB comparison as well,
which is a bigger change than this one.
- A one op window catches everything the frontends emit today but is
trivially defeated by anything scheduled in between.
v4:
- Hoisted the compilation mode tests -- tcg_use_softmmu and the 64-bit
address type -- out of the backend hook and into fold_ldst_disp(), next
to the TCG_TARGET_HAS_ldst_disp test, so the loop is not entered at all
when the mode rules the fold out.
- Pass MemOp rather than MemOpIdx to the backend hook; nothing about the
mmu_idx is relevant to it.
- Moved the alignment test into generic code as ldst_disp_needs_align(),
so a backend does not have to repeat the atom_and_align_for_opc() call.
The exact answer depends on the host's atomicity capabilities, which the
generic pass does not know, so it answers for the most restrictive host.
That is the same answer for everything the frontends actually emit --
MO_ATOM_IFALIGN is the default -- and conservative for the handful of
MO_ATOM_WITHIN16 and MO_ATOM_SUBALIGN accesses, which lose the fold on a
host that could have taken it.
- What is left of the x86_64 hook is the guest_base test, so it now lives
beside x86_guest_base under the CONFIG_USER_ONLY that declares it.
- Refuse a displacement that does not fit in an int32_t, which is what
out_disp() takes. Not reachable with any real guest_base, but the pass
should not offer the backend something the interface cannot carry.
- The numbers above are unchanged from v3: they have not been re-measured
on the restructured patch, which is not expected to move them since the
accesses in this workload are all MO_ATOM_IFALIGN.
Signed-off-by: Matt Turner <mattst88@gmail.com>
---
include/tcg/tcg-opc.h | 9 ++-
tcg/tcg-op-ldst.c | 3 +-
tcg/tcg.c | 132 +++++++++++++++++++++++++++++++++++-
tcg/x86_64/tcg-target.c.inc | 41 +++++++++++
tcg/x86_64/tcg-target.h | 3 +
5 files changed, 184 insertions(+), 4 deletions(-)
diff --git ./include/tcg/tcg-opc.h ./include/tcg/tcg-opc.h
index f3a81d5d7f..92fd34d3e3 100644
--- ./include/tcg/tcg-opc.h
+++ ./include/tcg/tcg-opc.h
@@ -125,8 +125,13 @@ DEF(goto_ptr, 0, 1, 0, TCG_OPF_BB_EXIT | TCG_OPF_BB_END)
DEF(plugin_cb, 0, 0, 1, TCG_OPF_NOT_PRESENT)
DEF(plugin_mem_cb, 0, 1, 1, TCG_OPF_NOT_PRESENT)
-DEF(qemu_ld, 1, 1, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT)
-DEF(qemu_st, 0, 2, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT)
+/*
+ * The second constant argument is a displacement to add to the address,
+ * zero unless a target advertises TCG_TARGET_HAS_ldst_disp and the fold in
+ * fold_ldst_disp() applied.
+ */
+DEF(qemu_ld, 1, 1, 2, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT)
+DEF(qemu_st, 0, 2, 2, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT)
DEF(qemu_ld2, 2, 1, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT)
DEF(qemu_st2, 0, 3, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT)
diff --git ./tcg/tcg-op-ldst.c ./tcg/tcg-op-ldst.c
index 22211ccb45..ffc5e651a6 100644
--- ./tcg/tcg-op-ldst.c
+++ ./tcg/tcg-op-ldst.c
@@ -92,7 +92,8 @@ static MemOp tcg_canonicalize_memop(MemOp op, bool is64, bool st)
static void gen_ldst1(TCGOpcode opc, TCGType type, TCGTemp *v,
TCGTemp *addr, MemOpIdx oi)
{
- TCGOp *op = tcg_gen_op3(opc, type, temp_arg(v), temp_arg(addr), oi);
+ /* The trailing zero is the address displacement; see fold_ldst_disp(). */
+ TCGOp *op = tcg_gen_op4(opc, type, temp_arg(v), temp_arg(addr), oi, 0);
TCGOP_FLAGS(op) = get_memop(oi) & MO_SIZE;
}
diff --git ./tcg/tcg.c ./tcg/tcg.c
index 489df0e738..466604eb97 100644
--- ./tcg/tcg.c
+++ ./tcg/tcg.c
@@ -1058,6 +1058,13 @@ typedef struct TCGOutOpQemuLdSt {
TCGOutOp base;
void (*out)(TCGContext *s, TCGType type, TCGReg dest,
TCGReg addr, MemOpIdx oi);
+ /*
+ * As out(), for an access at addr + disp. Only required of targets that
+ * define TCG_TARGET_HAS_ldst_disp; for everyone else fold_ldst_disp()
+ * never runs and the displacement is always zero.
+ */
+ void (*out_disp)(TCGContext *s, TCGType type, TCGReg dest,
+ TCGReg addr, MemOpIdx oi, int32_t disp);
} TCGOutOpQemuLdSt;
typedef struct TCGOutOpQemuLdSt2 {
@@ -3574,6 +3581,123 @@ static void move_label_uses(TCGLabel *to, TCGLabel *from)
QSIMPLEQ_CONCAT(&to->branches, &from->branches);
}
+#ifndef TCG_TARGET_HAS_ldst_disp
+#define TCG_TARGET_HAS_ldst_disp 0
+#define tcg_target_ldst_disp_ok(s, opc, disp) false
+#endif
+
+/*
+ * Return true if @opc needs an alignment test in the fast path.
+ *
+ * atom_and_align_for_opc() gives the exact answer, but only once the host's
+ * atomicity capabilities are known, and those belong to the backend. Answer
+ * instead for the most restrictive host, which is valid for all of them.
+ */
+static bool ldst_disp_needs_align(MemOp opc)
+{
+ MemOp size = opc & MO_SIZE;
+
+ if (memop_alignment_bits(opc)) {
+ return true;
+ }
+ switch (opc & MO_ATOM_MASK) {
+ case MO_ATOM_NONE:
+ case MO_ATOM_IFALIGN:
+ case MO_ATOM_IFALIGN_PAIR:
+ return false;
+ case MO_ATOM_WITHIN16:
+ /* Misalignment implies !within16, and therefore no atomicity. */
+ return size != MO_128;
+ case MO_ATOM_WITHIN16_PAIR:
+ case MO_ATOM_SUBALIGN:
+ return size != MO_8;
+ default:
+ g_assert_not_reached();
+ }
+}
+
+/*
+ * Fold "add addr, base, $disp" into the guest access that follows it, so
+ * that the displacement becomes part of the host addressing mode instead of
+ * a separate instruction. Frontends have no way to express this: there is
+ * no displacement operand on tcg_gen_qemu_ld/st, so a based access always
+ * costs an extra add, and an extra register to hold its result.
+ *
+ * Only an add in the op immediately before the access is recognized. That
+ * is what the frontends emit, and a window of one op means no analysis is
+ * needed of what might have happened in between. The add is left in place;
+ * liveness removes it if its result has no other use.
+ */
+static void __attribute__((noinline))
+fold_ldst_disp(TCGContext *s)
+{
+ TCGOp *op;
+
+ /*
+ * The fold requires that the access have no slow path, because the slow
+ * path hands the address operand to the helper and that register no
+ * longer holds the complete guest address. That means user-only, since
+ * softmmu compares the unadjusted address against the TLB. It also
+ * requires a 64-bit address type: for a 32-bit one the add wraps and a
+ * host displacement would not.
+ */
+ if (!TCG_TARGET_HAS_ldst_disp || tcg_use_softmmu ||
+ s->addr_type != TCG_TYPE_I64) {
+ return;
+ }
+
+ QTAILQ_FOREACH(op, &s->ops, link) {
+ TCGOp *prev;
+ TCGTemp *cts;
+ int64_t disp;
+ MemOp opc;
+
+ switch (op->opc) {
+ case INDEX_op_qemu_ld:
+ case INDEX_op_qemu_st:
+ break;
+ default:
+ continue;
+ }
+
+ opc = get_memop(op->args[2]);
+ if (ldst_disp_needs_align(opc)) {
+ continue;
+ }
+
+ prev = QTAILQ_PREV(op, link);
+ if (prev == NULL || prev->opc != INDEX_op_add ||
+ TCGOP_TYPE(prev) != s->addr_type) {
+ continue;
+ }
+
+ /*
+ * The add must define the address operand, and must not have
+ * clobbered the base it read: after the fold the access reads the
+ * base directly, so the base has to still hold its original value.
+ */
+ if (prev->args[0] != op->args[1] || prev->args[0] == prev->args[1]) {
+ continue;
+ }
+
+ cts = arg_temp(prev->args[2]);
+ if (cts->kind != TEMP_CONST) {
+ continue;
+ }
+ /* out_disp() takes an int32_t, so anything wider cannot be passed. */
+ disp = cts->val;
+ if (disp != (int32_t)disp) {
+ continue;
+ }
+ if (disp == 0 || !tcg_target_ldst_disp_ok(s, opc, disp)) {
+ continue;
+ }
+
+ op->args[1] = prev->args[1];
+ op->args[3] = disp;
+ }
+}
+
/* Reachable analysis : remove unreachable code. */
static void __attribute__((noinline))
reachable_code_pass(TCGContext *s)
@@ -5728,7 +5852,12 @@ static void tcg_reg_alloc_op(TCGContext *s, const TCGOp *op)
const TCGOutOpQemuLdSt *out =
container_of(all_outop[op->opc], TCGOutOpQemuLdSt, base);
- out->out(s, type, new_args[0], new_args[1], new_args[2]);
+ if (new_args[3]) {
+ out->out_disp(s, type, new_args[0], new_args[1],
+ new_args[2], new_args[3]);
+ } else {
+ out->out(s, type, new_args[0], new_args[1], new_args[2]);
+ }
}
break;
@@ -6611,6 +6740,7 @@ int tcg_gen_code(TCGContext *s, TranslationBlock *tb, uint64_t pc_start)
tcg_temp_ebb_reset_freed(s);
tcg_optimize(s);
+ fold_ldst_disp(s);
reachable_code_pass(s);
liveness_pass_0(s);
diff --git ./tcg/x86_64/tcg-target.c.inc ./tcg/x86_64/tcg-target.c.inc
index 2c8f1f3e58..9b177d3475 100644
--- ./tcg/x86_64/tcg-target.c.inc
+++ ./tcg/x86_64/tcg-target.c.inc
@@ -1892,6 +1892,18 @@ static HostAddress x86_guest_base = {
.index = -1
};
+/*
+ * Whether the displacement of a guest access can be folded into the host
+ * addressing mode rather than materialized by a separate lea. The generic
+ * pass has already established that the access has no slow path, so all
+ * that is left is guest_base, which shares the disp32 field.
+ */
+static bool tcg_target_ldst_disp_ok(TCGContext *s, MemOp opc, int32_t disp)
+{
+ int64_t ofs = (int64_t)x86_guest_base.ofs + disp;
+ return ofs == (int32_t)ofs;
+}
+
#if defined(__linux__)
# include <asm/prctl.h>
# include <sys/prctl.h>
@@ -1917,6 +1929,7 @@ static inline int setup_guest_base_seg(void)
#endif
#else
# define x86_guest_base (*(HostAddress *)({ qemu_build_not_reached(); NULL; }))
+# define tcg_target_ldst_disp_ok(s, opc, disp) false
#endif /* CONFIG_USER_ONLY */
#ifndef setup_guest_base_seg
# define setup_guest_base_seg() 0
@@ -2183,9 +2196,23 @@ static void tgen_qemu_ld(TCGContext *s, TCGType type, TCGReg data,
}
}
+static void tgen_qemu_ld_disp(TCGContext *s, TCGType type, TCGReg data,
+ TCGReg addr, MemOpIdx oi, int32_t disp)
+{
+ TCGLabelQemuLdst *ldst;
+ HostAddress h;
+
+ ldst = prepare_host_addr(s, &h, addr, oi, true);
+ /* tcg_target_ldst_disp_ok() has ruled out every slow path. */
+ tcg_debug_assert(ldst == NULL);
+ h.ofs += disp;
+ tcg_out_qemu_ld_direct(s, data, -1, h, type, get_memop(oi));
+}
+
static const TCGOutOpQemuLdSt outop_qemu_ld = {
.base.static_constraint = C_O1_I1(r, L),
.out = tgen_qemu_ld,
+ .out_disp = tgen_qemu_ld_disp,
};
static void tgen_qemu_ld2(TCGContext *s, TCGType type, TCGReg datalo,
@@ -2321,9 +2348,23 @@ static void tgen_qemu_st(TCGContext *s, TCGType type, TCGReg data,
}
}
+static void tgen_qemu_st_disp(TCGContext *s, TCGType type, TCGReg data,
+ TCGReg addr, MemOpIdx oi, int32_t disp)
+{
+ TCGLabelQemuLdst *ldst;
+ HostAddress h;
+
+ ldst = prepare_host_addr(s, &h, addr, oi, false);
+ /* tcg_target_ldst_disp_ok() has ruled out every slow path. */
+ tcg_debug_assert(ldst == NULL);
+ h.ofs += disp;
+ tcg_out_qemu_st_direct(s, data, -1, h, get_memop(oi));
+}
+
static const TCGOutOpQemuLdSt outop_qemu_st = {
.base.static_constraint = C_O0_I2(L, L),
.out = tgen_qemu_st,
+ .out_disp = tgen_qemu_st_disp,
};
static void tgen_qemu_st2(TCGContext *s, TCGType type, TCGReg datalo,
diff --git ./tcg/x86_64/tcg-target.h ./tcg/x86_64/tcg-target.h
index 7ebae56a7d..8f2315c15e 100644
--- ./tcg/x86_64/tcg-target.h
+++ ./tcg/x86_64/tcg-target.h
@@ -30,6 +30,9 @@
#define TCG_TARGET_NB_REGS 32
#define MAX_CODE_GEN_BUFFER_SIZE (2 * GiB)
+/* A guest displacement can go in the disp32 of the addressing mode. */
+#define TCG_TARGET_HAS_ldst_disp 1
+
typedef enum {
TCG_REG_EAX = 0,
TCG_REG_ECX,
--
2.54.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
end of thread, other threads:[~2026-09-01 3:50 UTC | newest]
Thread overview: 47+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-08-22 19:08 [PATCH v3 0/7] accel/tcg: cut per-block dispatch overhead Matt Turner
2026-08-22 19:08 ` [PATCH v3 1/7] accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags Matt Turner
2026-08-25 21:47 ` Richard Henderson
2026-08-27 4:57 ` Matt Turner
2026-08-26 7:46 ` Alex Bennée
2026-08-27 4:57 ` Matt Turner
2026-08-22 19:08 ` [PATCH v3 2/7] accel/tcg: enlarge the TB jump cache to 64K entries Matt Turner
2026-08-25 21:50 ` Richard Henderson
2026-08-27 4:57 ` Matt Turner
2026-08-22 19:08 ` [PATCH v3 3/7] accel/tcg: skip the can_do_io stores in user-only builds Matt Turner
2026-08-22 19:08 ` [PATCH v3 4/7] RFC: tcg: probe the TB jump cache inline instead of calling a helper Matt Turner
2026-08-25 22:28 ` Richard Henderson
2026-08-27 5:00 ` Matt Turner
2026-08-22 19:08 ` [PATCH v3 5/7] RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds Matt Turner
2026-08-26 7:51 ` Alex Bennée
2026-08-27 4:57 ` Matt Turner
2026-08-22 19:08 ` [PATCH v3 6/7] RFC: accel/tcg: poison the jump cache instead of polling for indirect exits Matt Turner
2026-08-22 19:08 ` [PATCH v3 7/7] RFC: tcg: fold a guest displacement into the host addressing mode Matt Turner
2026-08-25 22:52 ` Richard Henderson
2026-08-27 4:57 ` Matt Turner
2026-08-27 5:02 ` [PATCH v4 0/9] accel/tcg: cut per-block dispatch overhead Matt Turner
2026-09-01 3:47 ` [PATCH v5 " Matt Turner
2026-09-01 3:48 ` [PATCH v5 1/9] accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags Matt Turner
2026-09-01 3:48 ` [PATCH v5 2/9] accel/tcg: enlarge the TB jump cache to 64K entries Matt Turner
2026-09-01 3:48 ` [PATCH v5 3/9] accel/tcg: skip the can_do_io stores in user-only builds Matt Turner
2026-09-01 3:48 ` [PATCH v5 4/9] tcg: add tcg_gen_goto_jc_{i32,i64,tl}() Matt Turner
2026-09-01 3:48 ` [PATCH v5 5/9] accel/tcg: add CF_NO_GOTO_JC, set while a breakpoint is present Matt Turner
2026-09-01 3:48 ` [PATCH v5 6/9] RFC: tcg: probe the TB jump cache inline instead of calling a helper Matt Turner
2026-09-01 3:48 ` [PATCH v5 7/9] RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds Matt Turner
2026-09-01 3:48 ` [PATCH v5 8/9] RFC: accel/tcg: poison the jump cache instead of polling for indirect exits Matt Turner
2026-09-01 3:48 ` [PATCH v5 9/9] RFC: tcg: fold a guest displacement into the host addressing mode Matt Turner
2026-08-27 5:02 ` [PATCH v4 1/9] accel/tcg: fold the dynamic cflags into CPUState::tcg_cflags Matt Turner
2026-08-27 18:51 ` Richard Henderson
2026-08-27 5:02 ` [PATCH v4 2/9] accel/tcg: enlarge the TB jump cache to 64K entries Matt Turner
2026-08-27 5:02 ` [PATCH v4 3/9] accel/tcg: skip the can_do_io stores in user-only builds Matt Turner
2026-08-27 5:02 ` [PATCH v4 4/9] tcg: pass the destination to tcg_gen_lookup_and_goto_ptr() Matt Turner
2026-08-27 23:12 ` Richard Henderson
2026-09-01 2:55 ` Matt Turner
2026-08-27 5:02 ` [PATCH v4 5/9] accel/tcg: give the TB jump cache a second base pointer for generated code Matt Turner
2026-08-27 20:03 ` Richard Henderson
2026-09-01 2:55 ` Matt Turner
2026-08-27 5:02 ` [PATCH v4 6/9] RFC: tcg: probe the TB jump cache inline instead of calling a helper Matt Turner
2026-08-27 23:34 ` Richard Henderson
2026-09-01 2:55 ` Matt Turner
2026-08-27 5:02 ` [PATCH v4 7/9] RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds Matt Turner
2026-08-27 5:02 ` [PATCH v4 8/9] RFC: accel/tcg: poison the jump cache instead of polling for indirect exits Matt Turner
2026-08-27 5:02 ` [PATCH v4 9/9] RFC: tcg: fold a guest displacement into the host addressing mode Matt Turner
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.