* [PATCH 00/84] tcg: Build once for system, once for user
@ 2023-05-03 7:20 Richard Henderson
2023-05-03 7:20 ` [PULL v2 01/12] softmmu: Tidy dirtylimit_dirty_ring_full_time Richard Henderson
` (19 more replies)
0 siblings, 20 replies; 24+ messages in thread
From: Richard Henderson @ 2023-05-03 7:20 UTC (permalink / raw)
To: qemu-devel; +Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo
Based-on: 20230503070656.1746170-1-richard.henderson@linaro.org
("[PATCH v4 00/57] tcg: Improve atomicity support")
and also
Based-on: 20230502160846.1289975-1-richard.henderson@linaro.org
("[PATCH 00/16] tcg: Remove TARGET_ALIGNED_ONLY")
The goal here is only tcg/, leaving accel/tcg/ for future work.
r~
Richard Henderson (84):
tcg: Split out memory ops to tcg-op-ldst.c
tcg: Widen gen_insn_data to uint64_t
accel/tcg: Widen tcg-ldst.h addresses to uint64_t
tcg: Widen helper_{ld,st}_i128 addresses to uint64_t
tcg: Widen helper_atomic_* addresses to uint64_t
tcg: Widen tcg_gen_code pc_start argument to uint64_t
accel/tcg: Merge gen_mem_wrapped with plugin_gen_empty_mem_callback
accel/tcg: Merge do_gen_mem_cb into caller
tcg: Reduce copies for plugin_gen_mem_callbacks
accel/tcg: Widen plugin_gen_empty_mem_callback to i64
tcg: Add addr_type to TCGContext
tcg: Remove TCGv from tcg_gen_qemu_{ld,st}_*
tcg: Remove TCGv from tcg_gen_atomic_*
tcg: Split INDEX_op_qemu_{ld,st}* for guest address size
tcg/tci: Elimnate TARGET_LONG_BITS, target_ulong
tcg/i386: Always enable TCG_TARGET_HAS_extr[lh]_i64_i32
tcg/i386: Conditionalize tcg_out_extu_i32_i64
tcg/i386: Adjust type of tlb_mask
tcg/i386: Remove TARGET_LONG_BITS, TCG_TYPE_TL
tcg/arm: Remove TARGET_LONG_BITS
tcg/aarch64: Remove USE_GUEST_BASE
tcg/aarch64: Remove TARGET_LONG_BITS, TCG_TYPE_TL
tcg/loongarch64: Remove TARGET_LONG_BITS, TCG_TYPE_TL
tcg/mips: Remove TARGET_LONG_BITS, TCG_TYPE_TL
tcg/ppc: Remove TARGET_LONG_BITS, TCG_TYPE_TL
tcg/riscv: Remove TARGET_LONG_BITS, TCG_TYPE_TL
tcg/s390x: Remove TARGET_LONG_BITS, TCG_TYPE_TL
tcg/sparc64: Remove TARGET_LONG_BITS, TCG_TYPE_TL
tcg: Remove TARGET_LONG_BITS, TCG_TYPE_TL
tcg: Move TCG_TYPE_TL from tcg.h to tcg-op.h
tcg: Add page_bits and page_mask to TCGContext
tcg: Add tlb_dyn_max_bits to TCGContext
tcg: Widen CPUTLBEntry comparators to 64-bits
tcg: Add tlb_fast_offset to TCGContext
tcg: Remove TCG_TARGET_TLB_DISPLACEMENT_BITS
tcg: Split out tcg/debug-assert.h
*: Add missing includes of qemu/error-report.h
*: Add missing includes of tcg/debug-assert.h
*: Add missing includes of tcg/tcg.h
tcg: Split out tcg-target-reg-bits.h
target/arm: Fix test of TCG_OVERSIZED_GUEST
tcg: Split out tcg/oversized-guest.h
tcg: Move TCGv, dup_const_tl definitions to tcg-op.h
tcg: Split tcg/tcg-op-common.h from tcg/tcg-op.h
target/arm: Include helper-gen.h in translator.h
target/hexagon: Include helper-gen.h where needed
tcg: Remove outdated comments in helper-head.h
tcg: Move TCGHelperInfo and dependencies to tcg/helper-info.h
tcg: Pass TCGHelperInfo to tcg_gen_callN
tcg: Move temp_idx and tcgv_i32_temp debug out of line
tcg: Split tcg_gen_callN
tcg: Split helper-gen.h
tcg: Split helper-proto.h
tcg: Add insn_start_words to TCGContext
tcg: Add guest_mo to TCGContext
tcg: Move TLB_FLAGS_MASK check out of get_alignment_bits
tcg: Split tcg/tcg-op-gvec.h
tcg: Remove NO_CPU_IO_DEFS
exec-all: Widen tb_page_addr_t for user-only
exec-all: Widen TranslationBlock pc and cs_base to 64-bits
tcg: Remove DEBUG_DISAS
tcg: Remove USE_TCG_OPTIMIZATIONS
tcg: Spit out exec/translation-block.h
include/exec: Remove CODE_GEN_AVG_BLOCK_SIZE
accel/tcg: Move most of gen-icount.h into translator.c
accel/tcg: Introduce translator_io_start
accel/tcg: Move translator_fake_ldb out of line
target/arm: Tidy helpers for translation
target/mips: Tidy helpers for translation
*: Add missing includes of exec/translation-block.h
*: Add missing includes of exec/exec-all.h
accel/tcg: Tidy includes for translator.[ch]
tcg: Define IN_TCG
tcg: Fix PAGE/PROT confusion
tcg: Move env defines out of NEED_CPU_H in helper-head.h
tcg: Remove target-specific headers from tcg.[ch]
plugins: Move plugin_insn_append to translator.c
plugins: Drop unused headers from exec/plugin-gen.h
disas: Move disas.c to disas/
disas: Remove target_ulong from the interface
tcg: Split out exec/user/guest-base.h
disas: Remove target-specific headers
exec/poison: Do not poison CONFIG_SOFTMMU
tcg: Build once for system and once for user-only
accel/tcg/tcg-runtime.h | 50 +-
include/disas/disas.h | 23 +-
include/exec/cpu-all.h | 8 +-
include/exec/cpu-defs.h | 50 +-
include/exec/cpu_ldst.h | 22 +-
include/exec/exec-all.h | 148 +-
include/exec/gen-icount.h | 83 --
include/exec/helper-gen-common.h | 17 +
include/exec/helper-gen.h | 96 +-
include/exec/helper-head.h | 24 +-
include/exec/helper-proto-common.h | 17 +
include/exec/helper-proto.h | 72 +-
include/exec/helper-tcg.h | 75 -
include/exec/plugin-gen.h | 28 +-
include/exec/poison.h | 1 -
include/exec/tlb-common.h | 56 +
include/exec/translation-block.h | 152 ++
include/exec/translator.h | 24 +-
include/exec/user/guest-base.h | 12 +
include/qemu/typedefs.h | 1 +
include/tcg/debug-assert.h | 17 +
include/tcg/helper-info.h | 64 +
include/tcg/insn-start-words.h | 17 +
include/tcg/oversized-guest.h | 23 +
include/tcg/tcg-ldst.h | 26 +-
include/tcg/tcg-op-common.h | 996 +++++++++++++
include/tcg/tcg-op-gvec-common.h | 426 ++++++
include/tcg/tcg-op-gvec.h | 444 +-----
include/tcg/tcg-op.h | 1118 ++-------------
include/tcg/tcg-opc.h | 41 +-
include/tcg/tcg.h | 153 +-
target/arm/tcg/translate.h | 5 +
target/mips/tcg/translate.h | 5 +-
target/ppc/cpu.h | 2 -
target/sparc/cpu.h | 2 -
tcg/aarch64/tcg-target-reg-bits.h | 12 +
tcg/aarch64/tcg-target.h | 1 -
tcg/arm/tcg-target-reg-bits.h | 12 +
tcg/arm/tcg-target.h | 1 -
tcg/i386/tcg-target-reg-bits.h | 16 +
tcg/i386/tcg-target.h | 9 +-
tcg/loongarch64/tcg-target-reg-bits.h | 21 +
tcg/loongarch64/tcg-target.h | 11 -
tcg/mips/tcg-target-reg-bits.h | 18 +
tcg/mips/tcg-target.h | 9 -
tcg/ppc/tcg-target-reg-bits.h | 16 +
tcg/ppc/tcg-target.h | 6 -
tcg/riscv/tcg-target-reg-bits.h | 19 +
tcg/riscv/tcg-target.h | 10 -
tcg/s390x/tcg-target-reg-bits.h | 17 +
tcg/s390x/tcg-target.h | 1 -
tcg/sparc64/tcg-target-reg-bits.h | 12 +
tcg/sparc64/tcg-target.h | 1 -
tcg/tcg-internal.h | 47 +-
tcg/tci/tcg-target-reg-bits.h | 18 +
tcg/tci/tcg-target.h | 9 -
accel/tcg/cpu-exec.c | 4 +-
accel/tcg/cputlb.c | 43 +-
accel/tcg/monitor.c | 1 +
accel/tcg/perf.c | 8 +-
accel/tcg/plugin-gen.c | 74 +-
accel/tcg/tcg-accel-ops-mttcg.c | 2 +-
accel/tcg/tcg-accel-ops-rr.c | 2 +-
accel/tcg/tcg-all.c | 1 +
accel/tcg/tcg-runtime-gvec.c | 2 +-
accel/tcg/tcg-runtime.c | 6 +-
accel/tcg/translate-all.c | 58 +-
accel/tcg/translator.c | 142 +-
accel/tcg/user-exec.c | 31 +-
disas.c => disas/disas.c | 22 +-
linux-user/elfload.c | 5 +-
target/alpha/translate.c | 18 +-
target/arm/ptw.c | 8 +-
target/arm/tcg/translate-a64.c | 38 +-
target/arm/tcg/translate-m-nocp.c | 2 -
target/arm/tcg/translate-mve.c | 4 -
target/arm/tcg/translate-neon.c | 4 -
target/arm/tcg/translate-sme.c | 7 -
target/arm/tcg/translate-sve.c | 11 -
target/arm/tcg/translate-vfp.c | 7 +-
target/arm/tcg/translate.c | 41 +-
target/avr/cpu.c | 1 +
target/avr/helper.c | 1 +
target/avr/translate.c | 6 +-
target/cris/translate.c | 8 +-
target/hexagon/genptr.c | 1 +
target/hexagon/translate.c | 7 +
target/hppa/translate.c | 10 +-
target/i386/helper.c | 3 +
target/i386/tcg/translate.c | 57 +-
target/loongarch/translate.c | 6 +-
target/m68k/translate.c | 5 +-
target/microblaze/translate.c | 6 +-
target/mips/tcg/msa_translate.c | 3 -
target/mips/tcg/mxu_translate.c | 2 -
target/mips/tcg/octeon_translate.c | 4 +-
target/mips/tcg/rel6_translate.c | 2 -
target/mips/tcg/translate.c | 53 +-
target/mips/tcg/translate_addr_const.c | 1 -
target/mips/tcg/tx79_translate.c | 4 +-
target/mips/tcg/vr54xx_translate.c | 3 -
target/nios2/translate.c | 6 +-
target/openrisc/sys_helper.c | 1 +
target/openrisc/translate.c | 13 +-
target/ppc/translate.c | 17 +-
target/riscv/cpu_helper.c | 1 +
target/riscv/translate.c | 6 +-
target/rx/cpu.c | 1 +
target/rx/op_helper.c | 1 +
target/rx/translate.c | 7 +-
target/s390x/tcg/translate.c | 10 +-
target/sh4/translate.c | 8 +-
target/sparc/translate.c | 80 +-
target/tricore/cpu.c | 1 +
target/tricore/translate.c | 7 +-
target/xtensa/translate.c | 31 +-
tcg/optimize.c | 21 +-
tcg/region.c | 22 +-
tcg/tcg-common.c | 2 +
tcg/tcg-op-gvec.c | 6 +-
tcg/tcg-op-ldst.c | 1259 +++++++++++++++++
tcg/tcg-op-vec.c | 4 +-
tcg/tcg-op.c | 991 +------------
tcg/tcg.c | 359 +++--
tcg/tci.c | 96 +-
MAINTAINERS | 1 -
accel/tcg/atomic_common.c.inc | 14 +-
disas/meson.build | 4 +-
include/exec/helper-gen.h.inc | 101 ++
include/exec/helper-info.c.inc | 95 ++
include/exec/helper-proto.h.inc | 67 +
meson.build | 3 -
scripts/make-config-poison.sh | 5 +-
target/hexagon/idef-parser/idef-parser.y | 3 +-
target/loongarch/insn_trans/trans_extra.c.inc | 4 +-
.../insn_trans/trans_privileged.c.inc | 4 +-
target/riscv/insn_trans/trans_rvi.c.inc | 24 +-
tcg/aarch64/tcg-target.c.inc | 80 +-
tcg/arm/tcg-target.c.inc | 115 +-
tcg/i386/tcg-target.c.inc | 120 +-
tcg/loongarch64/tcg-target.c.inc | 45 +-
tcg/meson.build | 31 +-
tcg/mips/tcg-target.c.inc | 112 +-
tcg/ppc/tcg-target.c.inc | 151 +-
tcg/riscv/tcg-target.c.inc | 47 +-
tcg/s390x/tcg-target.c.inc | 62 +-
tcg/sparc64/tcg-target.c.inc | 48 +-
tcg/tci/tcg-target.c.inc | 53 +-
148 files changed, 5143 insertions(+), 4284 deletions(-)
delete mode 100644 include/exec/gen-icount.h
create mode 100644 include/exec/helper-gen-common.h
create mode 100644 include/exec/helper-proto-common.h
delete mode 100644 include/exec/helper-tcg.h
create mode 100644 include/exec/tlb-common.h
create mode 100644 include/exec/translation-block.h
create mode 100644 include/exec/user/guest-base.h
create mode 100644 include/tcg/debug-assert.h
create mode 100644 include/tcg/helper-info.h
create mode 100644 include/tcg/insn-start-words.h
create mode 100644 include/tcg/oversized-guest.h
create mode 100644 include/tcg/tcg-op-common.h
create mode 100644 include/tcg/tcg-op-gvec-common.h
create mode 100644 tcg/aarch64/tcg-target-reg-bits.h
create mode 100644 tcg/arm/tcg-target-reg-bits.h
create mode 100644 tcg/i386/tcg-target-reg-bits.h
create mode 100644 tcg/loongarch64/tcg-target-reg-bits.h
create mode 100644 tcg/mips/tcg-target-reg-bits.h
create mode 100644 tcg/ppc/tcg-target-reg-bits.h
create mode 100644 tcg/riscv/tcg-target-reg-bits.h
create mode 100644 tcg/s390x/tcg-target-reg-bits.h
create mode 100644 tcg/sparc64/tcg-target-reg-bits.h
create mode 100644 tcg/tci/tcg-target-reg-bits.h
rename disas.c => disas/disas.c (95%)
create mode 100644 tcg/tcg-op-ldst.c
create mode 100644 include/exec/helper-gen.h.inc
create mode 100644 include/exec/helper-info.c.inc
create mode 100644 include/exec/helper-proto.h.inc
--
2.34.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PULL v2 01/12] softmmu: Tidy dirtylimit_dirty_ring_full_time
2023-05-03 7:20 [PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
@ 2023-05-03 7:20 ` Richard Henderson
2023-05-03 7:20 ` [PATCH 01/84] tcg: Split out memory ops to tcg-op-ldst.c Richard Henderson
` (18 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Richard Henderson @ 2023-05-03 7:20 UTC (permalink / raw)
To: qemu-devel
Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo, Thomas Huth,
Juan Quintela
Drop inline marker: let compiler decide.
Change return type to uint64_t: this matches the computation in the
return statement and the local variable assignment in the caller.
Rename local to dirty_ring_size_MB to fix typo.
Simplify conversion to MiB via qemu_target_page_bits and right shift.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
softmmu/dirtylimit.c | 15 ++++++++++-----
1 file changed, 10 insertions(+), 5 deletions(-)
diff --git a/softmmu/dirtylimit.c b/softmmu/dirtylimit.c
index 82986c1499..71bf6dc7a4 100644
--- a/softmmu/dirtylimit.c
+++ b/softmmu/dirtylimit.c
@@ -232,18 +232,23 @@ bool dirtylimit_vcpu_index_valid(int cpu_index)
cpu_index >= ms->smp.max_cpus);
}
-static inline int64_t dirtylimit_dirty_ring_full_time(uint64_t dirtyrate)
+static uint64_t dirtylimit_dirty_ring_full_time(uint64_t dirtyrate)
{
static uint64_t max_dirtyrate;
- uint32_t dirty_ring_size = kvm_dirty_ring_size();
- uint64_t dirty_ring_size_meory_MB =
- dirty_ring_size * qemu_target_page_size() >> 20;
+ unsigned target_page_bits = qemu_target_page_bits();
+ uint64_t dirty_ring_size_MB;
+
+ /* So far, the largest (non-huge) page size is 64k, i.e. 16 bits. */
+ assert(target_page_bits < 20);
+
+ /* Convert ring size (pages) to MiB (2**20). */
+ dirty_ring_size_MB = kvm_dirty_ring_size() >> (20 - target_page_bits);
if (max_dirtyrate < dirtyrate) {
max_dirtyrate = dirtyrate;
}
- return dirty_ring_size_meory_MB * 1000000 / max_dirtyrate;
+ return dirty_ring_size_MB * 1000000 / max_dirtyrate;
}
static inline bool dirtylimit_done(uint64_t quota,
--
2.34.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH 01/84] tcg: Split out memory ops to tcg-op-ldst.c
2023-05-03 7:20 [PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
2023-05-03 7:20 ` [PULL v2 01/12] softmmu: Tidy dirtylimit_dirty_ring_full_time Richard Henderson
@ 2023-05-03 7:20 ` Richard Henderson
2023-05-03 7:20 ` [PULL v2 02/12] accel/tcg: Uncache the host address for instruction fetch when tlb size < 1 Richard Henderson
` (17 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Richard Henderson @ 2023-05-03 7:20 UTC (permalink / raw)
To: qemu-devel; +Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
tcg/tcg-op-ldst.c | 1017 +++++++++++++++++++++++++++++++++++++++++++++
tcg/tcg-op.c | 985 -------------------------------------------
tcg/meson.build | 1 +
3 files changed, 1018 insertions(+), 985 deletions(-)
create mode 100644 tcg/tcg-op-ldst.c
diff --git a/tcg/tcg-op-ldst.c b/tcg/tcg-op-ldst.c
new file mode 100644
index 0000000000..a3beede2f4
--- /dev/null
+++ b/tcg/tcg-op-ldst.c
@@ -0,0 +1,1017 @@
+/*
+ * Tiny Code Generator for QEMU
+ *
+ * Copyright (c) 2008 Fabrice Bellard
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#include "qemu/osdep.h"
+#include "exec/exec-all.h"
+#include "tcg/tcg.h"
+#include "tcg/tcg-temp-internal.h"
+#include "tcg/tcg-op.h"
+#include "tcg/tcg-mo.h"
+#include "exec/plugin-gen.h"
+#include "tcg-internal.h"
+
+
+static inline MemOp tcg_canonicalize_memop(MemOp op, bool is64, bool st)
+{
+ /* Trigger the asserts within as early as possible. */
+ unsigned a_bits = get_alignment_bits(op);
+
+ /* Prefer MO_ALIGN+MO_XX over MO_ALIGN_XX+MO_XX */
+ if (a_bits == (op & MO_SIZE)) {
+ op = (op & ~MO_AMASK) | MO_ALIGN;
+ }
+
+ switch (op & MO_SIZE) {
+ case MO_8:
+ op &= ~MO_BSWAP;
+ break;
+ case MO_16:
+ break;
+ case MO_32:
+ if (!is64) {
+ op &= ~MO_SIGN;
+ }
+ break;
+ case MO_64:
+ if (is64) {
+ op &= ~MO_SIGN;
+ break;
+ }
+ /* fall through */
+ default:
+ g_assert_not_reached();
+ }
+ if (st) {
+ op &= ~MO_SIGN;
+ }
+ return op;
+}
+
+static void gen_ldst_i32(TCGOpcode opc, TCGv_i32 val, TCGv addr,
+ MemOp memop, TCGArg idx)
+{
+ MemOpIdx oi = make_memop_idx(memop, idx);
+#if TARGET_LONG_BITS == 32
+ tcg_gen_op3i_i32(opc, val, addr, oi);
+#else
+ if (TCG_TARGET_REG_BITS == 32) {
+ tcg_gen_op4i_i32(opc, val, TCGV_LOW(addr), TCGV_HIGH(addr), oi);
+ } else {
+ tcg_gen_op3(opc, tcgv_i32_arg(val), tcgv_i64_arg(addr), oi);
+ }
+#endif
+}
+
+static void gen_ldst_i64(TCGOpcode opc, TCGv_i64 val, TCGv addr,
+ MemOp memop, TCGArg idx)
+{
+ MemOpIdx oi = make_memop_idx(memop, idx);
+#if TARGET_LONG_BITS == 32
+ if (TCG_TARGET_REG_BITS == 32) {
+ tcg_gen_op4i_i32(opc, TCGV_LOW(val), TCGV_HIGH(val), addr, oi);
+ } else {
+ tcg_gen_op3(opc, tcgv_i64_arg(val), tcgv_i32_arg(addr), oi);
+ }
+#else
+ if (TCG_TARGET_REG_BITS == 32) {
+ tcg_gen_op5i_i32(opc, TCGV_LOW(val), TCGV_HIGH(val),
+ TCGV_LOW(addr), TCGV_HIGH(addr), oi);
+ } else {
+ tcg_gen_op3i_i64(opc, val, addr, oi);
+ }
+#endif
+}
+
+static void tcg_gen_req_mo(TCGBar type)
+{
+#ifdef TCG_GUEST_DEFAULT_MO
+ type &= TCG_GUEST_DEFAULT_MO;
+#endif
+ type &= ~TCG_TARGET_DEFAULT_MO;
+ if (type) {
+ tcg_gen_mb(type | TCG_BAR_SC);
+ }
+}
+
+static inline TCGv plugin_prep_mem_callbacks(TCGv vaddr)
+{
+#ifdef CONFIG_PLUGIN
+ if (tcg_ctx->plugin_insn != NULL) {
+ /* Save a copy of the vaddr for use after a load. */
+ TCGv temp = tcg_temp_new();
+ tcg_gen_mov_tl(temp, vaddr);
+ return temp;
+ }
+#endif
+ return vaddr;
+}
+
+static void plugin_gen_mem_callbacks(TCGv vaddr, MemOpIdx oi,
+ enum qemu_plugin_mem_rw rw)
+{
+#ifdef CONFIG_PLUGIN
+ if (tcg_ctx->plugin_insn != NULL) {
+ qemu_plugin_meminfo_t info = make_plugin_meminfo(oi, rw);
+ plugin_gen_empty_mem_callback(vaddr, info);
+ tcg_temp_free(vaddr);
+ }
+#endif
+}
+
+void tcg_gen_qemu_ld_i32(TCGv_i32 val, TCGv addr, TCGArg idx, MemOp memop)
+{
+ MemOp orig_memop;
+ MemOpIdx oi;
+
+ tcg_gen_req_mo(TCG_MO_LD_LD | TCG_MO_ST_LD);
+ memop = tcg_canonicalize_memop(memop, 0, 0);
+ oi = make_memop_idx(memop, idx);
+
+ orig_memop = memop;
+ if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) {
+ memop &= ~MO_BSWAP;
+ /* The bswap primitive benefits from zero-extended input. */
+ if ((memop & MO_SSIZE) == MO_SW) {
+ memop &= ~MO_SIGN;
+ }
+ }
+
+ addr = plugin_prep_mem_callbacks(addr);
+ gen_ldst_i32(INDEX_op_qemu_ld_i32, val, addr, memop, idx);
+ plugin_gen_mem_callbacks(addr, oi, QEMU_PLUGIN_MEM_R);
+
+ if ((orig_memop ^ memop) & MO_BSWAP) {
+ switch (orig_memop & MO_SIZE) {
+ case MO_16:
+ tcg_gen_bswap16_i32(val, val, (orig_memop & MO_SIGN
+ ? TCG_BSWAP_IZ | TCG_BSWAP_OS
+ : TCG_BSWAP_IZ | TCG_BSWAP_OZ));
+ break;
+ case MO_32:
+ tcg_gen_bswap32_i32(val, val);
+ break;
+ default:
+ g_assert_not_reached();
+ }
+ }
+}
+
+void tcg_gen_qemu_st_i32(TCGv_i32 val, TCGv addr, TCGArg idx, MemOp memop)
+{
+ TCGv_i32 swap = NULL;
+ MemOpIdx oi;
+
+ tcg_gen_req_mo(TCG_MO_LD_ST | TCG_MO_ST_ST);
+ memop = tcg_canonicalize_memop(memop, 0, 1);
+ oi = make_memop_idx(memop, idx);
+
+ if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) {
+ swap = tcg_temp_ebb_new_i32();
+ switch (memop & MO_SIZE) {
+ case MO_16:
+ tcg_gen_bswap16_i32(swap, val, 0);
+ break;
+ case MO_32:
+ tcg_gen_bswap32_i32(swap, val);
+ break;
+ default:
+ g_assert_not_reached();
+ }
+ val = swap;
+ memop &= ~MO_BSWAP;
+ }
+
+ addr = plugin_prep_mem_callbacks(addr);
+ if (TCG_TARGET_HAS_qemu_st8_i32 && (memop & MO_SIZE) == MO_8) {
+ gen_ldst_i32(INDEX_op_qemu_st8_i32, val, addr, memop, idx);
+ } else {
+ gen_ldst_i32(INDEX_op_qemu_st_i32, val, addr, memop, idx);
+ }
+ plugin_gen_mem_callbacks(addr, oi, QEMU_PLUGIN_MEM_W);
+
+ if (swap) {
+ tcg_temp_free_i32(swap);
+ }
+}
+
+void tcg_gen_qemu_ld_i64(TCGv_i64 val, TCGv addr, TCGArg idx, MemOp memop)
+{
+ MemOp orig_memop;
+ MemOpIdx oi;
+
+ if (TCG_TARGET_REG_BITS == 32 && (memop & MO_SIZE) < MO_64) {
+ tcg_gen_qemu_ld_i32(TCGV_LOW(val), addr, idx, memop);
+ if (memop & MO_SIGN) {
+ tcg_gen_sari_i32(TCGV_HIGH(val), TCGV_LOW(val), 31);
+ } else {
+ tcg_gen_movi_i32(TCGV_HIGH(val), 0);
+ }
+ return;
+ }
+
+ tcg_gen_req_mo(TCG_MO_LD_LD | TCG_MO_ST_LD);
+ memop = tcg_canonicalize_memop(memop, 1, 0);
+ oi = make_memop_idx(memop, idx);
+
+ orig_memop = memop;
+ if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) {
+ memop &= ~MO_BSWAP;
+ /* The bswap primitive benefits from zero-extended input. */
+ if ((memop & MO_SIGN) && (memop & MO_SIZE) < MO_64) {
+ memop &= ~MO_SIGN;
+ }
+ }
+
+ addr = plugin_prep_mem_callbacks(addr);
+ gen_ldst_i64(INDEX_op_qemu_ld_i64, val, addr, memop, idx);
+ plugin_gen_mem_callbacks(addr, oi, QEMU_PLUGIN_MEM_R);
+
+ if ((orig_memop ^ memop) & MO_BSWAP) {
+ int flags = (orig_memop & MO_SIGN
+ ? TCG_BSWAP_IZ | TCG_BSWAP_OS
+ : TCG_BSWAP_IZ | TCG_BSWAP_OZ);
+ switch (orig_memop & MO_SIZE) {
+ case MO_16:
+ tcg_gen_bswap16_i64(val, val, flags);
+ break;
+ case MO_32:
+ tcg_gen_bswap32_i64(val, val, flags);
+ break;
+ case MO_64:
+ tcg_gen_bswap64_i64(val, val);
+ break;
+ default:
+ g_assert_not_reached();
+ }
+ }
+}
+
+void tcg_gen_qemu_st_i64(TCGv_i64 val, TCGv addr, TCGArg idx, MemOp memop)
+{
+ TCGv_i64 swap = NULL;
+ MemOpIdx oi;
+
+ if (TCG_TARGET_REG_BITS == 32 && (memop & MO_SIZE) < MO_64) {
+ tcg_gen_qemu_st_i32(TCGV_LOW(val), addr, idx, memop);
+ return;
+ }
+
+ tcg_gen_req_mo(TCG_MO_LD_ST | TCG_MO_ST_ST);
+ memop = tcg_canonicalize_memop(memop, 1, 1);
+ oi = make_memop_idx(memop, idx);
+
+ if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) {
+ swap = tcg_temp_ebb_new_i64();
+ switch (memop & MO_SIZE) {
+ case MO_16:
+ tcg_gen_bswap16_i64(swap, val, 0);
+ break;
+ case MO_32:
+ tcg_gen_bswap32_i64(swap, val, 0);
+ break;
+ case MO_64:
+ tcg_gen_bswap64_i64(swap, val);
+ break;
+ default:
+ g_assert_not_reached();
+ }
+ val = swap;
+ memop &= ~MO_BSWAP;
+ }
+
+ addr = plugin_prep_mem_callbacks(addr);
+ gen_ldst_i64(INDEX_op_qemu_st_i64, val, addr, memop, idx);
+ plugin_gen_mem_callbacks(addr, oi, QEMU_PLUGIN_MEM_W);
+
+ if (swap) {
+ tcg_temp_free_i64(swap);
+ }
+}
+
+/*
+ * Return true if @mop, without knowledge of the pointer alignment,
+ * does not require 16-byte atomicity, and it would be adventagous
+ * to avoid a call to a helper function.
+ */
+static bool use_two_i64_for_i128(MemOp mop)
+{
+#ifdef CONFIG_SOFTMMU
+ /* Two softmmu tlb lookups is larger than one function call. */
+ return false;
+#else
+ /*
+ * For user-only, two 64-bit operations may well be smaller than a call.
+ * Determine if that would be legal for the requested atomicity.
+ */
+ MemOp atom = mop & MO_ATOM_MASK;
+ MemOp atmax = mop & MO_ATMAX_MASK;
+
+ /* In a serialized context, no atomicity is required. */
+ if (!(tcg_ctx->gen_tb->cflags & CF_PARALLEL)) {
+ return true;
+ }
+
+ if (atmax == MO_ATMAX_SIZE) {
+ atmax = mop & MO_SIZE;
+ } else {
+ atmax >>= MO_ATMAX_SHIFT;
+ }
+ switch (atom) {
+ case MO_ATOM_NONE:
+ return true;
+ case MO_ATOM_IFALIGN:
+ case MO_ATOM_SUBALIGN:
+ return atmax < MO_128;
+ case MO_ATOM_WITHIN16:
+ return atmax == MO_8;
+ default:
+ g_assert_not_reached();
+ }
+#endif
+}
+
+static void canonicalize_memop_i128_as_i64(MemOp ret[2], MemOp orig)
+{
+ MemOp mop_1 = orig, mop_2;
+
+ tcg_debug_assert((orig & MO_SIZE) == MO_128);
+ tcg_debug_assert((orig & MO_SIGN) == 0);
+
+ /* Reduce the size to 64-bit. */
+ mop_1 = (mop_1 & ~MO_SIZE) | MO_64;
+
+ /* Retain the alignment constraints of the original. */
+ switch (orig & MO_AMASK) {
+ case MO_UNALN:
+ case MO_ALIGN_2:
+ case MO_ALIGN_4:
+ mop_2 = mop_1;
+ break;
+ case MO_ALIGN_8:
+ /* Prefer MO_ALIGN+MO_64 to MO_ALIGN_8+MO_64. */
+ mop_1 = (mop_1 & ~MO_AMASK) | MO_ALIGN;
+ mop_2 = mop_1;
+ break;
+ case MO_ALIGN:
+ /* Second has 8-byte alignment; first has 16-byte alignment. */
+ mop_2 = mop_1;
+ mop_1 = (mop_1 & ~MO_AMASK) | MO_ALIGN_16;
+ break;
+ case MO_ALIGN_16:
+ case MO_ALIGN_32:
+ case MO_ALIGN_64:
+ /* Second has 8-byte alignment; first retains original. */
+ mop_2 = (mop_1 & ~MO_AMASK) | MO_ALIGN;
+ break;
+ default:
+ g_assert_not_reached();
+ }
+
+ /* Use a memory ordering implemented by the host. */
+ if ((orig & MO_BSWAP) && !tcg_target_has_memory_bswap(mop_1)) {
+ mop_1 &= ~MO_BSWAP;
+ mop_2 &= ~MO_BSWAP;
+ }
+
+ ret[0] = mop_1;
+ ret[1] = mop_2;
+}
+
+#if TARGET_LONG_BITS == 64
+#define tcg_temp_ebb_new tcg_temp_ebb_new_i64
+#else
+#define tcg_temp_ebb_new tcg_temp_ebb_new_i32
+#endif
+
+void tcg_gen_qemu_ld_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop)
+{
+ const MemOpIdx oi = make_memop_idx(memop, idx);
+
+ tcg_debug_assert((memop & MO_SIZE) == MO_128);
+ tcg_debug_assert((memop & MO_SIGN) == 0);
+
+ tcg_gen_req_mo(TCG_MO_LD_LD | TCG_MO_ST_LD);
+ addr = plugin_prep_mem_callbacks(addr);
+
+ /* TODO: For now, force 32-bit hosts to use the helper. */
+ if (TCG_TARGET_HAS_qemu_ldst_i128 && TCG_TARGET_REG_BITS == 64) {
+ TCGv_i64 lo, hi;
+ TCGArg addr_arg;
+ MemOpIdx adj_oi;
+ bool need_bswap = false;
+
+ if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) {
+ lo = TCGV128_HIGH(val);
+ hi = TCGV128_LOW(val);
+ adj_oi = make_memop_idx(memop & ~MO_BSWAP, idx);
+ need_bswap = true;
+ } else {
+ lo = TCGV128_LOW(val);
+ hi = TCGV128_HIGH(val);
+ adj_oi = oi;
+ }
+
+#if TARGET_LONG_BITS == 32
+ addr_arg = tcgv_i32_arg(addr);
+#else
+ addr_arg = tcgv_i64_arg(addr);
+#endif
+ tcg_gen_op4ii_i64(INDEX_op_qemu_ld_i128, lo, hi, addr_arg, adj_oi);
+
+ if (need_bswap) {
+ tcg_gen_bswap64_i64(lo, lo);
+ tcg_gen_bswap64_i64(hi, hi);
+ }
+ } else if (use_two_i64_for_i128(memop)) {
+ MemOp mop[2];
+ TCGv addr_p8;
+ TCGv_i64 x, y;
+
+ canonicalize_memop_i128_as_i64(mop, memop);
+
+ /*
+ * Since there are no global TCGv_i128, there is no visible state
+ * changed if the second load faults. Load directly into the two
+ * subwords.
+ */
+ if ((memop & MO_BSWAP) == MO_LE) {
+ x = TCGV128_LOW(val);
+ y = TCGV128_HIGH(val);
+ } else {
+ x = TCGV128_HIGH(val);
+ y = TCGV128_LOW(val);
+ }
+
+ gen_ldst_i64(INDEX_op_qemu_ld_i64, x, addr, mop[0], idx);
+
+ if ((mop[0] ^ memop) & MO_BSWAP) {
+ tcg_gen_bswap64_i64(x, x);
+ }
+
+ addr_p8 = tcg_temp_ebb_new();
+ tcg_gen_addi_tl(addr_p8, addr, 8);
+ gen_ldst_i64(INDEX_op_qemu_ld_i64, y, addr_p8, mop[1], idx);
+ tcg_temp_free(addr_p8);
+
+ if ((mop[0] ^ memop) & MO_BSWAP) {
+ tcg_gen_bswap64_i64(y, y);
+ }
+ } else {
+ gen_helper_ld_i128(val, cpu_env, addr, tcg_constant_i32(oi));
+ }
+
+ plugin_gen_mem_callbacks(addr, oi, QEMU_PLUGIN_MEM_R);
+}
+
+void tcg_gen_qemu_st_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop)
+{
+ const MemOpIdx oi = make_memop_idx(memop, idx);
+
+ tcg_debug_assert((memop & MO_SIZE) == MO_128);
+ tcg_debug_assert((memop & MO_SIGN) == 0);
+
+ tcg_gen_req_mo(TCG_MO_ST_LD | TCG_MO_ST_ST);
+ addr = plugin_prep_mem_callbacks(addr);
+
+ /* TODO: For now, force 32-bit hosts to use the helper. */
+
+ if (TCG_TARGET_HAS_qemu_ldst_i128 && TCG_TARGET_REG_BITS == 64) {
+ TCGv_i64 lo, hi;
+ TCGArg addr_arg;
+ MemOpIdx adj_oi;
+ bool need_bswap = false;
+
+ if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) {
+ lo = tcg_temp_new_i64();
+ hi = tcg_temp_new_i64();
+ tcg_gen_bswap64_i64(lo, TCGV128_HIGH(val));
+ tcg_gen_bswap64_i64(hi, TCGV128_LOW(val));
+ adj_oi = make_memop_idx(memop & ~MO_BSWAP, idx);
+ need_bswap = true;
+ } else {
+ lo = TCGV128_LOW(val);
+ hi = TCGV128_HIGH(val);
+ adj_oi = oi;
+ }
+
+#if TARGET_LONG_BITS == 32
+ addr_arg = tcgv_i32_arg(addr);
+#else
+ addr_arg = tcgv_i64_arg(addr);
+#endif
+ tcg_gen_op4ii_i64(INDEX_op_qemu_st_i128, lo, hi, addr_arg, adj_oi);
+
+ if (need_bswap) {
+ tcg_temp_free_i64(lo);
+ tcg_temp_free_i64(hi);
+ }
+ } else if (use_two_i64_for_i128(memop)) {
+ MemOp mop[2];
+ TCGv addr_p8;
+ TCGv_i64 x, y;
+
+ canonicalize_memop_i128_as_i64(mop, memop);
+
+ if ((memop & MO_BSWAP) == MO_LE) {
+ x = TCGV128_LOW(val);
+ y = TCGV128_HIGH(val);
+ } else {
+ x = TCGV128_HIGH(val);
+ y = TCGV128_LOW(val);
+ }
+
+ addr_p8 = tcg_temp_ebb_new();
+ if ((mop[0] ^ memop) & MO_BSWAP) {
+ TCGv_i64 t = tcg_temp_ebb_new_i64();
+
+ tcg_gen_bswap64_i64(t, x);
+ gen_ldst_i64(INDEX_op_qemu_st_i64, t, addr, mop[0], idx);
+ tcg_gen_bswap64_i64(t, y);
+ tcg_gen_addi_tl(addr_p8, addr, 8);
+ gen_ldst_i64(INDEX_op_qemu_st_i64, t, addr_p8, mop[1], idx);
+ tcg_temp_free_i64(t);
+ } else {
+ gen_ldst_i64(INDEX_op_qemu_st_i64, x, addr, mop[0], idx);
+ tcg_gen_addi_tl(addr_p8, addr, 8);
+ gen_ldst_i64(INDEX_op_qemu_st_i64, y, addr_p8, mop[1], idx);
+ }
+ tcg_temp_free(addr_p8);
+ } else {
+ gen_helper_st_i128(cpu_env, addr, val, tcg_constant_i32(oi));
+ }
+
+ plugin_gen_mem_callbacks(addr, oi, QEMU_PLUGIN_MEM_W);
+}
+
+static void tcg_gen_ext_i32(TCGv_i32 ret, TCGv_i32 val, MemOp opc)
+{
+ switch (opc & MO_SSIZE) {
+ case MO_SB:
+ tcg_gen_ext8s_i32(ret, val);
+ break;
+ case MO_UB:
+ tcg_gen_ext8u_i32(ret, val);
+ break;
+ case MO_SW:
+ tcg_gen_ext16s_i32(ret, val);
+ break;
+ case MO_UW:
+ tcg_gen_ext16u_i32(ret, val);
+ break;
+ default:
+ tcg_gen_mov_i32(ret, val);
+ break;
+ }
+}
+
+static void tcg_gen_ext_i64(TCGv_i64 ret, TCGv_i64 val, MemOp opc)
+{
+ switch (opc & MO_SSIZE) {
+ case MO_SB:
+ tcg_gen_ext8s_i64(ret, val);
+ break;
+ case MO_UB:
+ tcg_gen_ext8u_i64(ret, val);
+ break;
+ case MO_SW:
+ tcg_gen_ext16s_i64(ret, val);
+ break;
+ case MO_UW:
+ tcg_gen_ext16u_i64(ret, val);
+ break;
+ case MO_SL:
+ tcg_gen_ext32s_i64(ret, val);
+ break;
+ case MO_UL:
+ tcg_gen_ext32u_i64(ret, val);
+ break;
+ default:
+ tcg_gen_mov_i64(ret, val);
+ break;
+ }
+}
+
+typedef void (*gen_atomic_cx_i32)(TCGv_i32, TCGv_env, TCGv,
+ TCGv_i32, TCGv_i32, TCGv_i32);
+typedef void (*gen_atomic_cx_i64)(TCGv_i64, TCGv_env, TCGv,
+ TCGv_i64, TCGv_i64, TCGv_i32);
+typedef void (*gen_atomic_cx_i128)(TCGv_i128, TCGv_env, TCGv,
+ TCGv_i128, TCGv_i128, TCGv_i32);
+typedef void (*gen_atomic_op_i32)(TCGv_i32, TCGv_env, TCGv,
+ TCGv_i32, TCGv_i32);
+typedef void (*gen_atomic_op_i64)(TCGv_i64, TCGv_env, TCGv,
+ TCGv_i64, TCGv_i32);
+
+#ifdef CONFIG_ATOMIC64
+# define WITH_ATOMIC64(X) X,
+#else
+# define WITH_ATOMIC64(X)
+#endif
+#ifdef CONFIG_CMPXCHG128
+# define WITH_ATOMIC128(X) X,
+#else
+# define WITH_ATOMIC128(X)
+#endif
+
+static void * const table_cmpxchg[(MO_SIZE | MO_BSWAP) + 1] = {
+ [MO_8] = gen_helper_atomic_cmpxchgb,
+ [MO_16 | MO_LE] = gen_helper_atomic_cmpxchgw_le,
+ [MO_16 | MO_BE] = gen_helper_atomic_cmpxchgw_be,
+ [MO_32 | MO_LE] = gen_helper_atomic_cmpxchgl_le,
+ [MO_32 | MO_BE] = gen_helper_atomic_cmpxchgl_be,
+ WITH_ATOMIC64([MO_64 | MO_LE] = gen_helper_atomic_cmpxchgq_le)
+ WITH_ATOMIC64([MO_64 | MO_BE] = gen_helper_atomic_cmpxchgq_be)
+ WITH_ATOMIC128([MO_128 | MO_LE] = gen_helper_atomic_cmpxchgo_le)
+ WITH_ATOMIC128([MO_128 | MO_BE] = gen_helper_atomic_cmpxchgo_be)
+};
+
+void tcg_gen_nonatomic_cmpxchg_i32(TCGv_i32 retv, TCGv addr, TCGv_i32 cmpv,
+ TCGv_i32 newv, TCGArg idx, MemOp memop)
+{
+ TCGv_i32 t1 = tcg_temp_ebb_new_i32();
+ TCGv_i32 t2 = tcg_temp_ebb_new_i32();
+
+ tcg_gen_ext_i32(t2, cmpv, memop & MO_SIZE);
+
+ tcg_gen_qemu_ld_i32(t1, addr, idx, memop & ~MO_SIGN);
+ tcg_gen_movcond_i32(TCG_COND_EQ, t2, t1, t2, newv, t1);
+ tcg_gen_qemu_st_i32(t2, addr, idx, memop);
+ tcg_temp_free_i32(t2);
+
+ if (memop & MO_SIGN) {
+ tcg_gen_ext_i32(retv, t1, memop);
+ } else {
+ tcg_gen_mov_i32(retv, t1);
+ }
+ tcg_temp_free_i32(t1);
+}
+
+void tcg_gen_atomic_cmpxchg_i32(TCGv_i32 retv, TCGv addr, TCGv_i32 cmpv,
+ TCGv_i32 newv, TCGArg idx, MemOp memop)
+{
+ gen_atomic_cx_i32 gen;
+ MemOpIdx oi;
+
+ if (!(tcg_ctx->gen_tb->cflags & CF_PARALLEL)) {
+ tcg_gen_nonatomic_cmpxchg_i32(retv, addr, cmpv, newv, idx, memop);
+ return;
+ }
+
+ memop = tcg_canonicalize_memop(memop, 0, 0);
+ gen = table_cmpxchg[memop & (MO_SIZE | MO_BSWAP)];
+ tcg_debug_assert(gen != NULL);
+
+ oi = make_memop_idx(memop & ~MO_SIGN, idx);
+ gen(retv, cpu_env, addr, cmpv, newv, tcg_constant_i32(oi));
+
+ if (memop & MO_SIGN) {
+ tcg_gen_ext_i32(retv, retv, memop);
+ }
+}
+
+void tcg_gen_nonatomic_cmpxchg_i64(TCGv_i64 retv, TCGv addr, TCGv_i64 cmpv,
+ TCGv_i64 newv, TCGArg idx, MemOp memop)
+{
+ TCGv_i64 t1, t2;
+
+ if (TCG_TARGET_REG_BITS == 32 && (memop & MO_SIZE) < MO_64) {
+ tcg_gen_nonatomic_cmpxchg_i32(TCGV_LOW(retv), addr, TCGV_LOW(cmpv),
+ TCGV_LOW(newv), idx, memop);
+ if (memop & MO_SIGN) {
+ tcg_gen_sari_i32(TCGV_HIGH(retv), TCGV_LOW(retv), 31);
+ } else {
+ tcg_gen_movi_i32(TCGV_HIGH(retv), 0);
+ }
+ return;
+ }
+
+ t1 = tcg_temp_ebb_new_i64();
+ t2 = tcg_temp_ebb_new_i64();
+
+ tcg_gen_ext_i64(t2, cmpv, memop & MO_SIZE);
+
+ tcg_gen_qemu_ld_i64(t1, addr, idx, memop & ~MO_SIGN);
+ tcg_gen_movcond_i64(TCG_COND_EQ, t2, t1, t2, newv, t1);
+ tcg_gen_qemu_st_i64(t2, addr, idx, memop);
+ tcg_temp_free_i64(t2);
+
+ if (memop & MO_SIGN) {
+ tcg_gen_ext_i64(retv, t1, memop);
+ } else {
+ tcg_gen_mov_i64(retv, t1);
+ }
+ tcg_temp_free_i64(t1);
+}
+
+void tcg_gen_atomic_cmpxchg_i64(TCGv_i64 retv, TCGv addr, TCGv_i64 cmpv,
+ TCGv_i64 newv, TCGArg idx, MemOp memop)
+{
+ if (!(tcg_ctx->gen_tb->cflags & CF_PARALLEL)) {
+ tcg_gen_nonatomic_cmpxchg_i64(retv, addr, cmpv, newv, idx, memop);
+ return;
+ }
+
+ if ((memop & MO_SIZE) == MO_64) {
+ gen_atomic_cx_i64 gen;
+
+ memop = tcg_canonicalize_memop(memop, 1, 0);
+ gen = table_cmpxchg[memop & (MO_SIZE | MO_BSWAP)];
+ if (gen) {
+ MemOpIdx oi = make_memop_idx(memop, idx);
+ gen(retv, cpu_env, addr, cmpv, newv, tcg_constant_i32(oi));
+ return;
+ }
+
+ gen_helper_exit_atomic(cpu_env);
+
+ /*
+ * Produce a result for a well-formed opcode stream. This satisfies
+ * liveness for set before used, which happens before this dead code
+ * is removed.
+ */
+ tcg_gen_movi_i64(retv, 0);
+ return;
+ }
+
+ if (TCG_TARGET_REG_BITS == 32) {
+ tcg_gen_atomic_cmpxchg_i32(TCGV_LOW(retv), addr, TCGV_LOW(cmpv),
+ TCGV_LOW(newv), idx, memop);
+ if (memop & MO_SIGN) {
+ tcg_gen_sari_i32(TCGV_HIGH(retv), TCGV_LOW(retv), 31);
+ } else {
+ tcg_gen_movi_i32(TCGV_HIGH(retv), 0);
+ }
+ } else {
+ TCGv_i32 c32 = tcg_temp_ebb_new_i32();
+ TCGv_i32 n32 = tcg_temp_ebb_new_i32();
+ TCGv_i32 r32 = tcg_temp_ebb_new_i32();
+
+ tcg_gen_extrl_i64_i32(c32, cmpv);
+ tcg_gen_extrl_i64_i32(n32, newv);
+ tcg_gen_atomic_cmpxchg_i32(r32, addr, c32, n32, idx, memop & ~MO_SIGN);
+ tcg_temp_free_i32(c32);
+ tcg_temp_free_i32(n32);
+
+ tcg_gen_extu_i32_i64(retv, r32);
+ tcg_temp_free_i32(r32);
+
+ if (memop & MO_SIGN) {
+ tcg_gen_ext_i64(retv, retv, memop);
+ }
+ }
+}
+
+void tcg_gen_nonatomic_cmpxchg_i128(TCGv_i128 retv, TCGv addr, TCGv_i128 cmpv,
+ TCGv_i128 newv, TCGArg idx, MemOp memop)
+{
+ if (TCG_TARGET_REG_BITS == 32) {
+ /* Inline expansion below is simply too large for 32-bit hosts. */
+ gen_atomic_cx_i128 gen = ((memop & MO_BSWAP) == MO_LE
+ ? gen_helper_nonatomic_cmpxchgo_le
+ : gen_helper_nonatomic_cmpxchgo_be);
+ MemOpIdx oi = make_memop_idx(memop, idx);
+
+ tcg_debug_assert((memop & MO_SIZE) == MO_128);
+ tcg_debug_assert((memop & MO_SIGN) == 0);
+
+ gen(retv, cpu_env, addr, cmpv, newv, tcg_constant_i32(oi));
+ } else {
+ TCGv_i128 oldv = tcg_temp_ebb_new_i128();
+ TCGv_i128 tmpv = tcg_temp_ebb_new_i128();
+ TCGv_i64 t0 = tcg_temp_ebb_new_i64();
+ TCGv_i64 t1 = tcg_temp_ebb_new_i64();
+ TCGv_i64 z = tcg_constant_i64(0);
+
+ tcg_gen_qemu_ld_i128(oldv, addr, idx, memop);
+
+ /* Compare i128 */
+ tcg_gen_xor_i64(t0, TCGV128_LOW(oldv), TCGV128_LOW(cmpv));
+ tcg_gen_xor_i64(t1, TCGV128_HIGH(oldv), TCGV128_HIGH(cmpv));
+ tcg_gen_or_i64(t0, t0, t1);
+
+ /* tmpv = equal ? newv : oldv */
+ tcg_gen_movcond_i64(TCG_COND_EQ, TCGV128_LOW(tmpv), t0, z,
+ TCGV128_LOW(newv), TCGV128_LOW(oldv));
+ tcg_gen_movcond_i64(TCG_COND_EQ, TCGV128_HIGH(tmpv), t0, z,
+ TCGV128_HIGH(newv), TCGV128_HIGH(oldv));
+
+ /* Unconditional writeback. */
+ tcg_gen_qemu_st_i128(tmpv, addr, idx, memop);
+ tcg_gen_mov_i128(retv, oldv);
+
+ tcg_temp_free_i64(t0);
+ tcg_temp_free_i64(t1);
+ tcg_temp_free_i128(tmpv);
+ tcg_temp_free_i128(oldv);
+ }
+}
+
+void tcg_gen_atomic_cmpxchg_i128(TCGv_i128 retv, TCGv addr, TCGv_i128 cmpv,
+ TCGv_i128 newv, TCGArg idx, MemOp memop)
+{
+ gen_atomic_cx_i128 gen;
+
+ if (!(tcg_ctx->gen_tb->cflags & CF_PARALLEL)) {
+ tcg_gen_nonatomic_cmpxchg_i128(retv, addr, cmpv, newv, idx, memop);
+ return;
+ }
+
+ tcg_debug_assert((memop & MO_SIZE) == MO_128);
+ tcg_debug_assert((memop & MO_SIGN) == 0);
+ gen = table_cmpxchg[memop & (MO_SIZE | MO_BSWAP)];
+
+ if (gen) {
+ MemOpIdx oi = make_memop_idx(memop, idx);
+ gen(retv, cpu_env, addr, cmpv, newv, tcg_constant_i32(oi));
+ return;
+ }
+
+ gen_helper_exit_atomic(cpu_env);
+
+ /*
+ * Produce a result for a well-formed opcode stream. This satisfies
+ * liveness for set before used, which happens before this dead code
+ * is removed.
+ */
+ tcg_gen_movi_i64(TCGV128_LOW(retv), 0);
+ tcg_gen_movi_i64(TCGV128_HIGH(retv), 0);
+}
+
+static void do_nonatomic_op_i32(TCGv_i32 ret, TCGv addr, TCGv_i32 val,
+ TCGArg idx, MemOp memop, bool new_val,
+ void (*gen)(TCGv_i32, TCGv_i32, TCGv_i32))
+{
+ TCGv_i32 t1 = tcg_temp_ebb_new_i32();
+ TCGv_i32 t2 = tcg_temp_ebb_new_i32();
+
+ memop = tcg_canonicalize_memop(memop, 0, 0);
+
+ tcg_gen_qemu_ld_i32(t1, addr, idx, memop);
+ tcg_gen_ext_i32(t2, val, memop);
+ gen(t2, t1, t2);
+ tcg_gen_qemu_st_i32(t2, addr, idx, memop);
+
+ tcg_gen_ext_i32(ret, (new_val ? t2 : t1), memop);
+ tcg_temp_free_i32(t1);
+ tcg_temp_free_i32(t2);
+}
+
+static void do_atomic_op_i32(TCGv_i32 ret, TCGv addr, TCGv_i32 val,
+ TCGArg idx, MemOp memop, void * const table[])
+{
+ gen_atomic_op_i32 gen;
+ MemOpIdx oi;
+
+ memop = tcg_canonicalize_memop(memop, 0, 0);
+
+ gen = table[memop & (MO_SIZE | MO_BSWAP)];
+ tcg_debug_assert(gen != NULL);
+
+ oi = make_memop_idx(memop & ~MO_SIGN, idx);
+ gen(ret, cpu_env, addr, val, tcg_constant_i32(oi));
+
+ if (memop & MO_SIGN) {
+ tcg_gen_ext_i32(ret, ret, memop);
+ }
+}
+
+static void do_nonatomic_op_i64(TCGv_i64 ret, TCGv addr, TCGv_i64 val,
+ TCGArg idx, MemOp memop, bool new_val,
+ void (*gen)(TCGv_i64, TCGv_i64, TCGv_i64))
+{
+ TCGv_i64 t1 = tcg_temp_ebb_new_i64();
+ TCGv_i64 t2 = tcg_temp_ebb_new_i64();
+
+ memop = tcg_canonicalize_memop(memop, 1, 0);
+
+ tcg_gen_qemu_ld_i64(t1, addr, idx, memop);
+ tcg_gen_ext_i64(t2, val, memop);
+ gen(t2, t1, t2);
+ tcg_gen_qemu_st_i64(t2, addr, idx, memop);
+
+ tcg_gen_ext_i64(ret, (new_val ? t2 : t1), memop);
+ tcg_temp_free_i64(t1);
+ tcg_temp_free_i64(t2);
+}
+
+static void do_atomic_op_i64(TCGv_i64 ret, TCGv addr, TCGv_i64 val,
+ TCGArg idx, MemOp memop, void * const table[])
+{
+ memop = tcg_canonicalize_memop(memop, 1, 0);
+
+ if ((memop & MO_SIZE) == MO_64) {
+#ifdef CONFIG_ATOMIC64
+ gen_atomic_op_i64 gen;
+ MemOpIdx oi;
+
+ gen = table[memop & (MO_SIZE | MO_BSWAP)];
+ tcg_debug_assert(gen != NULL);
+
+ oi = make_memop_idx(memop & ~MO_SIGN, idx);
+ gen(ret, cpu_env, addr, val, tcg_constant_i32(oi));
+#else
+ gen_helper_exit_atomic(cpu_env);
+ /* Produce a result, so that we have a well-formed opcode stream
+ with respect to uses of the result in the (dead) code following. */
+ tcg_gen_movi_i64(ret, 0);
+#endif /* CONFIG_ATOMIC64 */
+ } else {
+ TCGv_i32 v32 = tcg_temp_ebb_new_i32();
+ TCGv_i32 r32 = tcg_temp_ebb_new_i32();
+
+ tcg_gen_extrl_i64_i32(v32, val);
+ do_atomic_op_i32(r32, addr, v32, idx, memop & ~MO_SIGN, table);
+ tcg_temp_free_i32(v32);
+
+ tcg_gen_extu_i32_i64(ret, r32);
+ tcg_temp_free_i32(r32);
+
+ if (memop & MO_SIGN) {
+ tcg_gen_ext_i64(ret, ret, memop);
+ }
+ }
+}
+
+#define GEN_ATOMIC_HELPER(NAME, OP, NEW) \
+static void * const table_##NAME[(MO_SIZE | MO_BSWAP) + 1] = { \
+ [MO_8] = gen_helper_atomic_##NAME##b, \
+ [MO_16 | MO_LE] = gen_helper_atomic_##NAME##w_le, \
+ [MO_16 | MO_BE] = gen_helper_atomic_##NAME##w_be, \
+ [MO_32 | MO_LE] = gen_helper_atomic_##NAME##l_le, \
+ [MO_32 | MO_BE] = gen_helper_atomic_##NAME##l_be, \
+ WITH_ATOMIC64([MO_64 | MO_LE] = gen_helper_atomic_##NAME##q_le) \
+ WITH_ATOMIC64([MO_64 | MO_BE] = gen_helper_atomic_##NAME##q_be) \
+}; \
+void tcg_gen_atomic_##NAME##_i32 \
+ (TCGv_i32 ret, TCGv addr, TCGv_i32 val, TCGArg idx, MemOp memop) \
+{ \
+ if (tcg_ctx->gen_tb->cflags & CF_PARALLEL) { \
+ do_atomic_op_i32(ret, addr, val, idx, memop, table_##NAME); \
+ } else { \
+ do_nonatomic_op_i32(ret, addr, val, idx, memop, NEW, \
+ tcg_gen_##OP##_i32); \
+ } \
+} \
+void tcg_gen_atomic_##NAME##_i64 \
+ (TCGv_i64 ret, TCGv addr, TCGv_i64 val, TCGArg idx, MemOp memop) \
+{ \
+ if (tcg_ctx->gen_tb->cflags & CF_PARALLEL) { \
+ do_atomic_op_i64(ret, addr, val, idx, memop, table_##NAME); \
+ } else { \
+ do_nonatomic_op_i64(ret, addr, val, idx, memop, NEW, \
+ tcg_gen_##OP##_i64); \
+ } \
+}
+
+GEN_ATOMIC_HELPER(fetch_add, add, 0)
+GEN_ATOMIC_HELPER(fetch_and, and, 0)
+GEN_ATOMIC_HELPER(fetch_or, or, 0)
+GEN_ATOMIC_HELPER(fetch_xor, xor, 0)
+GEN_ATOMIC_HELPER(fetch_smin, smin, 0)
+GEN_ATOMIC_HELPER(fetch_umin, umin, 0)
+GEN_ATOMIC_HELPER(fetch_smax, smax, 0)
+GEN_ATOMIC_HELPER(fetch_umax, umax, 0)
+
+GEN_ATOMIC_HELPER(add_fetch, add, 1)
+GEN_ATOMIC_HELPER(and_fetch, and, 1)
+GEN_ATOMIC_HELPER(or_fetch, or, 1)
+GEN_ATOMIC_HELPER(xor_fetch, xor, 1)
+GEN_ATOMIC_HELPER(smin_fetch, smin, 1)
+GEN_ATOMIC_HELPER(umin_fetch, umin, 1)
+GEN_ATOMIC_HELPER(smax_fetch, smax, 1)
+GEN_ATOMIC_HELPER(umax_fetch, umax, 1)
+
+static void tcg_gen_mov2_i32(TCGv_i32 r, TCGv_i32 a, TCGv_i32 b)
+{
+ tcg_gen_mov_i32(r, b);
+}
+
+static void tcg_gen_mov2_i64(TCGv_i64 r, TCGv_i64 a, TCGv_i64 b)
+{
+ tcg_gen_mov_i64(r, b);
+}
+
+GEN_ATOMIC_HELPER(xchg, mov2, 0)
+
+#undef GEN_ATOMIC_HELPER
diff --git a/tcg/tcg-op.c b/tcg/tcg-op.c
index 06d3181fd0..edbd1c61d7 100644
--- a/tcg/tcg-op.c
+++ b/tcg/tcg-op.c
@@ -27,7 +27,6 @@
#include "tcg/tcg.h"
#include "tcg/tcg-temp-internal.h"
#include "tcg/tcg-op.h"
-#include "tcg/tcg-mo.h"
#include "exec/plugin-gen.h"
#include "tcg-internal.h"
@@ -2841,987 +2840,3 @@ void tcg_gen_lookup_and_goto_ptr(void)
tcg_gen_op1i(INDEX_op_goto_ptr, tcgv_ptr_arg(ptr));
tcg_temp_free_ptr(ptr);
}
-
-static inline MemOp tcg_canonicalize_memop(MemOp op, bool is64, bool st)
-{
- /* Trigger the asserts within as early as possible. */
- unsigned a_bits = get_alignment_bits(op);
-
- /* Prefer MO_ALIGN+MO_XX over MO_ALIGN_XX+MO_XX */
- if (a_bits == (op & MO_SIZE)) {
- op = (op & ~MO_AMASK) | MO_ALIGN;
- }
-
- switch (op & MO_SIZE) {
- case MO_8:
- op &= ~MO_BSWAP;
- break;
- case MO_16:
- break;
- case MO_32:
- if (!is64) {
- op &= ~MO_SIGN;
- }
- break;
- case MO_64:
- if (is64) {
- op &= ~MO_SIGN;
- break;
- }
- /* fall through */
- default:
- g_assert_not_reached();
- }
- if (st) {
- op &= ~MO_SIGN;
- }
- return op;
-}
-
-static void gen_ldst_i32(TCGOpcode opc, TCGv_i32 val, TCGv addr,
- MemOp memop, TCGArg idx)
-{
- MemOpIdx oi = make_memop_idx(memop, idx);
-#if TARGET_LONG_BITS == 32
- tcg_gen_op3i_i32(opc, val, addr, oi);
-#else
- if (TCG_TARGET_REG_BITS == 32) {
- tcg_gen_op4i_i32(opc, val, TCGV_LOW(addr), TCGV_HIGH(addr), oi);
- } else {
- tcg_gen_op3(opc, tcgv_i32_arg(val), tcgv_i64_arg(addr), oi);
- }
-#endif
-}
-
-static void gen_ldst_i64(TCGOpcode opc, TCGv_i64 val, TCGv addr,
- MemOp memop, TCGArg idx)
-{
- MemOpIdx oi = make_memop_idx(memop, idx);
-#if TARGET_LONG_BITS == 32
- if (TCG_TARGET_REG_BITS == 32) {
- tcg_gen_op4i_i32(opc, TCGV_LOW(val), TCGV_HIGH(val), addr, oi);
- } else {
- tcg_gen_op3(opc, tcgv_i64_arg(val), tcgv_i32_arg(addr), oi);
- }
-#else
- if (TCG_TARGET_REG_BITS == 32) {
- tcg_gen_op5i_i32(opc, TCGV_LOW(val), TCGV_HIGH(val),
- TCGV_LOW(addr), TCGV_HIGH(addr), oi);
- } else {
- tcg_gen_op3i_i64(opc, val, addr, oi);
- }
-#endif
-}
-
-static void tcg_gen_req_mo(TCGBar type)
-{
-#ifdef TCG_GUEST_DEFAULT_MO
- type &= TCG_GUEST_DEFAULT_MO;
-#endif
- type &= ~TCG_TARGET_DEFAULT_MO;
- if (type) {
- tcg_gen_mb(type | TCG_BAR_SC);
- }
-}
-
-static inline TCGv plugin_prep_mem_callbacks(TCGv vaddr)
-{
-#ifdef CONFIG_PLUGIN
- if (tcg_ctx->plugin_insn != NULL) {
- /* Save a copy of the vaddr for use after a load. */
- TCGv temp = tcg_temp_new();
- tcg_gen_mov_tl(temp, vaddr);
- return temp;
- }
-#endif
- return vaddr;
-}
-
-static void plugin_gen_mem_callbacks(TCGv vaddr, MemOpIdx oi,
- enum qemu_plugin_mem_rw rw)
-{
-#ifdef CONFIG_PLUGIN
- if (tcg_ctx->plugin_insn != NULL) {
- qemu_plugin_meminfo_t info = make_plugin_meminfo(oi, rw);
- plugin_gen_empty_mem_callback(vaddr, info);
- tcg_temp_free(vaddr);
- }
-#endif
-}
-
-void tcg_gen_qemu_ld_i32(TCGv_i32 val, TCGv addr, TCGArg idx, MemOp memop)
-{
- MemOp orig_memop;
- MemOpIdx oi;
-
- tcg_gen_req_mo(TCG_MO_LD_LD | TCG_MO_ST_LD);
- memop = tcg_canonicalize_memop(memop, 0, 0);
- oi = make_memop_idx(memop, idx);
-
- orig_memop = memop;
- if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) {
- memop &= ~MO_BSWAP;
- /* The bswap primitive benefits from zero-extended input. */
- if ((memop & MO_SSIZE) == MO_SW) {
- memop &= ~MO_SIGN;
- }
- }
-
- addr = plugin_prep_mem_callbacks(addr);
- gen_ldst_i32(INDEX_op_qemu_ld_i32, val, addr, memop, idx);
- plugin_gen_mem_callbacks(addr, oi, QEMU_PLUGIN_MEM_R);
-
- if ((orig_memop ^ memop) & MO_BSWAP) {
- switch (orig_memop & MO_SIZE) {
- case MO_16:
- tcg_gen_bswap16_i32(val, val, (orig_memop & MO_SIGN
- ? TCG_BSWAP_IZ | TCG_BSWAP_OS
- : TCG_BSWAP_IZ | TCG_BSWAP_OZ));
- break;
- case MO_32:
- tcg_gen_bswap32_i32(val, val);
- break;
- default:
- g_assert_not_reached();
- }
- }
-}
-
-void tcg_gen_qemu_st_i32(TCGv_i32 val, TCGv addr, TCGArg idx, MemOp memop)
-{
- TCGv_i32 swap = NULL;
- MemOpIdx oi;
-
- tcg_gen_req_mo(TCG_MO_LD_ST | TCG_MO_ST_ST);
- memop = tcg_canonicalize_memop(memop, 0, 1);
- oi = make_memop_idx(memop, idx);
-
- if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) {
- swap = tcg_temp_ebb_new_i32();
- switch (memop & MO_SIZE) {
- case MO_16:
- tcg_gen_bswap16_i32(swap, val, 0);
- break;
- case MO_32:
- tcg_gen_bswap32_i32(swap, val);
- break;
- default:
- g_assert_not_reached();
- }
- val = swap;
- memop &= ~MO_BSWAP;
- }
-
- addr = plugin_prep_mem_callbacks(addr);
- if (TCG_TARGET_HAS_qemu_st8_i32 && (memop & MO_SIZE) == MO_8) {
- gen_ldst_i32(INDEX_op_qemu_st8_i32, val, addr, memop, idx);
- } else {
- gen_ldst_i32(INDEX_op_qemu_st_i32, val, addr, memop, idx);
- }
- plugin_gen_mem_callbacks(addr, oi, QEMU_PLUGIN_MEM_W);
-
- if (swap) {
- tcg_temp_free_i32(swap);
- }
-}
-
-void tcg_gen_qemu_ld_i64(TCGv_i64 val, TCGv addr, TCGArg idx, MemOp memop)
-{
- MemOp orig_memop;
- MemOpIdx oi;
-
- if (TCG_TARGET_REG_BITS == 32 && (memop & MO_SIZE) < MO_64) {
- tcg_gen_qemu_ld_i32(TCGV_LOW(val), addr, idx, memop);
- if (memop & MO_SIGN) {
- tcg_gen_sari_i32(TCGV_HIGH(val), TCGV_LOW(val), 31);
- } else {
- tcg_gen_movi_i32(TCGV_HIGH(val), 0);
- }
- return;
- }
-
- tcg_gen_req_mo(TCG_MO_LD_LD | TCG_MO_ST_LD);
- memop = tcg_canonicalize_memop(memop, 1, 0);
- oi = make_memop_idx(memop, idx);
-
- orig_memop = memop;
- if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) {
- memop &= ~MO_BSWAP;
- /* The bswap primitive benefits from zero-extended input. */
- if ((memop & MO_SIGN) && (memop & MO_SIZE) < MO_64) {
- memop &= ~MO_SIGN;
- }
- }
-
- addr = plugin_prep_mem_callbacks(addr);
- gen_ldst_i64(INDEX_op_qemu_ld_i64, val, addr, memop, idx);
- plugin_gen_mem_callbacks(addr, oi, QEMU_PLUGIN_MEM_R);
-
- if ((orig_memop ^ memop) & MO_BSWAP) {
- int flags = (orig_memop & MO_SIGN
- ? TCG_BSWAP_IZ | TCG_BSWAP_OS
- : TCG_BSWAP_IZ | TCG_BSWAP_OZ);
- switch (orig_memop & MO_SIZE) {
- case MO_16:
- tcg_gen_bswap16_i64(val, val, flags);
- break;
- case MO_32:
- tcg_gen_bswap32_i64(val, val, flags);
- break;
- case MO_64:
- tcg_gen_bswap64_i64(val, val);
- break;
- default:
- g_assert_not_reached();
- }
- }
-}
-
-void tcg_gen_qemu_st_i64(TCGv_i64 val, TCGv addr, TCGArg idx, MemOp memop)
-{
- TCGv_i64 swap = NULL;
- MemOpIdx oi;
-
- if (TCG_TARGET_REG_BITS == 32 && (memop & MO_SIZE) < MO_64) {
- tcg_gen_qemu_st_i32(TCGV_LOW(val), addr, idx, memop);
- return;
- }
-
- tcg_gen_req_mo(TCG_MO_LD_ST | TCG_MO_ST_ST);
- memop = tcg_canonicalize_memop(memop, 1, 1);
- oi = make_memop_idx(memop, idx);
-
- if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) {
- swap = tcg_temp_ebb_new_i64();
- switch (memop & MO_SIZE) {
- case MO_16:
- tcg_gen_bswap16_i64(swap, val, 0);
- break;
- case MO_32:
- tcg_gen_bswap32_i64(swap, val, 0);
- break;
- case MO_64:
- tcg_gen_bswap64_i64(swap, val);
- break;
- default:
- g_assert_not_reached();
- }
- val = swap;
- memop &= ~MO_BSWAP;
- }
-
- addr = plugin_prep_mem_callbacks(addr);
- gen_ldst_i64(INDEX_op_qemu_st_i64, val, addr, memop, idx);
- plugin_gen_mem_callbacks(addr, oi, QEMU_PLUGIN_MEM_W);
-
- if (swap) {
- tcg_temp_free_i64(swap);
- }
-}
-
-/*
- * Return true if @mop, without knowledge of the pointer alignment,
- * does not require 16-byte atomicity, and it would be adventagous
- * to avoid a call to a helper function.
- */
-static bool use_two_i64_for_i128(MemOp mop)
-{
-#ifdef CONFIG_SOFTMMU
- /* Two softmmu tlb lookups is larger than one function call. */
- return false;
-#else
- /*
- * For user-only, two 64-bit operations may well be smaller than a call.
- * Determine if that would be legal for the requested atomicity.
- */
- MemOp atom = mop & MO_ATOM_MASK;
- MemOp atmax = mop & MO_ATMAX_MASK;
-
- /* In a serialized context, no atomicity is required. */
- if (!(tcg_ctx->gen_tb->cflags & CF_PARALLEL)) {
- return true;
- }
-
- if (atmax == MO_ATMAX_SIZE) {
- atmax = mop & MO_SIZE;
- } else {
- atmax >>= MO_ATMAX_SHIFT;
- }
- switch (atom) {
- case MO_ATOM_NONE:
- return true;
- case MO_ATOM_IFALIGN:
- case MO_ATOM_SUBALIGN:
- return atmax < MO_128;
- case MO_ATOM_WITHIN16:
- return atmax == MO_8;
- default:
- g_assert_not_reached();
- }
-#endif
-}
-
-static void canonicalize_memop_i128_as_i64(MemOp ret[2], MemOp orig)
-{
- MemOp mop_1 = orig, mop_2;
-
- tcg_debug_assert((orig & MO_SIZE) == MO_128);
- tcg_debug_assert((orig & MO_SIGN) == 0);
-
- /* Reduce the size to 64-bit. */
- mop_1 = (mop_1 & ~MO_SIZE) | MO_64;
-
- /* Retain the alignment constraints of the original. */
- switch (orig & MO_AMASK) {
- case MO_UNALN:
- case MO_ALIGN_2:
- case MO_ALIGN_4:
- mop_2 = mop_1;
- break;
- case MO_ALIGN_8:
- /* Prefer MO_ALIGN+MO_64 to MO_ALIGN_8+MO_64. */
- mop_1 = (mop_1 & ~MO_AMASK) | MO_ALIGN;
- mop_2 = mop_1;
- break;
- case MO_ALIGN:
- /* Second has 8-byte alignment; first has 16-byte alignment. */
- mop_2 = mop_1;
- mop_1 = (mop_1 & ~MO_AMASK) | MO_ALIGN_16;
- break;
- case MO_ALIGN_16:
- case MO_ALIGN_32:
- case MO_ALIGN_64:
- /* Second has 8-byte alignment; first retains original. */
- mop_2 = (mop_1 & ~MO_AMASK) | MO_ALIGN;
- break;
- default:
- g_assert_not_reached();
- }
-
- /* Use a memory ordering implemented by the host. */
- if ((orig & MO_BSWAP) && !tcg_target_has_memory_bswap(mop_1)) {
- mop_1 &= ~MO_BSWAP;
- mop_2 &= ~MO_BSWAP;
- }
-
- ret[0] = mop_1;
- ret[1] = mop_2;
-}
-
-#if TARGET_LONG_BITS == 64
-#define tcg_temp_ebb_new tcg_temp_ebb_new_i64
-#else
-#define tcg_temp_ebb_new tcg_temp_ebb_new_i32
-#endif
-
-void tcg_gen_qemu_ld_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop)
-{
- const MemOpIdx oi = make_memop_idx(memop, idx);
-
- tcg_debug_assert((memop & MO_SIZE) == MO_128);
- tcg_debug_assert((memop & MO_SIGN) == 0);
-
- tcg_gen_req_mo(TCG_MO_LD_LD | TCG_MO_ST_LD);
- addr = plugin_prep_mem_callbacks(addr);
-
- /* TODO: For now, force 32-bit hosts to use the helper. */
- if (TCG_TARGET_HAS_qemu_ldst_i128 && TCG_TARGET_REG_BITS == 64) {
- TCGv_i64 lo, hi;
- TCGArg addr_arg;
- MemOpIdx adj_oi;
- bool need_bswap = false;
-
- if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) {
- lo = TCGV128_HIGH(val);
- hi = TCGV128_LOW(val);
- adj_oi = make_memop_idx(memop & ~MO_BSWAP, idx);
- need_bswap = true;
- } else {
- lo = TCGV128_LOW(val);
- hi = TCGV128_HIGH(val);
- adj_oi = oi;
- }
-
-#if TARGET_LONG_BITS == 32
- addr_arg = tcgv_i32_arg(addr);
-#else
- addr_arg = tcgv_i64_arg(addr);
-#endif
- tcg_gen_op4ii_i64(INDEX_op_qemu_ld_i128, lo, hi, addr_arg, adj_oi);
-
- if (need_bswap) {
- tcg_gen_bswap64_i64(lo, lo);
- tcg_gen_bswap64_i64(hi, hi);
- }
- } else if (use_two_i64_for_i128(memop)) {
- MemOp mop[2];
- TCGv addr_p8;
- TCGv_i64 x, y;
-
- canonicalize_memop_i128_as_i64(mop, memop);
-
- /*
- * Since there are no global TCGv_i128, there is no visible state
- * changed if the second load faults. Load directly into the two
- * subwords.
- */
- if ((memop & MO_BSWAP) == MO_LE) {
- x = TCGV128_LOW(val);
- y = TCGV128_HIGH(val);
- } else {
- x = TCGV128_HIGH(val);
- y = TCGV128_LOW(val);
- }
-
- gen_ldst_i64(INDEX_op_qemu_ld_i64, x, addr, mop[0], idx);
-
- if ((mop[0] ^ memop) & MO_BSWAP) {
- tcg_gen_bswap64_i64(x, x);
- }
-
- addr_p8 = tcg_temp_ebb_new();
- tcg_gen_addi_tl(addr_p8, addr, 8);
- gen_ldst_i64(INDEX_op_qemu_ld_i64, y, addr_p8, mop[1], idx);
- tcg_temp_free(addr_p8);
-
- if ((mop[0] ^ memop) & MO_BSWAP) {
- tcg_gen_bswap64_i64(y, y);
- }
- } else {
- gen_helper_ld_i128(val, cpu_env, addr, tcg_constant_i32(oi));
- }
-
- plugin_gen_mem_callbacks(addr, oi, QEMU_PLUGIN_MEM_R);
-}
-
-void tcg_gen_qemu_st_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop)
-{
- const MemOpIdx oi = make_memop_idx(memop, idx);
-
- tcg_debug_assert((memop & MO_SIZE) == MO_128);
- tcg_debug_assert((memop & MO_SIGN) == 0);
-
- tcg_gen_req_mo(TCG_MO_ST_LD | TCG_MO_ST_ST);
- addr = plugin_prep_mem_callbacks(addr);
-
- /* TODO: For now, force 32-bit hosts to use the helper. */
-
- if (TCG_TARGET_HAS_qemu_ldst_i128 && TCG_TARGET_REG_BITS == 64) {
- TCGv_i64 lo, hi;
- TCGArg addr_arg;
- MemOpIdx adj_oi;
- bool need_bswap = false;
-
- if ((memop & MO_BSWAP) && !tcg_target_has_memory_bswap(memop)) {
- lo = tcg_temp_new_i64();
- hi = tcg_temp_new_i64();
- tcg_gen_bswap64_i64(lo, TCGV128_HIGH(val));
- tcg_gen_bswap64_i64(hi, TCGV128_LOW(val));
- adj_oi = make_memop_idx(memop & ~MO_BSWAP, idx);
- need_bswap = true;
- } else {
- lo = TCGV128_LOW(val);
- hi = TCGV128_HIGH(val);
- adj_oi = oi;
- }
-
-#if TARGET_LONG_BITS == 32
- addr_arg = tcgv_i32_arg(addr);
-#else
- addr_arg = tcgv_i64_arg(addr);
-#endif
- tcg_gen_op4ii_i64(INDEX_op_qemu_st_i128, lo, hi, addr_arg, adj_oi);
-
- if (need_bswap) {
- tcg_temp_free_i64(lo);
- tcg_temp_free_i64(hi);
- }
- } else if (use_two_i64_for_i128(memop)) {
- MemOp mop[2];
- TCGv addr_p8;
- TCGv_i64 x, y;
-
- canonicalize_memop_i128_as_i64(mop, memop);
-
- if ((memop & MO_BSWAP) == MO_LE) {
- x = TCGV128_LOW(val);
- y = TCGV128_HIGH(val);
- } else {
- x = TCGV128_HIGH(val);
- y = TCGV128_LOW(val);
- }
-
- addr_p8 = tcg_temp_ebb_new();
- if ((mop[0] ^ memop) & MO_BSWAP) {
- TCGv_i64 t = tcg_temp_ebb_new_i64();
-
- tcg_gen_bswap64_i64(t, x);
- gen_ldst_i64(INDEX_op_qemu_st_i64, t, addr, mop[0], idx);
- tcg_gen_bswap64_i64(t, y);
- tcg_gen_addi_tl(addr_p8, addr, 8);
- gen_ldst_i64(INDEX_op_qemu_st_i64, t, addr_p8, mop[1], idx);
- tcg_temp_free_i64(t);
- } else {
- gen_ldst_i64(INDEX_op_qemu_st_i64, x, addr, mop[0], idx);
- tcg_gen_addi_tl(addr_p8, addr, 8);
- gen_ldst_i64(INDEX_op_qemu_st_i64, y, addr_p8, mop[1], idx);
- }
- tcg_temp_free(addr_p8);
- } else {
- gen_helper_st_i128(cpu_env, addr, val, tcg_constant_i32(oi));
- }
-
- plugin_gen_mem_callbacks(addr, oi, QEMU_PLUGIN_MEM_W);
-}
-
-static void tcg_gen_ext_i32(TCGv_i32 ret, TCGv_i32 val, MemOp opc)
-{
- switch (opc & MO_SSIZE) {
- case MO_SB:
- tcg_gen_ext8s_i32(ret, val);
- break;
- case MO_UB:
- tcg_gen_ext8u_i32(ret, val);
- break;
- case MO_SW:
- tcg_gen_ext16s_i32(ret, val);
- break;
- case MO_UW:
- tcg_gen_ext16u_i32(ret, val);
- break;
- default:
- tcg_gen_mov_i32(ret, val);
- break;
- }
-}
-
-static void tcg_gen_ext_i64(TCGv_i64 ret, TCGv_i64 val, MemOp opc)
-{
- switch (opc & MO_SSIZE) {
- case MO_SB:
- tcg_gen_ext8s_i64(ret, val);
- break;
- case MO_UB:
- tcg_gen_ext8u_i64(ret, val);
- break;
- case MO_SW:
- tcg_gen_ext16s_i64(ret, val);
- break;
- case MO_UW:
- tcg_gen_ext16u_i64(ret, val);
- break;
- case MO_SL:
- tcg_gen_ext32s_i64(ret, val);
- break;
- case MO_UL:
- tcg_gen_ext32u_i64(ret, val);
- break;
- default:
- tcg_gen_mov_i64(ret, val);
- break;
- }
-}
-
-typedef void (*gen_atomic_cx_i32)(TCGv_i32, TCGv_env, TCGv,
- TCGv_i32, TCGv_i32, TCGv_i32);
-typedef void (*gen_atomic_cx_i64)(TCGv_i64, TCGv_env, TCGv,
- TCGv_i64, TCGv_i64, TCGv_i32);
-typedef void (*gen_atomic_cx_i128)(TCGv_i128, TCGv_env, TCGv,
- TCGv_i128, TCGv_i128, TCGv_i32);
-typedef void (*gen_atomic_op_i32)(TCGv_i32, TCGv_env, TCGv,
- TCGv_i32, TCGv_i32);
-typedef void (*gen_atomic_op_i64)(TCGv_i64, TCGv_env, TCGv,
- TCGv_i64, TCGv_i32);
-
-#ifdef CONFIG_ATOMIC64
-# define WITH_ATOMIC64(X) X,
-#else
-# define WITH_ATOMIC64(X)
-#endif
-#ifdef CONFIG_CMPXCHG128
-# define WITH_ATOMIC128(X) X,
-#else
-# define WITH_ATOMIC128(X)
-#endif
-
-static void * const table_cmpxchg[(MO_SIZE | MO_BSWAP) + 1] = {
- [MO_8] = gen_helper_atomic_cmpxchgb,
- [MO_16 | MO_LE] = gen_helper_atomic_cmpxchgw_le,
- [MO_16 | MO_BE] = gen_helper_atomic_cmpxchgw_be,
- [MO_32 | MO_LE] = gen_helper_atomic_cmpxchgl_le,
- [MO_32 | MO_BE] = gen_helper_atomic_cmpxchgl_be,
- WITH_ATOMIC64([MO_64 | MO_LE] = gen_helper_atomic_cmpxchgq_le)
- WITH_ATOMIC64([MO_64 | MO_BE] = gen_helper_atomic_cmpxchgq_be)
- WITH_ATOMIC128([MO_128 | MO_LE] = gen_helper_atomic_cmpxchgo_le)
- WITH_ATOMIC128([MO_128 | MO_BE] = gen_helper_atomic_cmpxchgo_be)
-};
-
-void tcg_gen_nonatomic_cmpxchg_i32(TCGv_i32 retv, TCGv addr, TCGv_i32 cmpv,
- TCGv_i32 newv, TCGArg idx, MemOp memop)
-{
- TCGv_i32 t1 = tcg_temp_ebb_new_i32();
- TCGv_i32 t2 = tcg_temp_ebb_new_i32();
-
- tcg_gen_ext_i32(t2, cmpv, memop & MO_SIZE);
-
- tcg_gen_qemu_ld_i32(t1, addr, idx, memop & ~MO_SIGN);
- tcg_gen_movcond_i32(TCG_COND_EQ, t2, t1, t2, newv, t1);
- tcg_gen_qemu_st_i32(t2, addr, idx, memop);
- tcg_temp_free_i32(t2);
-
- if (memop & MO_SIGN) {
- tcg_gen_ext_i32(retv, t1, memop);
- } else {
- tcg_gen_mov_i32(retv, t1);
- }
- tcg_temp_free_i32(t1);
-}
-
-void tcg_gen_atomic_cmpxchg_i32(TCGv_i32 retv, TCGv addr, TCGv_i32 cmpv,
- TCGv_i32 newv, TCGArg idx, MemOp memop)
-{
- gen_atomic_cx_i32 gen;
- MemOpIdx oi;
-
- if (!(tcg_ctx->gen_tb->cflags & CF_PARALLEL)) {
- tcg_gen_nonatomic_cmpxchg_i32(retv, addr, cmpv, newv, idx, memop);
- return;
- }
-
- memop = tcg_canonicalize_memop(memop, 0, 0);
- gen = table_cmpxchg[memop & (MO_SIZE | MO_BSWAP)];
- tcg_debug_assert(gen != NULL);
-
- oi = make_memop_idx(memop & ~MO_SIGN, idx);
- gen(retv, cpu_env, addr, cmpv, newv, tcg_constant_i32(oi));
-
- if (memop & MO_SIGN) {
- tcg_gen_ext_i32(retv, retv, memop);
- }
-}
-
-void tcg_gen_nonatomic_cmpxchg_i64(TCGv_i64 retv, TCGv addr, TCGv_i64 cmpv,
- TCGv_i64 newv, TCGArg idx, MemOp memop)
-{
- TCGv_i64 t1, t2;
-
- if (TCG_TARGET_REG_BITS == 32 && (memop & MO_SIZE) < MO_64) {
- tcg_gen_nonatomic_cmpxchg_i32(TCGV_LOW(retv), addr, TCGV_LOW(cmpv),
- TCGV_LOW(newv), idx, memop);
- if (memop & MO_SIGN) {
- tcg_gen_sari_i32(TCGV_HIGH(retv), TCGV_LOW(retv), 31);
- } else {
- tcg_gen_movi_i32(TCGV_HIGH(retv), 0);
- }
- return;
- }
-
- t1 = tcg_temp_ebb_new_i64();
- t2 = tcg_temp_ebb_new_i64();
-
- tcg_gen_ext_i64(t2, cmpv, memop & MO_SIZE);
-
- tcg_gen_qemu_ld_i64(t1, addr, idx, memop & ~MO_SIGN);
- tcg_gen_movcond_i64(TCG_COND_EQ, t2, t1, t2, newv, t1);
- tcg_gen_qemu_st_i64(t2, addr, idx, memop);
- tcg_temp_free_i64(t2);
-
- if (memop & MO_SIGN) {
- tcg_gen_ext_i64(retv, t1, memop);
- } else {
- tcg_gen_mov_i64(retv, t1);
- }
- tcg_temp_free_i64(t1);
-}
-
-void tcg_gen_atomic_cmpxchg_i64(TCGv_i64 retv, TCGv addr, TCGv_i64 cmpv,
- TCGv_i64 newv, TCGArg idx, MemOp memop)
-{
- if (!(tcg_ctx->gen_tb->cflags & CF_PARALLEL)) {
- tcg_gen_nonatomic_cmpxchg_i64(retv, addr, cmpv, newv, idx, memop);
- return;
- }
-
- if ((memop & MO_SIZE) == MO_64) {
- gen_atomic_cx_i64 gen;
-
- memop = tcg_canonicalize_memop(memop, 1, 0);
- gen = table_cmpxchg[memop & (MO_SIZE | MO_BSWAP)];
- if (gen) {
- MemOpIdx oi = make_memop_idx(memop, idx);
- gen(retv, cpu_env, addr, cmpv, newv, tcg_constant_i32(oi));
- return;
- }
-
- gen_helper_exit_atomic(cpu_env);
-
- /*
- * Produce a result for a well-formed opcode stream. This satisfies
- * liveness for set before used, which happens before this dead code
- * is removed.
- */
- tcg_gen_movi_i64(retv, 0);
- return;
- }
-
- if (TCG_TARGET_REG_BITS == 32) {
- tcg_gen_atomic_cmpxchg_i32(TCGV_LOW(retv), addr, TCGV_LOW(cmpv),
- TCGV_LOW(newv), idx, memop);
- if (memop & MO_SIGN) {
- tcg_gen_sari_i32(TCGV_HIGH(retv), TCGV_LOW(retv), 31);
- } else {
- tcg_gen_movi_i32(TCGV_HIGH(retv), 0);
- }
- } else {
- TCGv_i32 c32 = tcg_temp_ebb_new_i32();
- TCGv_i32 n32 = tcg_temp_ebb_new_i32();
- TCGv_i32 r32 = tcg_temp_ebb_new_i32();
-
- tcg_gen_extrl_i64_i32(c32, cmpv);
- tcg_gen_extrl_i64_i32(n32, newv);
- tcg_gen_atomic_cmpxchg_i32(r32, addr, c32, n32, idx, memop & ~MO_SIGN);
- tcg_temp_free_i32(c32);
- tcg_temp_free_i32(n32);
-
- tcg_gen_extu_i32_i64(retv, r32);
- tcg_temp_free_i32(r32);
-
- if (memop & MO_SIGN) {
- tcg_gen_ext_i64(retv, retv, memop);
- }
- }
-}
-
-void tcg_gen_nonatomic_cmpxchg_i128(TCGv_i128 retv, TCGv addr, TCGv_i128 cmpv,
- TCGv_i128 newv, TCGArg idx, MemOp memop)
-{
- if (TCG_TARGET_REG_BITS == 32) {
- /* Inline expansion below is simply too large for 32-bit hosts. */
- gen_atomic_cx_i128 gen = ((memop & MO_BSWAP) == MO_LE
- ? gen_helper_nonatomic_cmpxchgo_le
- : gen_helper_nonatomic_cmpxchgo_be);
- MemOpIdx oi = make_memop_idx(memop, idx);
-
- tcg_debug_assert((memop & MO_SIZE) == MO_128);
- tcg_debug_assert((memop & MO_SIGN) == 0);
-
- gen(retv, cpu_env, addr, cmpv, newv, tcg_constant_i32(oi));
- } else {
- TCGv_i128 oldv = tcg_temp_ebb_new_i128();
- TCGv_i128 tmpv = tcg_temp_ebb_new_i128();
- TCGv_i64 t0 = tcg_temp_ebb_new_i64();
- TCGv_i64 t1 = tcg_temp_ebb_new_i64();
- TCGv_i64 z = tcg_constant_i64(0);
-
- tcg_gen_qemu_ld_i128(oldv, addr, idx, memop);
-
- /* Compare i128 */
- tcg_gen_xor_i64(t0, TCGV128_LOW(oldv), TCGV128_LOW(cmpv));
- tcg_gen_xor_i64(t1, TCGV128_HIGH(oldv), TCGV128_HIGH(cmpv));
- tcg_gen_or_i64(t0, t0, t1);
-
- /* tmpv = equal ? newv : oldv */
- tcg_gen_movcond_i64(TCG_COND_EQ, TCGV128_LOW(tmpv), t0, z,
- TCGV128_LOW(newv), TCGV128_LOW(oldv));
- tcg_gen_movcond_i64(TCG_COND_EQ, TCGV128_HIGH(tmpv), t0, z,
- TCGV128_HIGH(newv), TCGV128_HIGH(oldv));
-
- /* Unconditional writeback. */
- tcg_gen_qemu_st_i128(tmpv, addr, idx, memop);
- tcg_gen_mov_i128(retv, oldv);
-
- tcg_temp_free_i64(t0);
- tcg_temp_free_i64(t1);
- tcg_temp_free_i128(tmpv);
- tcg_temp_free_i128(oldv);
- }
-}
-
-void tcg_gen_atomic_cmpxchg_i128(TCGv_i128 retv, TCGv addr, TCGv_i128 cmpv,
- TCGv_i128 newv, TCGArg idx, MemOp memop)
-{
- gen_atomic_cx_i128 gen;
-
- if (!(tcg_ctx->gen_tb->cflags & CF_PARALLEL)) {
- tcg_gen_nonatomic_cmpxchg_i128(retv, addr, cmpv, newv, idx, memop);
- return;
- }
-
- tcg_debug_assert((memop & MO_SIZE) == MO_128);
- tcg_debug_assert((memop & MO_SIGN) == 0);
- gen = table_cmpxchg[memop & (MO_SIZE | MO_BSWAP)];
-
- if (gen) {
- MemOpIdx oi = make_memop_idx(memop, idx);
- gen(retv, cpu_env, addr, cmpv, newv, tcg_constant_i32(oi));
- return;
- }
-
- gen_helper_exit_atomic(cpu_env);
-
- /*
- * Produce a result for a well-formed opcode stream. This satisfies
- * liveness for set before used, which happens before this dead code
- * is removed.
- */
- tcg_gen_movi_i64(TCGV128_LOW(retv), 0);
- tcg_gen_movi_i64(TCGV128_HIGH(retv), 0);
-}
-
-static void do_nonatomic_op_i32(TCGv_i32 ret, TCGv addr, TCGv_i32 val,
- TCGArg idx, MemOp memop, bool new_val,
- void (*gen)(TCGv_i32, TCGv_i32, TCGv_i32))
-{
- TCGv_i32 t1 = tcg_temp_ebb_new_i32();
- TCGv_i32 t2 = tcg_temp_ebb_new_i32();
-
- memop = tcg_canonicalize_memop(memop, 0, 0);
-
- tcg_gen_qemu_ld_i32(t1, addr, idx, memop);
- tcg_gen_ext_i32(t2, val, memop);
- gen(t2, t1, t2);
- tcg_gen_qemu_st_i32(t2, addr, idx, memop);
-
- tcg_gen_ext_i32(ret, (new_val ? t2 : t1), memop);
- tcg_temp_free_i32(t1);
- tcg_temp_free_i32(t2);
-}
-
-static void do_atomic_op_i32(TCGv_i32 ret, TCGv addr, TCGv_i32 val,
- TCGArg idx, MemOp memop, void * const table[])
-{
- gen_atomic_op_i32 gen;
- MemOpIdx oi;
-
- memop = tcg_canonicalize_memop(memop, 0, 0);
-
- gen = table[memop & (MO_SIZE | MO_BSWAP)];
- tcg_debug_assert(gen != NULL);
-
- oi = make_memop_idx(memop & ~MO_SIGN, idx);
- gen(ret, cpu_env, addr, val, tcg_constant_i32(oi));
-
- if (memop & MO_SIGN) {
- tcg_gen_ext_i32(ret, ret, memop);
- }
-}
-
-static void do_nonatomic_op_i64(TCGv_i64 ret, TCGv addr, TCGv_i64 val,
- TCGArg idx, MemOp memop, bool new_val,
- void (*gen)(TCGv_i64, TCGv_i64, TCGv_i64))
-{
- TCGv_i64 t1 = tcg_temp_ebb_new_i64();
- TCGv_i64 t2 = tcg_temp_ebb_new_i64();
-
- memop = tcg_canonicalize_memop(memop, 1, 0);
-
- tcg_gen_qemu_ld_i64(t1, addr, idx, memop);
- tcg_gen_ext_i64(t2, val, memop);
- gen(t2, t1, t2);
- tcg_gen_qemu_st_i64(t2, addr, idx, memop);
-
- tcg_gen_ext_i64(ret, (new_val ? t2 : t1), memop);
- tcg_temp_free_i64(t1);
- tcg_temp_free_i64(t2);
-}
-
-static void do_atomic_op_i64(TCGv_i64 ret, TCGv addr, TCGv_i64 val,
- TCGArg idx, MemOp memop, void * const table[])
-{
- memop = tcg_canonicalize_memop(memop, 1, 0);
-
- if ((memop & MO_SIZE) == MO_64) {
-#ifdef CONFIG_ATOMIC64
- gen_atomic_op_i64 gen;
- MemOpIdx oi;
-
- gen = table[memop & (MO_SIZE | MO_BSWAP)];
- tcg_debug_assert(gen != NULL);
-
- oi = make_memop_idx(memop & ~MO_SIGN, idx);
- gen(ret, cpu_env, addr, val, tcg_constant_i32(oi));
-#else
- gen_helper_exit_atomic(cpu_env);
- /* Produce a result, so that we have a well-formed opcode stream
- with respect to uses of the result in the (dead) code following. */
- tcg_gen_movi_i64(ret, 0);
-#endif /* CONFIG_ATOMIC64 */
- } else {
- TCGv_i32 v32 = tcg_temp_ebb_new_i32();
- TCGv_i32 r32 = tcg_temp_ebb_new_i32();
-
- tcg_gen_extrl_i64_i32(v32, val);
- do_atomic_op_i32(r32, addr, v32, idx, memop & ~MO_SIGN, table);
- tcg_temp_free_i32(v32);
-
- tcg_gen_extu_i32_i64(ret, r32);
- tcg_temp_free_i32(r32);
-
- if (memop & MO_SIGN) {
- tcg_gen_ext_i64(ret, ret, memop);
- }
- }
-}
-
-#define GEN_ATOMIC_HELPER(NAME, OP, NEW) \
-static void * const table_##NAME[(MO_SIZE | MO_BSWAP) + 1] = { \
- [MO_8] = gen_helper_atomic_##NAME##b, \
- [MO_16 | MO_LE] = gen_helper_atomic_##NAME##w_le, \
- [MO_16 | MO_BE] = gen_helper_atomic_##NAME##w_be, \
- [MO_32 | MO_LE] = gen_helper_atomic_##NAME##l_le, \
- [MO_32 | MO_BE] = gen_helper_atomic_##NAME##l_be, \
- WITH_ATOMIC64([MO_64 | MO_LE] = gen_helper_atomic_##NAME##q_le) \
- WITH_ATOMIC64([MO_64 | MO_BE] = gen_helper_atomic_##NAME##q_be) \
-}; \
-void tcg_gen_atomic_##NAME##_i32 \
- (TCGv_i32 ret, TCGv addr, TCGv_i32 val, TCGArg idx, MemOp memop) \
-{ \
- if (tcg_ctx->gen_tb->cflags & CF_PARALLEL) { \
- do_atomic_op_i32(ret, addr, val, idx, memop, table_##NAME); \
- } else { \
- do_nonatomic_op_i32(ret, addr, val, idx, memop, NEW, \
- tcg_gen_##OP##_i32); \
- } \
-} \
-void tcg_gen_atomic_##NAME##_i64 \
- (TCGv_i64 ret, TCGv addr, TCGv_i64 val, TCGArg idx, MemOp memop) \
-{ \
- if (tcg_ctx->gen_tb->cflags & CF_PARALLEL) { \
- do_atomic_op_i64(ret, addr, val, idx, memop, table_##NAME); \
- } else { \
- do_nonatomic_op_i64(ret, addr, val, idx, memop, NEW, \
- tcg_gen_##OP##_i64); \
- } \
-}
-
-GEN_ATOMIC_HELPER(fetch_add, add, 0)
-GEN_ATOMIC_HELPER(fetch_and, and, 0)
-GEN_ATOMIC_HELPER(fetch_or, or, 0)
-GEN_ATOMIC_HELPER(fetch_xor, xor, 0)
-GEN_ATOMIC_HELPER(fetch_smin, smin, 0)
-GEN_ATOMIC_HELPER(fetch_umin, umin, 0)
-GEN_ATOMIC_HELPER(fetch_smax, smax, 0)
-GEN_ATOMIC_HELPER(fetch_umax, umax, 0)
-
-GEN_ATOMIC_HELPER(add_fetch, add, 1)
-GEN_ATOMIC_HELPER(and_fetch, and, 1)
-GEN_ATOMIC_HELPER(or_fetch, or, 1)
-GEN_ATOMIC_HELPER(xor_fetch, xor, 1)
-GEN_ATOMIC_HELPER(smin_fetch, smin, 1)
-GEN_ATOMIC_HELPER(umin_fetch, umin, 1)
-GEN_ATOMIC_HELPER(smax_fetch, smax, 1)
-GEN_ATOMIC_HELPER(umax_fetch, umax, 1)
-
-static void tcg_gen_mov2_i32(TCGv_i32 r, TCGv_i32 a, TCGv_i32 b)
-{
- tcg_gen_mov_i32(r, b);
-}
-
-static void tcg_gen_mov2_i64(TCGv_i64 r, TCGv_i64 a, TCGv_i64 b)
-{
- tcg_gen_mov_i64(r, b);
-}
-
-GEN_ATOMIC_HELPER(xchg, mov2, 0)
-
-#undef GEN_ATOMIC_HELPER
diff --git a/tcg/meson.build b/tcg/meson.build
index c4c63b19d4..f56c465f4d 100644
--- a/tcg/meson.build
+++ b/tcg/meson.build
@@ -6,6 +6,7 @@ tcg_ss.add(files(
'tcg.c',
'tcg-common.c',
'tcg-op.c',
+ 'tcg-op-ldst.c',
'tcg-op-gvec.c',
'tcg-op-vec.c',
))
--
2.34.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PULL v2 02/12] accel/tcg: Uncache the host address for instruction fetch when tlb size < 1
2023-05-03 7:20 [PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
2023-05-03 7:20 ` [PULL v2 01/12] softmmu: Tidy dirtylimit_dirty_ring_full_time Richard Henderson
2023-05-03 7:20 ` [PATCH 01/84] tcg: Split out memory ops to tcg-op-ldst.c Richard Henderson
@ 2023-05-03 7:20 ` Richard Henderson
2023-05-03 7:20 ` [PATCH 02/84] tcg: Widen gen_insn_data to uint64_t Richard Henderson
` (16 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Richard Henderson @ 2023-05-03 7:20 UTC (permalink / raw)
To: qemu-devel
Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo, Weiwei Li,
Junqiang Wang, LIU Zhiwei
From: Weiwei Li <liweiwei@iscas.ac.cn>
When PMP entry overlap part of the page, we'll set the tlb_size to 1, which
will make the address in tlb entry set with TLB_INVALID_MASK, and the next
access will again go through tlb_fill.However, this way will not work in
tb_gen_code() => get_page_addr_code_hostp(): the TLB host address will be
cached, and the following instructions can use this host address directly
which may lead to the bypass of PMP related check.
Resolves: https://gitlab.com/qemu-project/qemu/-/issues/1542.
Signed-off-by: Weiwei Li <liweiwei@iscas.ac.cn>
Signed-off-by: Junqiang Wang <wangjunqiang@iscas.ac.cn>
Reviewed-by: LIU Zhiwei <zhiwei_liu@linux.alibaba.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20230422130329.23555-6-liweiwei@iscas.ac.cn>
---
accel/tcg/cputlb.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index e984a98dc4..efa0cb67c9 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -1696,6 +1696,11 @@ tb_page_addr_t get_page_addr_code_hostp(CPUArchState *env, target_ulong addr,
if (p == NULL) {
return -1;
}
+
+ if (full->lg_page_size < TARGET_PAGE_BITS) {
+ return -1;
+ }
+
if (hostp) {
*hostp = p;
}
--
2.34.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH 02/84] tcg: Widen gen_insn_data to uint64_t
2023-05-03 7:20 [PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
` (2 preceding siblings ...)
2023-05-03 7:20 ` [PULL v2 02/12] accel/tcg: Uncache the host address for instruction fetch when tlb size < 1 Richard Henderson
@ 2023-05-03 7:20 ` Richard Henderson
2023-05-03 7:20 ` [PATCH 03/84] accel/tcg: Widen tcg-ldst.h addresses " Richard Henderson
` (15 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Richard Henderson @ 2023-05-03 7:20 UTC (permalink / raw)
To: qemu-devel; +Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo
We already pass uint64_t to restore_state_to_opc; this changes all
of the other uses from insn_start through the encoding to decoding.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
include/tcg/tcg-op.h | 39 +++++++++------------------------------
include/tcg/tcg-opc.h | 2 +-
include/tcg/tcg.h | 30 +++++++++++++++---------------
accel/tcg/translate-all.c | 28 ++++++++++++++++------------
tcg/tcg.c | 18 ++++--------------
5 files changed, 45 insertions(+), 72 deletions(-)
diff --git a/include/tcg/tcg-op.h b/include/tcg/tcg-op.h
index 4401fa493c..de3b70aa84 100644
--- a/include/tcg/tcg-op.h
+++ b/include/tcg/tcg-op.h
@@ -723,48 +723,27 @@ static inline void tcg_gen_concat32_i64(TCGv_i64 ret, TCGv_i64 lo, TCGv_i64 hi)
#endif
#if TARGET_INSN_START_WORDS == 1
-# if TARGET_LONG_BITS <= TCG_TARGET_REG_BITS
static inline void tcg_gen_insn_start(target_ulong pc)
{
- tcg_gen_op1(INDEX_op_insn_start, pc);
+ TCGOp *op = tcg_emit_op(INDEX_op_insn_start, 64 / TCG_TARGET_REG_BITS);
+ tcg_set_insn_start_param(op, 0, pc);
}
-# else
-static inline void tcg_gen_insn_start(target_ulong pc)
-{
- tcg_gen_op2(INDEX_op_insn_start, (uint32_t)pc, (uint32_t)(pc >> 32));
-}
-# endif
#elif TARGET_INSN_START_WORDS == 2
-# if TARGET_LONG_BITS <= TCG_TARGET_REG_BITS
static inline void tcg_gen_insn_start(target_ulong pc, target_ulong a1)
{
- tcg_gen_op2(INDEX_op_insn_start, pc, a1);
+ TCGOp *op = tcg_emit_op(INDEX_op_insn_start, 2 * 64 / TCG_TARGET_REG_BITS);
+ tcg_set_insn_start_param(op, 0, pc);
+ tcg_set_insn_start_param(op, 1, a1);
}
-# else
-static inline void tcg_gen_insn_start(target_ulong pc, target_ulong a1)
-{
- tcg_gen_op4(INDEX_op_insn_start,
- (uint32_t)pc, (uint32_t)(pc >> 32),
- (uint32_t)a1, (uint32_t)(a1 >> 32));
-}
-# endif
#elif TARGET_INSN_START_WORDS == 3
-# if TARGET_LONG_BITS <= TCG_TARGET_REG_BITS
static inline void tcg_gen_insn_start(target_ulong pc, target_ulong a1,
target_ulong a2)
{
- tcg_gen_op3(INDEX_op_insn_start, pc, a1, a2);
+ TCGOp *op = tcg_emit_op(INDEX_op_insn_start, 3 * 64 / TCG_TARGET_REG_BITS);
+ tcg_set_insn_start_param(op, 0, pc);
+ tcg_set_insn_start_param(op, 1, a1);
+ tcg_set_insn_start_param(op, 2, a2);
}
-# else
-static inline void tcg_gen_insn_start(target_ulong pc, target_ulong a1,
- target_ulong a2)
-{
- tcg_gen_op6(INDEX_op_insn_start,
- (uint32_t)pc, (uint32_t)(pc >> 32),
- (uint32_t)a1, (uint32_t)(a1 >> 32),
- (uint32_t)a2, (uint32_t)(a2 >> 32));
-}
-# endif
#else
# error "Unhandled number of operands to insn_start"
#endif
diff --git a/include/tcg/tcg-opc.h b/include/tcg/tcg-opc.h
index 94cf7c5d6a..29216366d2 100644
--- a/include/tcg/tcg-opc.h
+++ b/include/tcg/tcg-opc.h
@@ -190,7 +190,7 @@ DEF(mulsh_i64, 1, 2, 0, IMPL64 | IMPL(TCG_TARGET_HAS_mulsh_i64))
#define DATA64_ARGS (TCG_TARGET_REG_BITS == 64 ? 1 : 2)
/* QEMU specific */
-DEF(insn_start, 0, 0, TLADDR_ARGS * TARGET_INSN_START_WORDS,
+DEF(insn_start, 0, 0, DATA64_ARGS * TARGET_INSN_START_WORDS,
TCG_OPF_NOT_PRESENT)
DEF(exit_tb, 0, 0, 1, TCG_OPF_BB_EXIT | TCG_OPF_BB_END)
DEF(goto_tb, 0, 0, 1, TCG_OPF_BB_EXIT | TCG_OPF_BB_END)
diff --git a/include/tcg/tcg.h b/include/tcg/tcg.h
index efbd891f87..7c6a613364 100644
--- a/include/tcg/tcg.h
+++ b/include/tcg/tcg.h
@@ -629,7 +629,7 @@ struct TCGContext {
TCGTemp *reg_to_temp[TCG_TARGET_NB_REGS];
uint16_t gen_insn_end_off[TCG_MAX_INSNS];
- target_ulong gen_insn_data[TCG_MAX_INSNS][TARGET_INSN_START_WORDS];
+ uint64_t gen_insn_data[TCG_MAX_INSNS][TARGET_INSN_START_WORDS];
/* Exit to translator on overflow. */
sigjmp_buf jmp_trans;
@@ -771,24 +771,24 @@ static inline void tcg_set_insn_param(TCGOp *op, int arg, TCGArg v)
op->args[arg] = v;
}
-static inline target_ulong tcg_get_insn_start_param(TCGOp *op, int arg)
+static inline uint64_t tcg_get_insn_start_param(TCGOp *op, int arg)
{
-#if TARGET_LONG_BITS <= TCG_TARGET_REG_BITS
- return tcg_get_insn_param(op, arg);
-#else
- return tcg_get_insn_param(op, arg * 2) |
- ((uint64_t)tcg_get_insn_param(op, arg * 2 + 1) << 32);
-#endif
+ if (TCG_TARGET_REG_BITS == 64) {
+ return tcg_get_insn_param(op, arg);
+ } else {
+ return deposit64(tcg_get_insn_param(op, arg * 2), 32, 32,
+ tcg_get_insn_param(op, arg * 2 + 1));
+ }
}
-static inline void tcg_set_insn_start_param(TCGOp *op, int arg, target_ulong v)
+static inline void tcg_set_insn_start_param(TCGOp *op, int arg, uint64_t v)
{
-#if TARGET_LONG_BITS <= TCG_TARGET_REG_BITS
- tcg_set_insn_param(op, arg, v);
-#else
- tcg_set_insn_param(op, arg * 2, v);
- tcg_set_insn_param(op, arg * 2 + 1, v >> 32);
-#endif
+ if (TCG_TARGET_REG_BITS == 64) {
+ tcg_set_insn_param(op, arg, v);
+ } else {
+ tcg_set_insn_param(op, arg * 2, v);
+ tcg_set_insn_param(op, arg * 2 + 1, v >> 32);
+ }
}
/* The last op that was emitted. */
diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
index 5b13281119..7b7d9a5fff 100644
--- a/accel/tcg/translate-all.c
+++ b/accel/tcg/translate-all.c
@@ -72,9 +72,11 @@ QEMU_BUILD_BUG_ON(CPU_TRACE_DSTATE_MAX_EVENTS >
TBContext tb_ctx;
-/* Encode VAL as a signed leb128 sequence at P.
- Return P incremented past the encoded value. */
-static uint8_t *encode_sleb128(uint8_t *p, target_long val)
+/*
+ * Encode VAL as a signed leb128 sequence at P.
+ * Return P incremented past the encoded value.
+ */
+static uint8_t *encode_sleb128(uint8_t *p, int64_t val)
{
int more, byte;
@@ -92,21 +94,23 @@ static uint8_t *encode_sleb128(uint8_t *p, target_long val)
return p;
}
-/* Decode a signed leb128 sequence at *PP; increment *PP past the
- decoded value. Return the decoded value. */
-static target_long decode_sleb128(const uint8_t **pp)
+/*
+ * Decode a signed leb128 sequence at *PP; increment *PP past the
+ * decoded value. Return the decoded value.
+ */
+static int64_t decode_sleb128(const uint8_t **pp)
{
const uint8_t *p = *pp;
- target_long val = 0;
+ int64_t val = 0;
int byte, shift = 0;
do {
byte = *p++;
- val |= (target_ulong)(byte & 0x7f) << shift;
+ val |= (int64_t)(byte & 0x7f) << shift;
shift += 7;
} while (byte & 0x80);
if (shift < TARGET_LONG_BITS && (byte & 0x40)) {
- val |= -(target_ulong)1 << shift;
+ val |= -(int64_t)1 << shift;
}
*pp = p;
@@ -132,7 +136,7 @@ static int encode_search(TranslationBlock *tb, uint8_t *block)
int i, j, n;
for (i = 0, n = tb->icount; i < n; ++i) {
- target_ulong prev;
+ uint64_t prev;
for (j = 0; j < TARGET_INSN_START_WORDS; ++j) {
if (i == 0) {
@@ -444,7 +448,7 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
/* Dump header and the first instruction */
fprintf(logfile, "OUT: [size=%d]\n", gen_code_size);
fprintf(logfile,
- " -- guest addr 0x" TARGET_FMT_lx " + tb prologue\n",
+ " -- guest addr 0x%016" PRIx64 " + tb prologue\n",
tcg_ctx->gen_insn_data[insn][0]);
chunk_start = tcg_ctx->gen_insn_end_off[insn];
disas(logfile, tb->tc.ptr, chunk_start);
@@ -457,7 +461,7 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
while (insn < tb->icount) {
size_t chunk_end = tcg_ctx->gen_insn_end_off[insn];
if (chunk_end > chunk_start) {
- fprintf(logfile, " -- guest addr 0x" TARGET_FMT_lx "\n",
+ fprintf(logfile, " -- guest addr 0x%016" PRIx64 "\n",
tcg_ctx->gen_insn_data[insn][0]);
disas(logfile, tb->tc.ptr + chunk_start,
chunk_end - chunk_start);
diff --git a/tcg/tcg.c b/tcg/tcg.c
index 45f8758fda..a3b8c82073 100644
--- a/tcg/tcg.c
+++ b/tcg/tcg.c
@@ -2276,13 +2276,8 @@ static void tcg_dump_ops(TCGContext *s, FILE *f, bool have_prefs)
col += ne_fprintf(f, "\n ----");
for (i = 0; i < TARGET_INSN_START_WORDS; ++i) {
- target_ulong a;
-#if TARGET_LONG_BITS > TCG_TARGET_REG_BITS
- a = deposit64(op->args[i * 2], 32, 32, op->args[i * 2 + 1]);
-#else
- a = op->args[i];
-#endif
- col += ne_fprintf(f, " " TARGET_FMT_lx, a);
+ col += ne_fprintf(f, " %016" PRIx64,
+ tcg_get_insn_start_param(op, i));
}
} else if (c == INDEX_op_call) {
const TCGHelperInfo *info = tcg_call_info(op);
@@ -5940,13 +5935,8 @@ int tcg_gen_code(TCGContext *s, TranslationBlock *tb, target_ulong pc_start)
}
num_insns++;
for (i = 0; i < TARGET_INSN_START_WORDS; ++i) {
- target_ulong a;
-#if TARGET_LONG_BITS > TCG_TARGET_REG_BITS
- a = deposit64(op->args[i * 2], 32, 32, op->args[i * 2 + 1]);
-#else
- a = op->args[i];
-#endif
- s->gen_insn_data[num_insns][i] = a;
+ s->gen_insn_data[num_insns][i] =
+ tcg_get_insn_start_param(op, i);
}
break;
case INDEX_op_discard:
--
2.34.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH 03/84] accel/tcg: Widen tcg-ldst.h addresses to uint64_t
2023-05-03 7:20 [PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
` (3 preceding siblings ...)
2023-05-03 7:20 ` [PATCH 02/84] tcg: Widen gen_insn_data to uint64_t Richard Henderson
@ 2023-05-03 7:20 ` Richard Henderson
2023-05-03 7:20 ` [PULL v2 03/12] qemu/bitops.h: Limit rotate amounts Richard Henderson
` (14 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Richard Henderson @ 2023-05-03 7:20 UTC (permalink / raw)
To: qemu-devel; +Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo
Always pass the target address as uint64_t.
Adjust tcg_out_{ld,st}_helper_args to match.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
include/tcg/tcg-ldst.h | 26 +++++++++---------
accel/tcg/cputlb.c | 26 +++++++++---------
accel/tcg/user-exec.c | 26 +++++++++---------
tcg/tcg.c | 62 ++++++++++++++++++++++++++++++++----------
4 files changed, 87 insertions(+), 53 deletions(-)
diff --git a/include/tcg/tcg-ldst.h b/include/tcg/tcg-ldst.h
index 7dd57013e9..6ccfe9131d 100644
--- a/include/tcg/tcg-ldst.h
+++ b/include/tcg/tcg-ldst.h
@@ -26,38 +26,38 @@
#define TCG_LDST_H
/* Value zero-extended to tcg register size. */
-tcg_target_ulong helper_ldub_mmu(CPUArchState *env, target_ulong addr,
+tcg_target_ulong helper_ldub_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t retaddr);
-tcg_target_ulong helper_lduw_mmu(CPUArchState *env, target_ulong addr,
+tcg_target_ulong helper_lduw_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t retaddr);
-tcg_target_ulong helper_ldul_mmu(CPUArchState *env, target_ulong addr,
+tcg_target_ulong helper_ldul_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t retaddr);
-uint64_t helper_ldq_mmu(CPUArchState *env, target_ulong addr,
+uint64_t helper_ldq_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t retaddr);
-Int128 helper_ld16_mmu(CPUArchState *env, target_ulong addr,
+Int128 helper_ld16_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t retaddr);
/* Value sign-extended to tcg register size. */
-tcg_target_ulong helper_ldsb_mmu(CPUArchState *env, target_ulong addr,
+tcg_target_ulong helper_ldsb_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t retaddr);
-tcg_target_ulong helper_ldsw_mmu(CPUArchState *env, target_ulong addr,
+tcg_target_ulong helper_ldsw_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t retaddr);
-tcg_target_ulong helper_ldsl_mmu(CPUArchState *env, target_ulong addr,
+tcg_target_ulong helper_ldsl_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t retaddr);
/*
* Value extended to at least uint32_t, so that some ABIs do not require
* zero-extension from uint8_t or uint16_t.
*/
-void helper_stb_mmu(CPUArchState *env, target_ulong addr, uint32_t val,
+void helper_stb_mmu(CPUArchState *env, uint64_t addr, uint32_t val,
MemOpIdx oi, uintptr_t retaddr);
-void helper_stw_mmu(CPUArchState *env, target_ulong addr, uint32_t val,
+void helper_stw_mmu(CPUArchState *env, uint64_t addr, uint32_t val,
MemOpIdx oi, uintptr_t retaddr);
-void helper_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val,
+void helper_stl_mmu(CPUArchState *env, uint64_t addr, uint32_t val,
MemOpIdx oi, uintptr_t retaddr);
-void helper_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val,
+void helper_stq_mmu(CPUArchState *env, uint64_t addr, uint64_t val,
MemOpIdx oi, uintptr_t retaddr);
-void helper_st16_mmu(CPUArchState *env, target_ulong addr, Int128 val,
+void helper_st16_mmu(CPUArchState *env, uint64_t addr, Int128 val,
MemOpIdx oi, uintptr_t retaddr);
#endif /* TCG_LDST_H */
diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index a77b439df8..b594401267 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -2357,7 +2357,7 @@ static uint8_t do_ld1_mmu(CPUArchState *env, target_ulong addr, MemOpIdx oi,
return do_ld_1(env, &l.page[0], l.mmu_idx, access_type, ra);
}
-tcg_target_ulong helper_ldub_mmu(CPUArchState *env, target_ulong addr,
+tcg_target_ulong helper_ldub_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t retaddr)
{
tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_8);
@@ -2388,7 +2388,7 @@ static uint16_t do_ld2_mmu(CPUArchState *env, target_ulong addr, MemOpIdx oi,
return ret;
}
-tcg_target_ulong helper_lduw_mmu(CPUArchState *env, target_ulong addr,
+tcg_target_ulong helper_lduw_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t retaddr)
{
tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_16);
@@ -2415,7 +2415,7 @@ static uint32_t do_ld4_mmu(CPUArchState *env, target_ulong addr, MemOpIdx oi,
return ret;
}
-tcg_target_ulong helper_ldul_mmu(CPUArchState *env, target_ulong addr,
+tcg_target_ulong helper_ldul_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t retaddr)
{
tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_32);
@@ -2442,7 +2442,7 @@ static uint64_t do_ld8_mmu(CPUArchState *env, target_ulong addr, MemOpIdx oi,
return ret;
}
-uint64_t helper_ldq_mmu(CPUArchState *env, target_ulong addr,
+uint64_t helper_ldq_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t retaddr)
{
tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_64);
@@ -2454,19 +2454,19 @@ uint64_t helper_ldq_mmu(CPUArchState *env, target_ulong addr,
* avoid this for 64-bit data, or for 32-bit data on 32-bit host.
*/
-tcg_target_ulong helper_ldsb_mmu(CPUArchState *env, target_ulong addr,
+tcg_target_ulong helper_ldsb_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t retaddr)
{
return (int8_t)helper_ldub_mmu(env, addr, oi, retaddr);
}
-tcg_target_ulong helper_ldsw_mmu(CPUArchState *env, target_ulong addr,
+tcg_target_ulong helper_ldsw_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t retaddr)
{
return (int16_t)helper_lduw_mmu(env, addr, oi, retaddr);
}
-tcg_target_ulong helper_ldsl_mmu(CPUArchState *env, target_ulong addr,
+tcg_target_ulong helper_ldsl_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t retaddr)
{
return (int32_t)helper_ldul_mmu(env, addr, oi, retaddr);
@@ -2534,7 +2534,7 @@ static Int128 do_ld16_mmu(CPUArchState *env, target_ulong addr,
return ret;
}
-Int128 helper_ld16_mmu(CPUArchState *env, target_ulong addr,
+Int128 helper_ld16_mmu(CPUArchState *env, uint64_t addr,
uint32_t oi, uintptr_t retaddr)
{
tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_128);
@@ -2842,7 +2842,7 @@ static void do_st_8(CPUArchState *env, MMULookupPageData *p, uint64_t val,
}
}
-void helper_stb_mmu(CPUArchState *env, target_ulong addr, uint32_t val,
+void helper_stb_mmu(CPUArchState *env, uint64_t addr, uint32_t val,
MemOpIdx oi, uintptr_t ra)
{
MMULookupLocals l;
@@ -2877,7 +2877,7 @@ static void do_st2_mmu(CPUArchState *env, target_ulong addr, uint16_t val,
do_st_1(env, &l.page[1], b, l.mmu_idx, ra);
}
-void helper_stw_mmu(CPUArchState *env, target_ulong addr, uint32_t val,
+void helper_stw_mmu(CPUArchState *env, uint64_t addr, uint32_t val,
MemOpIdx oi, uintptr_t retaddr)
{
tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_16);
@@ -2904,7 +2904,7 @@ static void do_st4_mmu(CPUArchState *env, target_ulong addr, uint32_t val,
(void) do_st_leN(env, &l.page[1], val, l.mmu_idx, l.memop, ra);
}
-void helper_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val,
+void helper_stl_mmu(CPUArchState *env, uint64_t addr, uint32_t val,
MemOpIdx oi, uintptr_t retaddr)
{
tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_32);
@@ -2931,7 +2931,7 @@ static void do_st8_mmu(CPUArchState *env, target_ulong addr, uint64_t val,
(void) do_st_leN(env, &l.page[1], val, l.mmu_idx, l.memop, ra);
}
-void helper_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val,
+void helper_stq_mmu(CPUArchState *env, uint64_t addr, uint64_t val,
MemOpIdx oi, uintptr_t retaddr)
{
tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_64);
@@ -2999,7 +2999,7 @@ static void do_st16_mmu(CPUArchState *env, target_ulong addr, Int128 val,
}
}
-void helper_st16_mmu(CPUArchState *env, target_ulong addr, Int128 val,
+void helper_st16_mmu(CPUArchState *env, uint64_t addr, Int128 val,
MemOpIdx oi, uintptr_t retaddr)
{
tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_128);
diff --git a/accel/tcg/user-exec.c b/accel/tcg/user-exec.c
index 7b824dcde8..9a7afb6f78 100644
--- a/accel/tcg/user-exec.c
+++ b/accel/tcg/user-exec.c
@@ -920,13 +920,13 @@ static uint8_t do_ld1_mmu(CPUArchState *env, abi_ptr addr,
return ret;
}
-tcg_target_ulong helper_ldub_mmu(CPUArchState *env, target_ulong addr,
+tcg_target_ulong helper_ldub_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t ra)
{
return do_ld1_mmu(env, addr, get_memop(oi), ra);
}
-tcg_target_ulong helper_ldsb_mmu(CPUArchState *env, target_ulong addr,
+tcg_target_ulong helper_ldsb_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t ra)
{
return (int8_t)do_ld1_mmu(env, addr, get_memop(oi), ra);
@@ -953,7 +953,7 @@ static uint16_t do_ld2_he_mmu(CPUArchState *env, abi_ptr addr,
return ret;
}
-tcg_target_ulong helper_lduw_mmu(CPUArchState *env, target_ulong addr,
+tcg_target_ulong helper_lduw_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t ra)
{
MemOp mop = get_memop(oi);
@@ -965,7 +965,7 @@ tcg_target_ulong helper_lduw_mmu(CPUArchState *env, target_ulong addr,
return ret;
}
-tcg_target_ulong helper_ldsw_mmu(CPUArchState *env, target_ulong addr,
+tcg_target_ulong helper_ldsw_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t ra)
{
MemOp mop = get_memop(oi);
@@ -1014,7 +1014,7 @@ static uint32_t do_ld4_he_mmu(CPUArchState *env, abi_ptr addr,
return ret;
}
-tcg_target_ulong helper_ldul_mmu(CPUArchState *env, target_ulong addr,
+tcg_target_ulong helper_ldul_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t ra)
{
MemOp mop = get_memop(oi);
@@ -1026,7 +1026,7 @@ tcg_target_ulong helper_ldul_mmu(CPUArchState *env, target_ulong addr,
return ret;
}
-tcg_target_ulong helper_ldsl_mmu(CPUArchState *env, target_ulong addr,
+tcg_target_ulong helper_ldsl_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t ra)
{
MemOp mop = get_memop(oi);
@@ -1075,7 +1075,7 @@ static uint64_t do_ld8_he_mmu(CPUArchState *env, abi_ptr addr,
return ret;
}
-uint64_t helper_ldq_mmu(CPUArchState *env, target_ulong addr,
+uint64_t helper_ldq_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t ra)
{
MemOp mop = get_memop(oi);
@@ -1124,7 +1124,7 @@ static Int128 do_ld16_he_mmu(CPUArchState *env, abi_ptr addr,
return ret;
}
-Int128 helper_ld16_mmu(CPUArchState *env, target_ulong addr,
+Int128 helper_ld16_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t ra)
{
MemOp mop = get_memop(oi);
@@ -1182,7 +1182,7 @@ static void do_st1_mmu(CPUArchState *env, abi_ptr addr, uint8_t val,
clear_helper_retaddr();
}
-void helper_stb_mmu(CPUArchState *env, target_ulong addr, uint32_t val,
+void helper_stb_mmu(CPUArchState *env, uint64_t addr, uint32_t val,
MemOpIdx oi, uintptr_t ra)
{
do_st1_mmu(env, addr, val, get_memop(oi), ra);
@@ -1206,7 +1206,7 @@ static void do_st2_he_mmu(CPUArchState *env, abi_ptr addr, uint16_t val,
clear_helper_retaddr();
}
-void helper_stw_mmu(CPUArchState *env, target_ulong addr, uint32_t val,
+void helper_stw_mmu(CPUArchState *env, uint64_t addr, uint32_t val,
MemOpIdx oi, uintptr_t ra)
{
MemOp mop = get_memop(oi);
@@ -1248,7 +1248,7 @@ static void do_st4_he_mmu(CPUArchState *env, abi_ptr addr, uint32_t val,
clear_helper_retaddr();
}
-void helper_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val,
+void helper_stl_mmu(CPUArchState *env, uint64_t addr, uint32_t val,
MemOpIdx oi, uintptr_t ra)
{
MemOp mop = get_memop(oi);
@@ -1290,7 +1290,7 @@ static void do_st8_he_mmu(CPUArchState *env, abi_ptr addr, uint64_t val,
clear_helper_retaddr();
}
-void helper_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val,
+void helper_stq_mmu(CPUArchState *env, uint64_t addr, uint64_t val,
MemOpIdx oi, uintptr_t ra)
{
MemOp mop = get_memop(oi);
@@ -1332,7 +1332,7 @@ static void do_st16_he_mmu(CPUArchState *env, abi_ptr addr, Int128 val,
clear_helper_retaddr();
}
-void helper_st16_mmu(CPUArchState *env, target_ulong addr, Int128 val,
+void helper_st16_mmu(CPUArchState *env, uint64_t addr, Int128 val,
MemOpIdx oi, uintptr_t ra)
{
MemOp mop = get_memop(oi);
diff --git a/tcg/tcg.c b/tcg/tcg.c
index a3b8c82073..685e555754 100644
--- a/tcg/tcg.c
+++ b/tcg/tcg.c
@@ -768,7 +768,7 @@ static TCGHelperInfo info_helper_ld32_mmu = {
.flags = TCG_CALL_NO_WG,
.typemask = dh_typemask(ttl, 0) /* return tcg_target_ulong */
| dh_typemask(env, 1)
- | dh_typemask(tl, 2) /* target_ulong addr */
+ | dh_typemask(i64, 2) /* uint64_t addr */
| dh_typemask(i32, 3) /* unsigned oi */
| dh_typemask(ptr, 4) /* uintptr_t ra */
};
@@ -777,7 +777,7 @@ static TCGHelperInfo info_helper_ld64_mmu = {
.flags = TCG_CALL_NO_WG,
.typemask = dh_typemask(i64, 0) /* return uint64_t */
| dh_typemask(env, 1)
- | dh_typemask(tl, 2) /* target_ulong addr */
+ | dh_typemask(i64, 2) /* uint64_t addr */
| dh_typemask(i32, 3) /* unsigned oi */
| dh_typemask(ptr, 4) /* uintptr_t ra */
};
@@ -786,7 +786,7 @@ static TCGHelperInfo info_helper_ld128_mmu = {
.flags = TCG_CALL_NO_WG,
.typemask = dh_typemask(i128, 0) /* return Int128 */
| dh_typemask(env, 1)
- | dh_typemask(tl, 2) /* target_ulong addr */
+ | dh_typemask(i64, 2) /* uint64_t addr */
| dh_typemask(i32, 3) /* unsigned oi */
| dh_typemask(ptr, 4) /* uintptr_t ra */
};
@@ -795,7 +795,7 @@ static TCGHelperInfo info_helper_st32_mmu = {
.flags = TCG_CALL_NO_WG,
.typemask = dh_typemask(void, 0)
| dh_typemask(env, 1)
- | dh_typemask(tl, 2) /* target_ulong addr */
+ | dh_typemask(i64, 2) /* uint64_t addr */
| dh_typemask(i32, 3) /* uint32_t data */
| dh_typemask(i32, 4) /* unsigned oi */
| dh_typemask(ptr, 5) /* uintptr_t ra */
@@ -805,7 +805,7 @@ static TCGHelperInfo info_helper_st64_mmu = {
.flags = TCG_CALL_NO_WG,
.typemask = dh_typemask(void, 0)
| dh_typemask(env, 1)
- | dh_typemask(tl, 2) /* target_ulong addr */
+ | dh_typemask(i64, 2) /* uint64_t addr */
| dh_typemask(i64, 3) /* uint64_t data */
| dh_typemask(i32, 4) /* unsigned oi */
| dh_typemask(ptr, 5) /* uintptr_t ra */
@@ -815,7 +815,7 @@ static TCGHelperInfo info_helper_st128_mmu = {
.flags = TCG_CALL_NO_WG,
.typemask = dh_typemask(void, 0)
| dh_typemask(env, 1)
- | dh_typemask(tl, 2) /* target_ulong addr */
+ | dh_typemask(i64, 2) /* uint64_t addr */
| dh_typemask(i128, 3) /* Int128 data */
| dh_typemask(i32, 4) /* unsigned oi */
| dh_typemask(ptr, 5) /* uintptr_t ra */
@@ -5456,11 +5456,26 @@ static void tcg_out_ld_helper_args(TCGContext *s, const TCGLabelQemuLdst *ldst,
next_arg = 1;
loc = &info->in[next_arg];
- nmov = tcg_out_helper_add_mov(mov, loc, TCG_TYPE_TL, TCG_TYPE_TL,
- ldst->addrlo_reg, ldst->addrhi_reg);
- next_arg += nmov;
+ if (TCG_TARGET_REG_BITS == 64 || TARGET_LONG_BITS == 64) {
+ nmov = tcg_out_helper_add_mov(mov, loc, TCG_TYPE_I64, TCG_TYPE_TL,
+ ldst->addrlo_reg, ldst->addrhi_reg);
+ tcg_out_helper_load_slots(s, nmov, mov, parm);
+ next_arg += nmov;
+ } else {
+ /*
+ * 32-bit host with 32-bit guest: zero-extend the guest address
+ * to 64-bits for the helper by storing the low part, then
+ * load a zero for the high part.
+ */
+ tcg_out_helper_add_mov(mov, loc + HOST_BIG_ENDIAN,
+ TCG_TYPE_I32, TCG_TYPE_I32,
+ ldst->addrlo_reg, -1);
+ tcg_out_helper_load_slots(s, 1, mov, parm);
- tcg_out_helper_load_slots(s, nmov, mov, parm);
+ tcg_out_helper_load_imm(s, loc[!HOST_BIG_ENDIAN].arg_slot,
+ TCG_TYPE_I32, 0, parm);
+ next_arg += 2;
+ }
switch (info->out_kind) {
case TCG_CALL_RET_NORMAL:
@@ -5614,10 +5629,24 @@ static void tcg_out_st_helper_args(TCGContext *s, const TCGLabelQemuLdst *ldst,
/* Handle addr argument. */
loc = &info->in[next_arg];
- n = tcg_out_helper_add_mov(mov, loc, TCG_TYPE_TL, TCG_TYPE_TL,
- ldst->addrlo_reg, ldst->addrhi_reg);
- next_arg += n;
- nmov += n;
+ if (TCG_TARGET_REG_BITS == 64 || TARGET_LONG_BITS == 64) {
+ n = tcg_out_helper_add_mov(mov, loc, TCG_TYPE_I64, TCG_TYPE_TL,
+ ldst->addrlo_reg, ldst->addrhi_reg);
+ next_arg += n;
+ nmov += n;
+ } else {
+ /*
+ * 32-bit host with 32-bit guest: zero-extend the guest address
+ * to 64-bits for the helper by storing the low part. Later,
+ * after we have processed the register inputs, we will load a
+ * zero for the high part.
+ */
+ tcg_out_helper_add_mov(mov, loc + HOST_BIG_ENDIAN,
+ TCG_TYPE_I32, TCG_TYPE_I32,
+ ldst->addrlo_reg, -1);
+ next_arg += 2;
+ nmov += 1;
+ }
/* Handle data argument. */
loc = &info->in[next_arg];
@@ -5662,6 +5691,11 @@ static void tcg_out_st_helper_args(TCGContext *s, const TCGLabelQemuLdst *ldst,
g_assert_not_reached();
}
+ if (TCG_TARGET_REG_BITS == 32 && TARGET_LONG_BITS == 32) {
+ loc = &info->in[1 + !HOST_BIG_ENDIAN];
+ tcg_out_helper_load_imm(s, loc->arg_slot, TCG_TYPE_I32, 0, parm);
+ }
+
tcg_out_helper_load_common_args(s, ldst, parm, info, next_arg);
}
--
2.34.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PULL v2 03/12] qemu/bitops.h: Limit rotate amounts
2023-05-03 7:20 [PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
` (4 preceding siblings ...)
2023-05-03 7:20 ` [PATCH 03/84] accel/tcg: Widen tcg-ldst.h addresses " Richard Henderson
@ 2023-05-03 7:20 ` Richard Henderson
2023-05-03 7:20 ` [PULL v2 04/12] qemu/host-utils.h: Add clz and ctz functions for lower-bit integers Richard Henderson
` (13 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Richard Henderson @ 2023-05-03 7:20 UTC (permalink / raw)
To: qemu-devel; +Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo, Dickon Hood
From: Dickon Hood <dickon.hood@codethink.co.uk>
Rotates have been fixed up to only allow for reasonable rotate amounts
(ie, no rotates >7 on an 8b value etc.) This fixes a problem with riscv
vector rotate instructions.
Signed-off-by: Dickon Hood <dickon.hood@codethink.co.uk>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20230428144757.57530-9-lawrence.hunter@codethink.co.uk>
[rth: Mask shifts in both directions.]
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
include/qemu/bitops.h | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/include/qemu/bitops.h b/include/qemu/bitops.h
index 03213ce952..cb3526d1f4 100644
--- a/include/qemu/bitops.h
+++ b/include/qemu/bitops.h
@@ -218,7 +218,7 @@ static inline unsigned long find_first_zero_bit(const unsigned long *addr,
*/
static inline uint8_t rol8(uint8_t word, unsigned int shift)
{
- return (word << shift) | (word >> ((8 - shift) & 7));
+ return (word << (shift & 7)) | (word >> (-shift & 7));
}
/**
@@ -228,7 +228,7 @@ static inline uint8_t rol8(uint8_t word, unsigned int shift)
*/
static inline uint8_t ror8(uint8_t word, unsigned int shift)
{
- return (word >> shift) | (word << ((8 - shift) & 7));
+ return (word >> (shift & 7)) | (word << (-shift & 7));
}
/**
@@ -238,7 +238,7 @@ static inline uint8_t ror8(uint8_t word, unsigned int shift)
*/
static inline uint16_t rol16(uint16_t word, unsigned int shift)
{
- return (word << shift) | (word >> ((16 - shift) & 15));
+ return (word << (shift & 15)) | (word >> (-shift & 15));
}
/**
@@ -248,7 +248,7 @@ static inline uint16_t rol16(uint16_t word, unsigned int shift)
*/
static inline uint16_t ror16(uint16_t word, unsigned int shift)
{
- return (word >> shift) | (word << ((16 - shift) & 15));
+ return (word >> (shift & 15)) | (word << (-shift & 15));
}
/**
@@ -258,7 +258,7 @@ static inline uint16_t ror16(uint16_t word, unsigned int shift)
*/
static inline uint32_t rol32(uint32_t word, unsigned int shift)
{
- return (word << shift) | (word >> ((32 - shift) & 31));
+ return (word << (shift & 31)) | (word >> (-shift & 31));
}
/**
@@ -268,7 +268,7 @@ static inline uint32_t rol32(uint32_t word, unsigned int shift)
*/
static inline uint32_t ror32(uint32_t word, unsigned int shift)
{
- return (word >> shift) | (word << ((32 - shift) & 31));
+ return (word >> (shift & 31)) | (word << (-shift & 31));
}
/**
@@ -278,7 +278,7 @@ static inline uint32_t ror32(uint32_t word, unsigned int shift)
*/
static inline uint64_t rol64(uint64_t word, unsigned int shift)
{
- return (word << shift) | (word >> ((64 - shift) & 63));
+ return (word << (shift & 63)) | (word >> (-shift & 63));
}
/**
@@ -288,7 +288,7 @@ static inline uint64_t rol64(uint64_t word, unsigned int shift)
*/
static inline uint64_t ror64(uint64_t word, unsigned int shift)
{
- return (word >> shift) | (word << ((64 - shift) & 63));
+ return (word >> (shift & 63)) | (word << (-shift & 63));
}
/**
--
2.34.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PULL v2 04/12] qemu/host-utils.h: Add clz and ctz functions for lower-bit integers
2023-05-03 7:20 [PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
` (5 preceding siblings ...)
2023-05-03 7:20 ` [PULL v2 03/12] qemu/bitops.h: Limit rotate amounts Richard Henderson
@ 2023-05-03 7:20 ` Richard Henderson
2023-05-03 7:20 ` [PATCH 04/84] tcg: Widen helper_{ld,st}_i128 addresses to uint64_t Richard Henderson
` (12 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Richard Henderson @ 2023-05-03 7:20 UTC (permalink / raw)
To: qemu-devel
Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo, Kiran Ostrolenk
From: Kiran Ostrolenk <kiran.ostrolenk@codethink.co.uk>
This is for use in the RISC-V vclz and vctz instructions (implemented in
proceeding commit).
Signed-off-by: Kiran Ostrolenk <kiran.ostrolenk@codethink.co.uk>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20230428144757.57530-11-lawrence.hunter@codethink.co.uk>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
include/qemu/host-utils.h | 54 +++++++++++++++++++++++++++++++++++++++
1 file changed, 54 insertions(+)
diff --git a/include/qemu/host-utils.h b/include/qemu/host-utils.h
index 3ce62bf4a5..d3b4dce6a9 100644
--- a/include/qemu/host-utils.h
+++ b/include/qemu/host-utils.h
@@ -107,6 +107,36 @@ static inline uint64_t muldiv64(uint64_t a, uint32_t b, uint32_t c)
}
#endif
+/**
+ * clz8 - count leading zeros in a 8-bit value.
+ * @val: The value to search
+ *
+ * Returns 8 if the value is zero. Note that the GCC builtin is
+ * undefined if the value is zero.
+ *
+ * Note that the GCC builtin will upcast its argument to an `unsigned int`
+ * so this function subtracts off the number of prepended zeroes.
+ */
+static inline int clz8(uint8_t val)
+{
+ return val ? __builtin_clz(val) - 24 : 8;
+}
+
+/**
+ * clz16 - count leading zeros in a 16-bit value.
+ * @val: The value to search
+ *
+ * Returns 16 if the value is zero. Note that the GCC builtin is
+ * undefined if the value is zero.
+ *
+ * Note that the GCC builtin will upcast its argument to an `unsigned int`
+ * so this function subtracts off the number of prepended zeroes.
+ */
+static inline int clz16(uint16_t val)
+{
+ return val ? __builtin_clz(val) - 16 : 16;
+}
+
/**
* clz32 - count leading zeros in a 32-bit value.
* @val: The value to search
@@ -153,6 +183,30 @@ static inline int clo64(uint64_t val)
return clz64(~val);
}
+/**
+ * ctz8 - count trailing zeros in a 8-bit value.
+ * @val: The value to search
+ *
+ * Returns 8 if the value is zero. Note that the GCC builtin is
+ * undefined if the value is zero.
+ */
+static inline int ctz8(uint8_t val)
+{
+ return val ? __builtin_ctz(val) : 8;
+}
+
+/**
+ * ctz16 - count trailing zeros in a 16-bit value.
+ * @val: The value to search
+ *
+ * Returns 16 if the value is zero. Note that the GCC builtin is
+ * undefined if the value is zero.
+ */
+static inline int ctz16(uint16_t val)
+{
+ return val ? __builtin_ctz(val) : 16;
+}
+
/**
* ctz32 - count trailing zeros in a 32-bit value.
* @val: The value to search
--
2.34.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH 04/84] tcg: Widen helper_{ld,st}_i128 addresses to uint64_t
2023-05-03 7:20 [PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
` (6 preceding siblings ...)
2023-05-03 7:20 ` [PULL v2 04/12] qemu/host-utils.h: Add clz and ctz functions for lower-bit integers Richard Henderson
@ 2023-05-03 7:20 ` Richard Henderson
2023-05-03 7:20 ` [PULL v2 05/12] tcg: Add tcg_gen_gvec_andcs Richard Henderson
` (11 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Richard Henderson @ 2023-05-03 7:20 UTC (permalink / raw)
To: qemu-devel; +Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo
Always pass the target address as uint64_t.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
accel/tcg/tcg-runtime.h | 4 ++--
accel/tcg/cputlb.c | 5 ++---
accel/tcg/user-exec.c | 5 ++---
tcg/tcg-op-ldst.c | 26 ++++++++++++++++++++++++--
4 files changed, 30 insertions(+), 10 deletions(-)
diff --git a/accel/tcg/tcg-runtime.h b/accel/tcg/tcg-runtime.h
index d9adc646c1..0e6c5f55fd 100644
--- a/accel/tcg/tcg-runtime.h
+++ b/accel/tcg/tcg-runtime.h
@@ -39,8 +39,8 @@ DEF_HELPER_FLAGS_1(exit_atomic, TCG_CALL_NO_WG, noreturn, env)
DEF_HELPER_FLAGS_3(memset, TCG_CALL_NO_RWG, ptr, ptr, int, ptr)
#endif /* IN_HELPER_PROTO */
-DEF_HELPER_FLAGS_3(ld_i128, TCG_CALL_NO_WG, i128, env, tl, i32)
-DEF_HELPER_FLAGS_4(st_i128, TCG_CALL_NO_WG, void, env, tl, i128, i32)
+DEF_HELPER_FLAGS_3(ld_i128, TCG_CALL_NO_WG, i128, env, i64, i32)
+DEF_HELPER_FLAGS_4(st_i128, TCG_CALL_NO_WG, void, env, i64, i128, i32)
DEF_HELPER_FLAGS_5(atomic_cmpxchgb, TCG_CALL_NO_WG,
i32, env, tl, i32, i32, i32)
diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index b594401267..e494404a0d 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -2541,7 +2541,7 @@ Int128 helper_ld16_mmu(CPUArchState *env, uint64_t addr,
return do_ld16_mmu(env, addr, oi, retaddr);
}
-Int128 helper_ld_i128(CPUArchState *env, target_ulong addr, uint32_t oi)
+Int128 helper_ld_i128(CPUArchState *env, uint64_t addr, uint32_t oi)
{
return helper_ld16_mmu(env, addr, oi, GETPC());
}
@@ -3006,8 +3006,7 @@ void helper_st16_mmu(CPUArchState *env, uint64_t addr, Int128 val,
do_st16_mmu(env, addr, val, oi, retaddr);
}
-void helper_st_i128(CPUArchState *env, target_ulong addr, Int128 val,
- MemOpIdx oi)
+void helper_st_i128(CPUArchState *env, uint64_t addr, Int128 val, MemOpIdx oi)
{
helper_st16_mmu(env, addr, val, oi, GETPC());
}
diff --git a/accel/tcg/user-exec.c b/accel/tcg/user-exec.c
index 9a7afb6f78..36ad8284a5 100644
--- a/accel/tcg/user-exec.c
+++ b/accel/tcg/user-exec.c
@@ -1136,7 +1136,7 @@ Int128 helper_ld16_mmu(CPUArchState *env, uint64_t addr,
return ret;
}
-Int128 helper_ld_i128(CPUArchState *env, target_ulong addr, MemOpIdx oi)
+Int128 helper_ld_i128(CPUArchState *env, uint64_t addr, MemOpIdx oi)
{
return helper_ld16_mmu(env, addr, oi, GETPC());
}
@@ -1343,8 +1343,7 @@ void helper_st16_mmu(CPUArchState *env, uint64_t addr, Int128 val,
do_st16_he_mmu(env, addr, val, mop, ra);
}
-void helper_st_i128(CPUArchState *env, target_ulong addr,
- Int128 val, MemOpIdx oi)
+void helper_st_i128(CPUArchState *env, uint64_t addr, Int128 val, MemOpIdx oi)
{
helper_st16_mmu(env, addr, val, oi, GETPC());
}
diff --git a/tcg/tcg-op-ldst.c b/tcg/tcg-op-ldst.c
index a3beede2f4..0010afb1a1 100644
--- a/tcg/tcg-op-ldst.c
+++ b/tcg/tcg-op-ldst.c
@@ -404,6 +404,24 @@ static void canonicalize_memop_i128_as_i64(MemOp ret[2], MemOp orig)
#define tcg_temp_ebb_new tcg_temp_ebb_new_i32
#endif
+static TCGv_i64 maybe_extend_addr64(TCGv addr)
+{
+#if TARGET_LONG_BITS == 32
+ TCGv_i64 a64 = tcg_temp_ebb_new_i64();
+ tcg_gen_extu_i32_i64(a64, addr);
+ return a64;
+#else
+ return addr;
+#endif
+}
+
+static void maybe_free_addr64(TCGv_i64 a64)
+{
+#if TARGET_LONG_BITS == 32
+ tcg_temp_free_i64(a64);
+#endif
+}
+
void tcg_gen_qemu_ld_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop)
{
const MemOpIdx oi = make_memop_idx(memop, idx);
@@ -478,7 +496,9 @@ void tcg_gen_qemu_ld_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop)
tcg_gen_bswap64_i64(y, y);
}
} else {
- gen_helper_ld_i128(val, cpu_env, addr, tcg_constant_i32(oi));
+ TCGv_i64 a64 = maybe_extend_addr64(addr);
+ gen_helper_ld_i128(val, cpu_env, a64, tcg_constant_i32(oi));
+ maybe_free_addr64(a64);
}
plugin_gen_mem_callbacks(addr, oi, QEMU_PLUGIN_MEM_R);
@@ -558,7 +578,9 @@ void tcg_gen_qemu_st_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop)
}
tcg_temp_free(addr_p8);
} else {
- gen_helper_st_i128(cpu_env, addr, val, tcg_constant_i32(oi));
+ TCGv_i64 a64 = maybe_extend_addr64(addr);
+ gen_helper_st_i128(cpu_env, a64, val, tcg_constant_i32(oi));
+ maybe_free_addr64(a64);
}
plugin_gen_mem_callbacks(addr, oi, QEMU_PLUGIN_MEM_W);
--
2.34.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PULL v2 05/12] tcg: Add tcg_gen_gvec_andcs
2023-05-03 7:20 [PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
` (7 preceding siblings ...)
2023-05-03 7:20 ` [PATCH 04/84] tcg: Widen helper_{ld,st}_i128 addresses to uint64_t Richard Henderson
@ 2023-05-03 7:20 ` Richard Henderson
2023-05-03 7:20 ` [PATCH 05/84] tcg: Widen helper_atomic_* addresses to uint64_t Richard Henderson
` (10 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Richard Henderson @ 2023-05-03 7:20 UTC (permalink / raw)
To: qemu-devel
Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo, Nazar Kazakov
From: Nazar Kazakov <nazar.kazakov@codethink.co.uk>
Add tcg expander and helper functions for and-compliment
vector with scalar operand.
Signed-off-by: Nazar Kazakov <nazar.kazakov@codethink.co.uk>
Message-Id: <20230428144757.57530-10-lawrence.hunter@codethink.co.uk>
[rth: Split out of larger patch.]
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
accel/tcg/tcg-runtime.h | 1 +
include/tcg/tcg-op-gvec.h | 2 ++
accel/tcg/tcg-runtime-gvec.c | 11 +++++++++++
tcg/tcg-op-gvec.c | 17 +++++++++++++++++
4 files changed, 31 insertions(+)
diff --git a/accel/tcg/tcg-runtime.h b/accel/tcg/tcg-runtime.h
index e141a6ab24..b8e6421c8a 100644
--- a/accel/tcg/tcg-runtime.h
+++ b/accel/tcg/tcg-runtime.h
@@ -217,6 +217,7 @@ DEF_HELPER_FLAGS_4(gvec_nor, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32)
DEF_HELPER_FLAGS_4(gvec_eqv, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32)
DEF_HELPER_FLAGS_4(gvec_ands, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32)
+DEF_HELPER_FLAGS_4(gvec_andcs, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32)
DEF_HELPER_FLAGS_4(gvec_xors, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32)
DEF_HELPER_FLAGS_4(gvec_ors, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32)
diff --git a/include/tcg/tcg-op-gvec.h b/include/tcg/tcg-op-gvec.h
index 28cafbcc5c..6d58683171 100644
--- a/include/tcg/tcg-op-gvec.h
+++ b/include/tcg/tcg-op-gvec.h
@@ -330,6 +330,8 @@ void tcg_gen_gvec_ori(unsigned vece, uint32_t dofs, uint32_t aofs,
void tcg_gen_gvec_ands(unsigned vece, uint32_t dofs, uint32_t aofs,
TCGv_i64 c, uint32_t oprsz, uint32_t maxsz);
+void tcg_gen_gvec_andcs(unsigned vece, uint32_t dofs, uint32_t aofs,
+ TCGv_i64 c, uint32_t oprsz, uint32_t maxsz);
void tcg_gen_gvec_xors(unsigned vece, uint32_t dofs, uint32_t aofs,
TCGv_i64 c, uint32_t oprsz, uint32_t maxsz);
void tcg_gen_gvec_ors(unsigned vece, uint32_t dofs, uint32_t aofs,
diff --git a/accel/tcg/tcg-runtime-gvec.c b/accel/tcg/tcg-runtime-gvec.c
index ac7d28c251..97399493d5 100644
--- a/accel/tcg/tcg-runtime-gvec.c
+++ b/accel/tcg/tcg-runtime-gvec.c
@@ -550,6 +550,17 @@ void HELPER(gvec_ands)(void *d, void *a, uint64_t b, uint32_t desc)
clear_high(d, oprsz, desc);
}
+void HELPER(gvec_andcs)(void *d, void *a, uint64_t b, uint32_t desc)
+{
+ intptr_t oprsz = simd_oprsz(desc);
+ intptr_t i;
+
+ for (i = 0; i < oprsz; i += sizeof(uint64_t)) {
+ *(uint64_t *)(d + i) = *(uint64_t *)(a + i) & ~b;
+ }
+ clear_high(d, oprsz, desc);
+}
+
void HELPER(gvec_xors)(void *d, void *a, uint64_t b, uint32_t desc)
{
intptr_t oprsz = simd_oprsz(desc);
diff --git a/tcg/tcg-op-gvec.c b/tcg/tcg-op-gvec.c
index 047a832f44..9c14908a46 100644
--- a/tcg/tcg-op-gvec.c
+++ b/tcg/tcg-op-gvec.c
@@ -2761,6 +2761,23 @@ void tcg_gen_gvec_andi(unsigned vece, uint32_t dofs, uint32_t aofs,
tcg_gen_gvec_2s(dofs, aofs, oprsz, maxsz, tmp, &gop_ands);
}
+void tcg_gen_gvec_andcs(unsigned vece, uint32_t dofs, uint32_t aofs,
+ TCGv_i64 c, uint32_t oprsz, uint32_t maxsz)
+{
+ static GVecGen2s g = {
+ .fni8 = tcg_gen_andc_i64,
+ .fniv = tcg_gen_andc_vec,
+ .fno = gen_helper_gvec_andcs,
+ .prefer_i64 = TCG_TARGET_REG_BITS == 64,
+ .vece = MO_64
+ };
+
+ TCGv_i64 tmp = tcg_temp_ebb_new_i64();
+ tcg_gen_dup_i64(vece, tmp, c);
+ tcg_gen_gvec_2s(dofs, aofs, oprsz, maxsz, c, &g);
+ tcg_temp_free_i64(tmp);
+}
+
static const GVecGen2s gop_xors = {
.fni8 = tcg_gen_xor_i64,
.fniv = tcg_gen_xor_vec,
--
2.34.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH 05/84] tcg: Widen helper_atomic_* addresses to uint64_t
2023-05-03 7:20 [PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
` (8 preceding siblings ...)
2023-05-03 7:20 ` [PULL v2 05/12] tcg: Add tcg_gen_gvec_andcs Richard Henderson
@ 2023-05-03 7:20 ` Richard Henderson
2023-05-03 7:20 ` [PULL v2 06/12] tcg: Add tcg_gen_gvec_rotrs Richard Henderson
` (9 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Richard Henderson @ 2023-05-03 7:20 UTC (permalink / raw)
To: qemu-devel; +Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo
Always pass the target address as uint64_t.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
accel/tcg/tcg-runtime.h | 46 +++++++++++++++++------------------
tcg/tcg-op-ldst.c | 38 ++++++++++++++++++++---------
accel/tcg/atomic_common.c.inc | 14 +++++------
3 files changed, 57 insertions(+), 41 deletions(-)
diff --git a/accel/tcg/tcg-runtime.h b/accel/tcg/tcg-runtime.h
index 0e6c5f55fd..6f8c2061d0 100644
--- a/accel/tcg/tcg-runtime.h
+++ b/accel/tcg/tcg-runtime.h
@@ -43,61 +43,61 @@ DEF_HELPER_FLAGS_3(ld_i128, TCG_CALL_NO_WG, i128, env, i64, i32)
DEF_HELPER_FLAGS_4(st_i128, TCG_CALL_NO_WG, void, env, i64, i128, i32)
DEF_HELPER_FLAGS_5(atomic_cmpxchgb, TCG_CALL_NO_WG,
- i32, env, tl, i32, i32, i32)
+ i32, env, i64, i32, i32, i32)
DEF_HELPER_FLAGS_5(atomic_cmpxchgw_be, TCG_CALL_NO_WG,
- i32, env, tl, i32, i32, i32)
+ i32, env, i64, i32, i32, i32)
DEF_HELPER_FLAGS_5(atomic_cmpxchgw_le, TCG_CALL_NO_WG,
- i32, env, tl, i32, i32, i32)
+ i32, env, i64, i32, i32, i32)
DEF_HELPER_FLAGS_5(atomic_cmpxchgl_be, TCG_CALL_NO_WG,
- i32, env, tl, i32, i32, i32)
+ i32, env, i64, i32, i32, i32)
DEF_HELPER_FLAGS_5(atomic_cmpxchgl_le, TCG_CALL_NO_WG,
- i32, env, tl, i32, i32, i32)
+ i32, env, i64, i32, i32, i32)
#ifdef CONFIG_ATOMIC64
DEF_HELPER_FLAGS_5(atomic_cmpxchgq_be, TCG_CALL_NO_WG,
- i64, env, tl, i64, i64, i32)
+ i64, env, i64, i64, i64, i32)
DEF_HELPER_FLAGS_5(atomic_cmpxchgq_le, TCG_CALL_NO_WG,
- i64, env, tl, i64, i64, i32)
+ i64, env, i64, i64, i64, i32)
#endif
#ifdef CONFIG_CMPXCHG128
DEF_HELPER_FLAGS_5(atomic_cmpxchgo_be, TCG_CALL_NO_WG,
- i128, env, tl, i128, i128, i32)
+ i128, env, i64, i128, i128, i32)
DEF_HELPER_FLAGS_5(atomic_cmpxchgo_le, TCG_CALL_NO_WG,
- i128, env, tl, i128, i128, i32)
+ i128, env, i64, i128, i128, i32)
#endif
DEF_HELPER_FLAGS_5(nonatomic_cmpxchgo_be, TCG_CALL_NO_WG,
- i128, env, tl, i128, i128, i32)
+ i128, env, i64, i128, i128, i32)
DEF_HELPER_FLAGS_5(nonatomic_cmpxchgo_le, TCG_CALL_NO_WG,
- i128, env, tl, i128, i128, i32)
+ i128, env, i64, i128, i128, i32)
#ifdef CONFIG_ATOMIC64
#define GEN_ATOMIC_HELPERS(NAME) \
DEF_HELPER_FLAGS_4(glue(glue(atomic_, NAME), b), \
- TCG_CALL_NO_WG, i32, env, tl, i32, i32) \
+ TCG_CALL_NO_WG, i32, env, i64, i32, i32) \
DEF_HELPER_FLAGS_4(glue(glue(atomic_, NAME), w_le), \
- TCG_CALL_NO_WG, i32, env, tl, i32, i32) \
+ TCG_CALL_NO_WG, i32, env, i64, i32, i32) \
DEF_HELPER_FLAGS_4(glue(glue(atomic_, NAME), w_be), \
- TCG_CALL_NO_WG, i32, env, tl, i32, i32) \
+ TCG_CALL_NO_WG, i32, env, i64, i32, i32) \
DEF_HELPER_FLAGS_4(glue(glue(atomic_, NAME), l_le), \
- TCG_CALL_NO_WG, i32, env, tl, i32, i32) \
+ TCG_CALL_NO_WG, i32, env, i64, i32, i32) \
DEF_HELPER_FLAGS_4(glue(glue(atomic_, NAME), l_be), \
- TCG_CALL_NO_WG, i32, env, tl, i32, i32) \
+ TCG_CALL_NO_WG, i32, env, i64, i32, i32) \
DEF_HELPER_FLAGS_4(glue(glue(atomic_, NAME), q_le), \
- TCG_CALL_NO_WG, i64, env, tl, i64, i32) \
+ TCG_CALL_NO_WG, i64, env, i64, i64, i32) \
DEF_HELPER_FLAGS_4(glue(glue(atomic_, NAME), q_be), \
- TCG_CALL_NO_WG, i64, env, tl, i64, i32)
+ TCG_CALL_NO_WG, i64, env, i64, i64, i32)
#else
#define GEN_ATOMIC_HELPERS(NAME) \
DEF_HELPER_FLAGS_4(glue(glue(atomic_, NAME), b), \
- TCG_CALL_NO_WG, i32, env, tl, i32, i32) \
+ TCG_CALL_NO_WG, i32, env, i64, i32, i32) \
DEF_HELPER_FLAGS_4(glue(glue(atomic_, NAME), w_le), \
- TCG_CALL_NO_WG, i32, env, tl, i32, i32) \
+ TCG_CALL_NO_WG, i32, env, i64, i32, i32) \
DEF_HELPER_FLAGS_4(glue(glue(atomic_, NAME), w_be), \
- TCG_CALL_NO_WG, i32, env, tl, i32, i32) \
+ TCG_CALL_NO_WG, i32, env, i64, i32, i32) \
DEF_HELPER_FLAGS_4(glue(glue(atomic_, NAME), l_le), \
- TCG_CALL_NO_WG, i32, env, tl, i32, i32) \
+ TCG_CALL_NO_WG, i32, env, i64, i32, i32) \
DEF_HELPER_FLAGS_4(glue(glue(atomic_, NAME), l_be), \
- TCG_CALL_NO_WG, i32, env, tl, i32, i32)
+ TCG_CALL_NO_WG, i32, env, i64, i32, i32)
#endif /* CONFIG_ATOMIC64 */
GEN_ATOMIC_HELPERS(fetch_add)
diff --git a/tcg/tcg-op-ldst.c b/tcg/tcg-op-ldst.c
index 0010afb1a1..17fe35b93c 100644
--- a/tcg/tcg-op-ldst.c
+++ b/tcg/tcg-op-ldst.c
@@ -634,15 +634,15 @@ static void tcg_gen_ext_i64(TCGv_i64 ret, TCGv_i64 val, MemOp opc)
}
}
-typedef void (*gen_atomic_cx_i32)(TCGv_i32, TCGv_env, TCGv,
+typedef void (*gen_atomic_cx_i32)(TCGv_i32, TCGv_env, TCGv_i64,
TCGv_i32, TCGv_i32, TCGv_i32);
-typedef void (*gen_atomic_cx_i64)(TCGv_i64, TCGv_env, TCGv,
+typedef void (*gen_atomic_cx_i64)(TCGv_i64, TCGv_env, TCGv_i64,
TCGv_i64, TCGv_i64, TCGv_i32);
-typedef void (*gen_atomic_cx_i128)(TCGv_i128, TCGv_env, TCGv,
+typedef void (*gen_atomic_cx_i128)(TCGv_i128, TCGv_env, TCGv_i64,
TCGv_i128, TCGv_i128, TCGv_i32);
-typedef void (*gen_atomic_op_i32)(TCGv_i32, TCGv_env, TCGv,
+typedef void (*gen_atomic_op_i32)(TCGv_i32, TCGv_env, TCGv_i64,
TCGv_i32, TCGv_i32);
-typedef void (*gen_atomic_op_i64)(TCGv_i64, TCGv_env, TCGv,
+typedef void (*gen_atomic_op_i64)(TCGv_i64, TCGv_env, TCGv_i64,
TCGv_i64, TCGv_i32);
#ifdef CONFIG_ATOMIC64
@@ -693,6 +693,7 @@ void tcg_gen_atomic_cmpxchg_i32(TCGv_i32 retv, TCGv addr, TCGv_i32 cmpv,
TCGv_i32 newv, TCGArg idx, MemOp memop)
{
gen_atomic_cx_i32 gen;
+ TCGv_i64 a64;
MemOpIdx oi;
if (!(tcg_ctx->gen_tb->cflags & CF_PARALLEL)) {
@@ -705,7 +706,9 @@ void tcg_gen_atomic_cmpxchg_i32(TCGv_i32 retv, TCGv addr, TCGv_i32 cmpv,
tcg_debug_assert(gen != NULL);
oi = make_memop_idx(memop & ~MO_SIGN, idx);
- gen(retv, cpu_env, addr, cmpv, newv, tcg_constant_i32(oi));
+ a64 = maybe_extend_addr64(addr);
+ gen(retv, cpu_env, a64, cmpv, newv, tcg_constant_i32(oi));
+ maybe_free_addr64(a64);
if (memop & MO_SIGN) {
tcg_gen_ext_i32(retv, retv, memop);
@@ -761,7 +764,9 @@ void tcg_gen_atomic_cmpxchg_i64(TCGv_i64 retv, TCGv addr, TCGv_i64 cmpv,
gen = table_cmpxchg[memop & (MO_SIZE | MO_BSWAP)];
if (gen) {
MemOpIdx oi = make_memop_idx(memop, idx);
- gen(retv, cpu_env, addr, cmpv, newv, tcg_constant_i32(oi));
+ TCGv_i64 a64 = maybe_extend_addr64(addr);
+ gen(retv, cpu_env, a64, cmpv, newv, tcg_constant_i32(oi));
+ maybe_free_addr64(a64);
return;
}
@@ -813,11 +818,14 @@ void tcg_gen_nonatomic_cmpxchg_i128(TCGv_i128 retv, TCGv addr, TCGv_i128 cmpv,
? gen_helper_nonatomic_cmpxchgo_le
: gen_helper_nonatomic_cmpxchgo_be);
MemOpIdx oi = make_memop_idx(memop, idx);
+ TCGv_i64 a64;
tcg_debug_assert((memop & MO_SIZE) == MO_128);
tcg_debug_assert((memop & MO_SIGN) == 0);
- gen(retv, cpu_env, addr, cmpv, newv, tcg_constant_i32(oi));
+ a64 = maybe_extend_addr64(addr);
+ gen(retv, cpu_env, a64, cmpv, newv, tcg_constant_i32(oi));
+ maybe_free_addr64(a64);
} else {
TCGv_i128 oldv = tcg_temp_ebb_new_i128();
TCGv_i128 tmpv = tcg_temp_ebb_new_i128();
@@ -865,7 +873,9 @@ void tcg_gen_atomic_cmpxchg_i128(TCGv_i128 retv, TCGv addr, TCGv_i128 cmpv,
if (gen) {
MemOpIdx oi = make_memop_idx(memop, idx);
- gen(retv, cpu_env, addr, cmpv, newv, tcg_constant_i32(oi));
+ TCGv_i64 a64 = maybe_extend_addr64(addr);
+ gen(retv, cpu_env, a64, cmpv, newv, tcg_constant_i32(oi));
+ maybe_free_addr64(a64);
return;
}
@@ -903,6 +913,7 @@ static void do_atomic_op_i32(TCGv_i32 ret, TCGv addr, TCGv_i32 val,
TCGArg idx, MemOp memop, void * const table[])
{
gen_atomic_op_i32 gen;
+ TCGv_i64 a64;
MemOpIdx oi;
memop = tcg_canonicalize_memop(memop, 0, 0);
@@ -911,7 +922,9 @@ static void do_atomic_op_i32(TCGv_i32 ret, TCGv addr, TCGv_i32 val,
tcg_debug_assert(gen != NULL);
oi = make_memop_idx(memop & ~MO_SIGN, idx);
- gen(ret, cpu_env, addr, val, tcg_constant_i32(oi));
+ a64 = maybe_extend_addr64(addr);
+ gen(ret, cpu_env, a64, val, tcg_constant_i32(oi));
+ maybe_free_addr64(a64);
if (memop & MO_SIGN) {
tcg_gen_ext_i32(ret, ret, memop);
@@ -945,13 +958,16 @@ static void do_atomic_op_i64(TCGv_i64 ret, TCGv addr, TCGv_i64 val,
if ((memop & MO_SIZE) == MO_64) {
#ifdef CONFIG_ATOMIC64
gen_atomic_op_i64 gen;
+ TCGv_i64 a64;
MemOpIdx oi;
gen = table[memop & (MO_SIZE | MO_BSWAP)];
tcg_debug_assert(gen != NULL);
oi = make_memop_idx(memop & ~MO_SIGN, idx);
- gen(ret, cpu_env, addr, val, tcg_constant_i32(oi));
+ a64 = maybe_extend_addr64(addr);
+ gen(ret, cpu_env, a64, val, tcg_constant_i32(oi));
+ maybe_free_addr64(a64);
#else
gen_helper_exit_atomic(cpu_env);
/* Produce a result, so that we have a well-formed opcode stream
diff --git a/accel/tcg/atomic_common.c.inc b/accel/tcg/atomic_common.c.inc
index 8f2ce43ee6..fe0eea018f 100644
--- a/accel/tcg/atomic_common.c.inc
+++ b/accel/tcg/atomic_common.c.inc
@@ -13,20 +13,20 @@
* See the COPYING file in the top-level directory.
*/
-static void atomic_trace_rmw_post(CPUArchState *env, target_ulong addr,
+static void atomic_trace_rmw_post(CPUArchState *env, uint64_t addr,
MemOpIdx oi)
{
qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_RW);
}
#if HAVE_ATOMIC128
-static void atomic_trace_ld_post(CPUArchState *env, target_ulong addr,
+static void atomic_trace_ld_post(CPUArchState *env, uint64_t addr,
MemOpIdx oi)
{
qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R);
}
-static void atomic_trace_st_post(CPUArchState *env, target_ulong addr,
+static void atomic_trace_st_post(CPUArchState *env, uint64_t addr,
MemOpIdx oi)
{
qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W);
@@ -40,7 +40,7 @@ static void atomic_trace_st_post(CPUArchState *env, target_ulong addr,
*/
#define CMPXCHG_HELPER(OP, TYPE) \
- TYPE HELPER(atomic_##OP)(CPUArchState *env, target_ulong addr, \
+ TYPE HELPER(atomic_##OP)(CPUArchState *env, uint64_t addr, \
TYPE oldv, TYPE newv, uint32_t oi) \
{ return cpu_atomic_##OP##_mmu(env, addr, oldv, newv, oi, GETPC()); }
@@ -62,7 +62,7 @@ CMPXCHG_HELPER(cmpxchgo_le, Int128)
#undef CMPXCHG_HELPER
-Int128 HELPER(nonatomic_cmpxchgo_be)(CPUArchState *env, target_ulong addr,
+Int128 HELPER(nonatomic_cmpxchgo_be)(CPUArchState *env, uint64_t addr,
Int128 cmpv, Int128 newv, uint32_t oi)
{
#if TCG_TARGET_REG_BITS == 32
@@ -82,7 +82,7 @@ Int128 HELPER(nonatomic_cmpxchgo_be)(CPUArchState *env, target_ulong addr,
#endif
}
-Int128 HELPER(nonatomic_cmpxchgo_le)(CPUArchState *env, target_ulong addr,
+Int128 HELPER(nonatomic_cmpxchgo_le)(CPUArchState *env, uint64_t addr,
Int128 cmpv, Int128 newv, uint32_t oi)
{
#if TCG_TARGET_REG_BITS == 32
@@ -103,7 +103,7 @@ Int128 HELPER(nonatomic_cmpxchgo_le)(CPUArchState *env, target_ulong addr,
}
#define ATOMIC_HELPER(OP, TYPE) \
- TYPE HELPER(glue(atomic_,OP))(CPUArchState *env, target_ulong addr, \
+ TYPE HELPER(glue(atomic_,OP))(CPUArchState *env, uint64_t addr, \
TYPE val, uint32_t oi) \
{ return glue(glue(cpu_atomic_,OP),_mmu)(env, addr, val, oi, GETPC()); }
--
2.34.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PULL v2 06/12] tcg: Add tcg_gen_gvec_rotrs
2023-05-03 7:20 [PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
` (9 preceding siblings ...)
2023-05-03 7:20 ` [PATCH 05/84] tcg: Widen helper_atomic_* addresses to uint64_t Richard Henderson
@ 2023-05-03 7:20 ` Richard Henderson
2023-05-03 7:20 ` [PATCH 06/84] tcg: Widen tcg_gen_code pc_start argument to uint64_t Richard Henderson
` (8 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Richard Henderson @ 2023-05-03 7:20 UTC (permalink / raw)
To: qemu-devel
Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo, Nazar Kazakov
From: Nazar Kazakov <nazar.kazakov@codethink.co.uk>
Add tcg expander and helper functions for rotate right
vector with scalar operand.
Signed-off-by: Nazar Kazakov <nazar.kazakov@codethink.co.uk>
Message-Id: <20230428144757.57530-10-lawrence.hunter@codethink.co.uk>
[rth: Split out of larger patch; mask rotation count.]
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
include/tcg/tcg-op-gvec.h | 2 ++
tcg/tcg-op-gvec.c | 11 +++++++++++
2 files changed, 13 insertions(+)
diff --git a/include/tcg/tcg-op-gvec.h b/include/tcg/tcg-op-gvec.h
index 6d58683171..a8183bfeab 100644
--- a/include/tcg/tcg-op-gvec.h
+++ b/include/tcg/tcg-op-gvec.h
@@ -371,6 +371,8 @@ void tcg_gen_gvec_sars(unsigned vece, uint32_t dofs, uint32_t aofs,
TCGv_i32 shift, uint32_t oprsz, uint32_t maxsz);
void tcg_gen_gvec_rotls(unsigned vece, uint32_t dofs, uint32_t aofs,
TCGv_i32 shift, uint32_t oprsz, uint32_t maxsz);
+void tcg_gen_gvec_rotrs(unsigned vece, uint32_t dofs, uint32_t aofs,
+ TCGv_i32 shift, uint32_t oprsz, uint32_t maxsz);
/*
* Perform vector shift by vector element, modulo the element size.
diff --git a/tcg/tcg-op-gvec.c b/tcg/tcg-op-gvec.c
index 9c14908a46..f51bcaa87b 100644
--- a/tcg/tcg-op-gvec.c
+++ b/tcg/tcg-op-gvec.c
@@ -3353,6 +3353,17 @@ void tcg_gen_gvec_rotls(unsigned vece, uint32_t dofs, uint32_t aofs,
do_gvec_shifts(vece, dofs, aofs, shift, oprsz, maxsz, &g);
}
+void tcg_gen_gvec_rotrs(unsigned vece, uint32_t dofs, uint32_t aofs,
+ TCGv_i32 shift, uint32_t oprsz, uint32_t maxsz)
+{
+ TCGv_i32 tmp = tcg_temp_ebb_new_i32();
+
+ tcg_gen_neg_i32(tmp, shift);
+ tcg_gen_andi_i32(tmp, tmp, (8 << vece) - 1);
+ tcg_gen_gvec_rotls(vece, dofs, aofs, tmp, oprsz, maxsz);
+ tcg_temp_free_i32(tmp);
+}
+
/*
* Expand D = A << (B % element bits)
*
--
2.34.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH 06/84] tcg: Widen tcg_gen_code pc_start argument to uint64_t
2023-05-03 7:20 [PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
` (10 preceding siblings ...)
2023-05-03 7:20 ` [PULL v2 06/12] tcg: Add tcg_gen_gvec_rotrs Richard Henderson
@ 2023-05-03 7:20 ` Richard Henderson
2023-05-03 7:20 ` [PATCH 07/84] accel/tcg: Merge gen_mem_wrapped with plugin_gen_empty_mem_callback Richard Henderson
` (7 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Richard Henderson @ 2023-05-03 7:20 UTC (permalink / raw)
To: qemu-devel; +Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
include/tcg/tcg.h | 2 +-
tcg/tcg.c | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/include/tcg/tcg.h b/include/tcg/tcg.h
index 7c6a613364..7d6df5eabe 100644
--- a/include/tcg/tcg.h
+++ b/include/tcg/tcg.h
@@ -852,7 +852,7 @@ void tcg_register_thread(void);
void tcg_prologue_init(TCGContext *s);
void tcg_func_start(TCGContext *s);
-int tcg_gen_code(TCGContext *s, TranslationBlock *tb, target_ulong pc_start);
+int tcg_gen_code(TCGContext *s, TranslationBlock *tb, uint64_t pc_start);
void tb_target_set_jmp_target(const TranslationBlock *, int,
uintptr_t, uintptr_t);
diff --git a/tcg/tcg.c b/tcg/tcg.c
index 685e555754..b4e1e0e612 100644
--- a/tcg/tcg.c
+++ b/tcg/tcg.c
@@ -5808,7 +5808,7 @@ int64_t tcg_cpu_exec_time(void)
#endif
-int tcg_gen_code(TCGContext *s, TranslationBlock *tb, target_ulong pc_start)
+int tcg_gen_code(TCGContext *s, TranslationBlock *tb, uint64_t pc_start)
{
#ifdef CONFIG_PROFILER
TCGProfile *prof = &s->prof;
--
2.34.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH 07/84] accel/tcg: Merge gen_mem_wrapped with plugin_gen_empty_mem_callback
2023-05-03 7:20 [PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
` (11 preceding siblings ...)
2023-05-03 7:20 ` [PATCH 06/84] tcg: Widen tcg_gen_code pc_start argument to uint64_t Richard Henderson
@ 2023-05-03 7:20 ` Richard Henderson
2023-05-03 7:20 ` [PULL v2 07/12] qemu/int128: Re-shuffle Int128Alias members Richard Henderson
` (6 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Richard Henderson @ 2023-05-03 7:20 UTC (permalink / raw)
To: qemu-devel; +Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo
As gen_mem_wrapped is only used in plugin_gen_empty_mem_callback,
we can avoid the curiosity of union mem_gen_fn by inlining it.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
accel/tcg/plugin-gen.c | 30 ++++++------------------------
1 file changed, 6 insertions(+), 24 deletions(-)
diff --git a/accel/tcg/plugin-gen.c b/accel/tcg/plugin-gen.c
index 5efb8db258..04facd6305 100644
--- a/accel/tcg/plugin-gen.c
+++ b/accel/tcg/plugin-gen.c
@@ -202,35 +202,17 @@ static void plugin_gen_empty_callback(enum plugin_gen_from from)
}
}
-union mem_gen_fn {
- void (*mem_fn)(TCGv, uint32_t);
- void (*inline_fn)(void);
-};
-
-static void gen_mem_wrapped(enum plugin_gen_cb type,
- const union mem_gen_fn *f, TCGv addr,
- uint32_t info, bool is_mem)
+void plugin_gen_empty_mem_callback(TCGv addr, uint32_t info)
{
enum qemu_plugin_mem_rw rw = get_plugin_meminfo_rw(info);
- gen_plugin_cb_start(PLUGIN_GEN_FROM_MEM, type, rw);
- if (is_mem) {
- f->mem_fn(addr, info);
- } else {
- f->inline_fn();
- }
+ gen_plugin_cb_start(PLUGIN_GEN_FROM_MEM, PLUGIN_GEN_CB_MEM, rw);
+ gen_empty_mem_cb(addr, info);
tcg_gen_plugin_cb_end();
-}
-void plugin_gen_empty_mem_callback(TCGv addr, uint32_t info)
-{
- union mem_gen_fn fn;
-
- fn.mem_fn = gen_empty_mem_cb;
- gen_mem_wrapped(PLUGIN_GEN_CB_MEM, &fn, addr, info, true);
-
- fn.inline_fn = gen_empty_inline_cb;
- gen_mem_wrapped(PLUGIN_GEN_CB_INLINE, &fn, 0, info, false);
+ gen_plugin_cb_start(PLUGIN_GEN_FROM_MEM, PLUGIN_GEN_CB_INLINE, rw);
+ gen_empty_inline_cb();
+ tcg_gen_plugin_cb_end();
}
static TCGOp *find_op(TCGOp *op, TCGOpcode opc)
--
2.34.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PULL v2 07/12] qemu/int128: Re-shuffle Int128Alias members
2023-05-03 7:20 [PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
` (12 preceding siblings ...)
2023-05-03 7:20 ` [PATCH 07/84] accel/tcg: Merge gen_mem_wrapped with plugin_gen_empty_mem_callback Richard Henderson
@ 2023-05-03 7:20 ` Richard Henderson
2023-05-03 7:21 ` [PATCH 08/84] accel/tcg: Merge do_gen_mem_cb into caller Richard Henderson
` (5 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Richard Henderson @ 2023-05-03 7:20 UTC (permalink / raw)
To: qemu-devel
Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo,
Alex Bennée
Clang 14, with --enable-tcg-interpreter errors with
include/qemu/int128.h:487:16: error: alignment of field 'i' (128 bits)
does not match the alignment of the first field in transparent union;
transparent_union attribute ignored [-Werror,-Wignored-attributes]
__int128_t i;
^
include/qemu/int128.h:486:12: note: alignment of first field is 64 bits
Int128 s;
^
1 error generated.
By placing the __uint128_t member first, this is avoided.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20230501204625.277361-1-richard.henderson@linaro.org>
---
include/qemu/int128.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/include/qemu/int128.h b/include/qemu/int128.h
index f62a46b48c..9e46cfaefc 100644
--- a/include/qemu/int128.h
+++ b/include/qemu/int128.h
@@ -483,9 +483,9 @@ static inline void bswap128s(Int128 *s)
*/
#ifdef CONFIG_INT128
typedef union {
- Int128 s;
- __int128_t i;
__uint128_t u;
+ __int128_t i;
+ Int128 s;
} Int128Alias __attribute__((transparent_union));
#else
typedef Int128 Int128Alias;
--
2.34.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH 08/84] accel/tcg: Merge do_gen_mem_cb into caller
2023-05-03 7:20 [PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
` (13 preceding siblings ...)
2023-05-03 7:20 ` [PULL v2 07/12] qemu/int128: Re-shuffle Int128Alias members Richard Henderson
@ 2023-05-03 7:21 ` Richard Henderson
2023-05-03 7:21 ` [PULL v2 08/12] migration/xbzrle: Use __attribute__((target)) for avx512 Richard Henderson
` (4 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Richard Henderson @ 2023-05-03 7:21 UTC (permalink / raw)
To: qemu-devel; +Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo
As do_gen_mem_cb is called once, merge it into gen_empty_mem_cb.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
accel/tcg/plugin-gen.c | 41 ++++++++++++++++++-----------------------
1 file changed, 18 insertions(+), 23 deletions(-)
diff --git a/accel/tcg/plugin-gen.c b/accel/tcg/plugin-gen.c
index 04facd6305..55e892b684 100644
--- a/accel/tcg/plugin-gen.c
+++ b/accel/tcg/plugin-gen.c
@@ -92,27 +92,6 @@ void HELPER(plugin_vcpu_mem_cb)(unsigned int vcpu_index,
void *userdata)
{ }
-static void do_gen_mem_cb(TCGv vaddr, uint32_t info)
-{
- TCGv_i32 cpu_index = tcg_temp_ebb_new_i32();
- TCGv_i32 meminfo = tcg_temp_ebb_new_i32();
- TCGv_i64 vaddr64 = tcg_temp_ebb_new_i64();
- TCGv_ptr udata = tcg_temp_ebb_new_ptr();
-
- tcg_gen_movi_i32(meminfo, info);
- tcg_gen_movi_ptr(udata, 0);
- tcg_gen_ld_i32(cpu_index, cpu_env,
- -offsetof(ArchCPU, env) + offsetof(CPUState, cpu_index));
- tcg_gen_extu_tl_i64(vaddr64, vaddr);
-
- gen_helper_plugin_vcpu_mem_cb(cpu_index, meminfo, vaddr64, udata);
-
- tcg_temp_free_ptr(udata);
- tcg_temp_free_i64(vaddr64);
- tcg_temp_free_i32(meminfo);
- tcg_temp_free_i32(cpu_index);
-}
-
static void gen_empty_udata_cb(void)
{
TCGv_i32 cpu_index = tcg_temp_ebb_new_i32();
@@ -145,9 +124,25 @@ static void gen_empty_inline_cb(void)
tcg_temp_free_i64(val);
}
-static void gen_empty_mem_cb(TCGv addr, uint32_t info)
+static void gen_empty_mem_cb(TCGv vaddr, uint32_t info)
{
- do_gen_mem_cb(addr, info);
+ TCGv_i32 cpu_index = tcg_temp_ebb_new_i32();
+ TCGv_i32 meminfo = tcg_temp_ebb_new_i32();
+ TCGv_i64 vaddr64 = tcg_temp_ebb_new_i64();
+ TCGv_ptr udata = tcg_temp_ebb_new_ptr();
+
+ tcg_gen_movi_i32(meminfo, info);
+ tcg_gen_movi_ptr(udata, 0);
+ tcg_gen_ld_i32(cpu_index, cpu_env,
+ -offsetof(ArchCPU, env) + offsetof(CPUState, cpu_index));
+ tcg_gen_extu_tl_i64(vaddr64, vaddr);
+
+ gen_helper_plugin_vcpu_mem_cb(cpu_index, meminfo, vaddr64, udata);
+
+ tcg_temp_free_ptr(udata);
+ tcg_temp_free_i64(vaddr64);
+ tcg_temp_free_i32(meminfo);
+ tcg_temp_free_i32(cpu_index);
}
/*
--
2.34.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PULL v2 08/12] migration/xbzrle: Use __attribute__((target)) for avx512
2023-05-03 7:20 [PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
` (14 preceding siblings ...)
2023-05-03 7:21 ` [PATCH 08/84] accel/tcg: Merge do_gen_mem_cb into caller Richard Henderson
@ 2023-05-03 7:21 ` Richard Henderson
2023-05-03 7:21 ` [PULL v2 09/12] accel/tcg: Add cpu_ld*_code_mmu Richard Henderson
` (3 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Richard Henderson @ 2023-05-03 7:21 UTC (permalink / raw)
To: qemu-devel
Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo, Juan Quintela
Use the attribute, which is supported by clang, instead of
the #pragma, which is not supported and, for some reason,
also not detected by the meson probe, so we fail by -Werror.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Message-Id: <20230501210555.289806-1-richard.henderson@linaro.org>
---
meson.build | 5 +----
migration/xbzrle.c | 9 ++++-----
2 files changed, 5 insertions(+), 9 deletions(-)
diff --git a/meson.build b/meson.build
index c44d05a13f..77d42898c8 100644
--- a/meson.build
+++ b/meson.build
@@ -2370,12 +2370,9 @@ config_host_data.set('CONFIG_AVX512F_OPT', get_option('avx512f') \
config_host_data.set('CONFIG_AVX512BW_OPT', get_option('avx512bw') \
.require(have_cpuid_h, error_message: 'cpuid.h not available, cannot enable AVX512BW') \
.require(cc.links('''
- #pragma GCC push_options
- #pragma GCC target("avx512bw")
#include <cpuid.h>
#include <immintrin.h>
- static int bar(void *a) {
-
+ static int __attribute__((target("avx512bw"))) bar(void *a) {
__m512i *x = a;
__m512i res= _mm512_abs_epi8(*x);
return res[1];
diff --git a/migration/xbzrle.c b/migration/xbzrle.c
index c6f8b20917..258e4959c9 100644
--- a/migration/xbzrle.c
+++ b/migration/xbzrle.c
@@ -177,11 +177,11 @@ int xbzrle_decode_buffer(uint8_t *src, int slen, uint8_t *dst, int dlen)
}
#if defined(CONFIG_AVX512BW_OPT)
-#pragma GCC push_options
-#pragma GCC target("avx512bw")
#include <immintrin.h>
-int xbzrle_encode_buffer_avx512(uint8_t *old_buf, uint8_t *new_buf, int slen,
- uint8_t *dst, int dlen)
+
+int __attribute__((target("avx512bw")))
+xbzrle_encode_buffer_avx512(uint8_t *old_buf, uint8_t *new_buf, int slen,
+ uint8_t *dst, int dlen)
{
uint32_t zrun_len = 0, nzrun_len = 0;
int d = 0, i = 0, num = 0;
@@ -296,5 +296,4 @@ int xbzrle_encode_buffer_avx512(uint8_t *old_buf, uint8_t *new_buf, int slen,
}
return d;
}
-#pragma GCC pop_options
#endif
--
2.34.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PULL v2 09/12] accel/tcg: Add cpu_ld*_code_mmu
2023-05-03 7:20 [PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
` (15 preceding siblings ...)
2023-05-03 7:21 ` [PULL v2 08/12] migration/xbzrle: Use __attribute__((target)) for avx512 Richard Henderson
@ 2023-05-03 7:21 ` Richard Henderson
2023-05-03 7:21 ` [PATCH 09/84] tcg: Reduce copies for plugin_gen_mem_callbacks Richard Henderson
` (2 subsequent siblings)
19 siblings, 0 replies; 24+ messages in thread
From: Richard Henderson @ 2023-05-03 7:21 UTC (permalink / raw)
To: qemu-devel
Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo,
Alistair Francis, Weiwei Li, Daniel Henrique Barboza
At least RISC-V has the need to be able to perform a read
using execute permissions, outside of translation.
Add helpers to facilitate this.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Acked-by: Alistair Francis <alistair.francis@wdc.com>
Reviewed-by: Weiwei Li <liweiwei@iscas.ac.cn>
Tested-by: Daniel Henrique Barboza <dbarboza@ventanamicro.com>
Message-Id: <20230325105429.1142530-9-richard.henderson@linaro.org>
Message-Id: <20230412114333.118895-9-richard.henderson@linaro.org>
---
include/exec/cpu_ldst.h | 9 +++++++
accel/tcg/cputlb.c | 48 ++++++++++++++++++++++++++++++++++
accel/tcg/user-exec.c | 58 +++++++++++++++++++++++++++++++++++++++++
3 files changed, 115 insertions(+)
diff --git a/include/exec/cpu_ldst.h b/include/exec/cpu_ldst.h
index 09b55cc0ee..c141f0394f 100644
--- a/include/exec/cpu_ldst.h
+++ b/include/exec/cpu_ldst.h
@@ -445,6 +445,15 @@ static inline CPUTLBEntry *tlb_entry(CPUArchState *env, uintptr_t mmu_idx,
# define cpu_stq_mmu cpu_stq_le_mmu
#endif
+uint8_t cpu_ldb_code_mmu(CPUArchState *env, abi_ptr addr,
+ MemOpIdx oi, uintptr_t ra);
+uint16_t cpu_ldw_code_mmu(CPUArchState *env, abi_ptr addr,
+ MemOpIdx oi, uintptr_t ra);
+uint32_t cpu_ldl_code_mmu(CPUArchState *env, abi_ptr addr,
+ MemOpIdx oi, uintptr_t ra);
+uint64_t cpu_ldq_code_mmu(CPUArchState *env, abi_ptr addr,
+ MemOpIdx oi, uintptr_t ra);
+
uint32_t cpu_ldub_code(CPUArchState *env, abi_ptr addr);
uint32_t cpu_lduw_code(CPUArchState *env, abi_ptr addr);
uint32_t cpu_ldl_code(CPUArchState *env, abi_ptr addr);
diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index efa0cb67c9..c8bd642d0e 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -2773,3 +2773,51 @@ uint64_t cpu_ldq_code(CPUArchState *env, abi_ptr addr)
MemOpIdx oi = make_memop_idx(MO_TEUQ, cpu_mmu_index(env, true));
return full_ldq_code(env, addr, oi, 0);
}
+
+uint8_t cpu_ldb_code_mmu(CPUArchState *env, abi_ptr addr,
+ MemOpIdx oi, uintptr_t retaddr)
+{
+ return full_ldub_code(env, addr, oi, retaddr);
+}
+
+uint16_t cpu_ldw_code_mmu(CPUArchState *env, abi_ptr addr,
+ MemOpIdx oi, uintptr_t retaddr)
+{
+ MemOp mop = get_memop(oi);
+ int idx = get_mmuidx(oi);
+ uint16_t ret;
+
+ ret = full_lduw_code(env, addr, make_memop_idx(MO_TEUW, idx), retaddr);
+ if ((mop & MO_BSWAP) != MO_TE) {
+ ret = bswap16(ret);
+ }
+ return ret;
+}
+
+uint32_t cpu_ldl_code_mmu(CPUArchState *env, abi_ptr addr,
+ MemOpIdx oi, uintptr_t retaddr)
+{
+ MemOp mop = get_memop(oi);
+ int idx = get_mmuidx(oi);
+ uint32_t ret;
+
+ ret = full_ldl_code(env, addr, make_memop_idx(MO_TEUL, idx), retaddr);
+ if ((mop & MO_BSWAP) != MO_TE) {
+ ret = bswap32(ret);
+ }
+ return ret;
+}
+
+uint64_t cpu_ldq_code_mmu(CPUArchState *env, abi_ptr addr,
+ MemOpIdx oi, uintptr_t retaddr)
+{
+ MemOp mop = get_memop(oi);
+ int idx = get_mmuidx(oi);
+ uint64_t ret;
+
+ ret = full_ldq_code(env, addr, make_memop_idx(MO_TEUQ, idx), retaddr);
+ if ((mop & MO_BSWAP) != MO_TE) {
+ ret = bswap64(ret);
+ }
+ return ret;
+}
diff --git a/accel/tcg/user-exec.c b/accel/tcg/user-exec.c
index a7e0c3e2f4..fc597a010d 100644
--- a/accel/tcg/user-exec.c
+++ b/accel/tcg/user-exec.c
@@ -1219,6 +1219,64 @@ uint64_t cpu_ldq_code(CPUArchState *env, abi_ptr ptr)
return ret;
}
+uint8_t cpu_ldb_code_mmu(CPUArchState *env, abi_ptr addr,
+ MemOpIdx oi, uintptr_t ra)
+{
+ void *haddr;
+ uint8_t ret;
+
+ haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_INST_FETCH);
+ ret = ldub_p(haddr);
+ clear_helper_retaddr();
+ return ret;
+}
+
+uint16_t cpu_ldw_code_mmu(CPUArchState *env, abi_ptr addr,
+ MemOpIdx oi, uintptr_t ra)
+{
+ void *haddr;
+ uint16_t ret;
+
+ haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_INST_FETCH);
+ ret = lduw_p(haddr);
+ clear_helper_retaddr();
+ if (get_memop(oi) & MO_BSWAP) {
+ ret = bswap16(ret);
+ }
+ return ret;
+}
+
+uint32_t cpu_ldl_code_mmu(CPUArchState *env, abi_ptr addr,
+ MemOpIdx oi, uintptr_t ra)
+{
+ void *haddr;
+ uint32_t ret;
+
+ haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_INST_FETCH);
+ ret = ldl_p(haddr);
+ clear_helper_retaddr();
+ if (get_memop(oi) & MO_BSWAP) {
+ ret = bswap32(ret);
+ }
+ return ret;
+}
+
+uint64_t cpu_ldq_code_mmu(CPUArchState *env, abi_ptr addr,
+ MemOpIdx oi, uintptr_t ra)
+{
+ void *haddr;
+ uint64_t ret;
+
+ validate_memop(oi, MO_BEUQ);
+ haddr = cpu_mmu_lookup(env, addr, oi, ra, MMU_DATA_LOAD);
+ ret = ldq_p(haddr);
+ clear_helper_retaddr();
+ if (get_memop(oi) & MO_BSWAP) {
+ ret = bswap64(ret);
+ }
+ return ret;
+}
+
#include "ldst_common.c.inc"
/*
--
2.34.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH 09/84] tcg: Reduce copies for plugin_gen_mem_callbacks
2023-05-03 7:20 [PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
` (16 preceding siblings ...)
2023-05-03 7:21 ` [PULL v2 09/12] accel/tcg: Add cpu_ld*_code_mmu Richard Henderson
@ 2023-05-03 7:21 ` Richard Henderson
2023-05-03 7:21 ` [PATCH 10/84] accel/tcg: Widen plugin_gen_empty_mem_callback to i64 Richard Henderson
2023-05-03 7:21 ` [PULL v2 10/12] tcg/loongarch64: Conditionalize tcg_out_exts_i32_i64 Richard Henderson
19 siblings, 0 replies; 24+ messages in thread
From: Richard Henderson @ 2023-05-03 7:21 UTC (permalink / raw)
To: qemu-devel; +Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo
We only need to make copies for loads, when the destination
overlaps the address. For now, only eliminate the copy for
stores and 128-bit loads.
Rename plugin_prep_mem_callbacks to plugin_maybe_preserve_addr,
returning NULL if no copy is made.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
tcg/tcg-op-ldst.c | 38 ++++++++++++++++++++------------------
1 file changed, 20 insertions(+), 18 deletions(-)
diff --git a/tcg/tcg-op-ldst.c b/tcg/tcg-op-ldst.c
index 17fe35b93c..cbd85f793c 100644
--- a/tcg/tcg-op-ldst.c
+++ b/tcg/tcg-op-ldst.c
@@ -114,7 +114,8 @@ static void tcg_gen_req_mo(TCGBar type)
}
}
-static inline TCGv plugin_prep_mem_callbacks(TCGv vaddr)
+/* Only required for loads, where value might overlap addr. */
+static TCGv plugin_maybe_preserve_addr(TCGv vaddr)
{
#ifdef CONFIG_PLUGIN
if (tcg_ctx->plugin_insn != NULL) {
@@ -124,17 +125,20 @@ static inline TCGv plugin_prep_mem_callbacks(TCGv vaddr)
return temp;
}
#endif
- return vaddr;
+ return NULL;
}
-static void plugin_gen_mem_callbacks(TCGv vaddr, MemOpIdx oi,
- enum qemu_plugin_mem_rw rw)
+static void
+plugin_gen_mem_callbacks(TCGv copy_addr, TCGv orig_addr, MemOpIdx oi,
+ enum qemu_plugin_mem_rw rw)
{
#ifdef CONFIG_PLUGIN
if (tcg_ctx->plugin_insn != NULL) {
qemu_plugin_meminfo_t info = make_plugin_meminfo(oi, rw);
- plugin_gen_empty_mem_callback(vaddr, info);
- tcg_temp_free(vaddr);
+ plugin_gen_empty_mem_callback(copy_addr ? : orig_addr, info);
+ if (copy_addr) {
+ tcg_temp_free(copy_addr);
+ }
}
#endif
}
@@ -143,6 +147,7 @@ void tcg_gen_qemu_ld_i32(TCGv_i32 val, TCGv addr, TCGArg idx, MemOp memop)
{
MemOp orig_memop;
MemOpIdx oi;
+ TCGv copy_addr;
tcg_gen_req_mo(TCG_MO_LD_LD | TCG_MO_ST_LD);
memop = tcg_canonicalize_memop(memop, 0, 0);
@@ -157,9 +162,9 @@ void tcg_gen_qemu_ld_i32(TCGv_i32 val, TCGv addr, TCGArg idx, MemOp memop)
}
}
- addr = plugin_prep_mem_callbacks(addr);
+ copy_addr = plugin_maybe_preserve_addr(addr);
gen_ldst_i32(INDEX_op_qemu_ld_i32, val, addr, memop, idx);
- plugin_gen_mem_callbacks(addr, oi, QEMU_PLUGIN_MEM_R);
+ plugin_gen_mem_callbacks(copy_addr, addr, oi, QEMU_PLUGIN_MEM_R);
if ((orig_memop ^ memop) & MO_BSWAP) {
switch (orig_memop & MO_SIZE) {
@@ -202,13 +207,12 @@ void tcg_gen_qemu_st_i32(TCGv_i32 val, TCGv addr, TCGArg idx, MemOp memop)
memop &= ~MO_BSWAP;
}
- addr = plugin_prep_mem_callbacks(addr);
if (TCG_TARGET_HAS_qemu_st8_i32 && (memop & MO_SIZE) == MO_8) {
gen_ldst_i32(INDEX_op_qemu_st8_i32, val, addr, memop, idx);
} else {
gen_ldst_i32(INDEX_op_qemu_st_i32, val, addr, memop, idx);
}
- plugin_gen_mem_callbacks(addr, oi, QEMU_PLUGIN_MEM_W);
+ plugin_gen_mem_callbacks(NULL, addr, oi, QEMU_PLUGIN_MEM_W);
if (swap) {
tcg_temp_free_i32(swap);
@@ -219,6 +223,7 @@ void tcg_gen_qemu_ld_i64(TCGv_i64 val, TCGv addr, TCGArg idx, MemOp memop)
{
MemOp orig_memop;
MemOpIdx oi;
+ TCGv copy_addr;
if (TCG_TARGET_REG_BITS == 32 && (memop & MO_SIZE) < MO_64) {
tcg_gen_qemu_ld_i32(TCGV_LOW(val), addr, idx, memop);
@@ -243,9 +248,9 @@ void tcg_gen_qemu_ld_i64(TCGv_i64 val, TCGv addr, TCGArg idx, MemOp memop)
}
}
- addr = plugin_prep_mem_callbacks(addr);
+ copy_addr = plugin_maybe_preserve_addr(addr);
gen_ldst_i64(INDEX_op_qemu_ld_i64, val, addr, memop, idx);
- plugin_gen_mem_callbacks(addr, oi, QEMU_PLUGIN_MEM_R);
+ plugin_gen_mem_callbacks(copy_addr, addr, oi, QEMU_PLUGIN_MEM_R);
if ((orig_memop ^ memop) & MO_BSWAP) {
int flags = (orig_memop & MO_SIGN
@@ -300,9 +305,8 @@ void tcg_gen_qemu_st_i64(TCGv_i64 val, TCGv addr, TCGArg idx, MemOp memop)
memop &= ~MO_BSWAP;
}
- addr = plugin_prep_mem_callbacks(addr);
gen_ldst_i64(INDEX_op_qemu_st_i64, val, addr, memop, idx);
- plugin_gen_mem_callbacks(addr, oi, QEMU_PLUGIN_MEM_W);
+ plugin_gen_mem_callbacks(NULL, addr, oi, QEMU_PLUGIN_MEM_W);
if (swap) {
tcg_temp_free_i64(swap);
@@ -430,7 +434,6 @@ void tcg_gen_qemu_ld_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop)
tcg_debug_assert((memop & MO_SIGN) == 0);
tcg_gen_req_mo(TCG_MO_LD_LD | TCG_MO_ST_LD);
- addr = plugin_prep_mem_callbacks(addr);
/* TODO: For now, force 32-bit hosts to use the helper. */
if (TCG_TARGET_HAS_qemu_ldst_i128 && TCG_TARGET_REG_BITS == 64) {
@@ -501,7 +504,7 @@ void tcg_gen_qemu_ld_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop)
maybe_free_addr64(a64);
}
- plugin_gen_mem_callbacks(addr, oi, QEMU_PLUGIN_MEM_R);
+ plugin_gen_mem_callbacks(NULL, addr, oi, QEMU_PLUGIN_MEM_R);
}
void tcg_gen_qemu_st_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop)
@@ -512,7 +515,6 @@ void tcg_gen_qemu_st_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop)
tcg_debug_assert((memop & MO_SIGN) == 0);
tcg_gen_req_mo(TCG_MO_ST_LD | TCG_MO_ST_ST);
- addr = plugin_prep_mem_callbacks(addr);
/* TODO: For now, force 32-bit hosts to use the helper. */
@@ -583,7 +585,7 @@ void tcg_gen_qemu_st_i128(TCGv_i128 val, TCGv addr, TCGArg idx, MemOp memop)
maybe_free_addr64(a64);
}
- plugin_gen_mem_callbacks(addr, oi, QEMU_PLUGIN_MEM_W);
+ plugin_gen_mem_callbacks(NULL, addr, oi, QEMU_PLUGIN_MEM_W);
}
static void tcg_gen_ext_i32(TCGv_i32 ret, TCGv_i32 val, MemOp opc)
--
2.34.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH 10/84] accel/tcg: Widen plugin_gen_empty_mem_callback to i64
2023-05-03 7:20 [PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
` (17 preceding siblings ...)
2023-05-03 7:21 ` [PATCH 09/84] tcg: Reduce copies for plugin_gen_mem_callbacks Richard Henderson
@ 2023-05-03 7:21 ` Richard Henderson
2023-05-03 7:21 ` [PULL v2 10/12] tcg/loongarch64: Conditionalize tcg_out_exts_i32_i64 Richard Henderson
19 siblings, 0 replies; 24+ messages in thread
From: Richard Henderson @ 2023-05-03 7:21 UTC (permalink / raw)
To: qemu-devel; +Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo
Since we do this inside gen_empty_mem_cb anyway, let's
do this earlier inside tcg expansion.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
include/exec/plugin-gen.h | 4 ++--
accel/tcg/plugin-gen.c | 9 +++------
tcg/tcg-op-ldst.c | 28 ++++++++++++++++++++--------
3 files changed, 25 insertions(+), 16 deletions(-)
diff --git a/include/exec/plugin-gen.h b/include/exec/plugin-gen.h
index 5f5506f1cc..3af0168e65 100644
--- a/include/exec/plugin-gen.h
+++ b/include/exec/plugin-gen.h
@@ -27,7 +27,7 @@ void plugin_gen_insn_start(CPUState *cpu, const struct DisasContextBase *db);
void plugin_gen_insn_end(void);
void plugin_gen_disable_mem_helpers(void);
-void plugin_gen_empty_mem_callback(TCGv addr, uint32_t info);
+void plugin_gen_empty_mem_callback(TCGv_i64 addr, uint32_t info);
static inline void plugin_insn_append(abi_ptr pc, const void *from, size_t size)
{
@@ -69,7 +69,7 @@ static inline void plugin_gen_tb_end(CPUState *cpu)
static inline void plugin_gen_disable_mem_helpers(void)
{ }
-static inline void plugin_gen_empty_mem_callback(TCGv addr, uint32_t info)
+static inline void plugin_gen_empty_mem_callback(TCGv_i64 addr, uint32_t info)
{ }
static inline void plugin_insn_append(abi_ptr pc, const void *from, size_t size)
diff --git a/accel/tcg/plugin-gen.c b/accel/tcg/plugin-gen.c
index 55e892b684..34be1b940c 100644
--- a/accel/tcg/plugin-gen.c
+++ b/accel/tcg/plugin-gen.c
@@ -124,23 +124,20 @@ static void gen_empty_inline_cb(void)
tcg_temp_free_i64(val);
}
-static void gen_empty_mem_cb(TCGv vaddr, uint32_t info)
+static void gen_empty_mem_cb(TCGv_i64 addr, uint32_t info)
{
TCGv_i32 cpu_index = tcg_temp_ebb_new_i32();
TCGv_i32 meminfo = tcg_temp_ebb_new_i32();
- TCGv_i64 vaddr64 = tcg_temp_ebb_new_i64();
TCGv_ptr udata = tcg_temp_ebb_new_ptr();
tcg_gen_movi_i32(meminfo, info);
tcg_gen_movi_ptr(udata, 0);
tcg_gen_ld_i32(cpu_index, cpu_env,
-offsetof(ArchCPU, env) + offsetof(CPUState, cpu_index));
- tcg_gen_extu_tl_i64(vaddr64, vaddr);
- gen_helper_plugin_vcpu_mem_cb(cpu_index, meminfo, vaddr64, udata);
+ gen_helper_plugin_vcpu_mem_cb(cpu_index, meminfo, addr, udata);
tcg_temp_free_ptr(udata);
- tcg_temp_free_i64(vaddr64);
tcg_temp_free_i32(meminfo);
tcg_temp_free_i32(cpu_index);
}
@@ -197,7 +194,7 @@ static void plugin_gen_empty_callback(enum plugin_gen_from from)
}
}
-void plugin_gen_empty_mem_callback(TCGv addr, uint32_t info)
+void plugin_gen_empty_mem_callback(TCGv_i64 addr, uint32_t info)
{
enum qemu_plugin_mem_rw rw = get_plugin_meminfo_rw(info);
diff --git a/tcg/tcg-op-ldst.c b/tcg/tcg-op-ldst.c
index cbd85f793c..0530dcf00a 100644
--- a/tcg/tcg-op-ldst.c
+++ b/tcg/tcg-op-ldst.c
@@ -115,13 +115,13 @@ static void tcg_gen_req_mo(TCGBar type)
}
/* Only required for loads, where value might overlap addr. */
-static TCGv plugin_maybe_preserve_addr(TCGv vaddr)
+static TCGv_i64 plugin_maybe_preserve_addr(TCGv vaddr)
{
#ifdef CONFIG_PLUGIN
if (tcg_ctx->plugin_insn != NULL) {
/* Save a copy of the vaddr for use after a load. */
- TCGv temp = tcg_temp_new();
- tcg_gen_mov_tl(temp, vaddr);
+ TCGv_i64 temp = tcg_temp_ebb_new_i64();
+ tcg_gen_extu_tl_i64(temp, vaddr);
return temp;
}
#endif
@@ -129,16 +129,28 @@ static TCGv plugin_maybe_preserve_addr(TCGv vaddr)
}
static void
-plugin_gen_mem_callbacks(TCGv copy_addr, TCGv orig_addr, MemOpIdx oi,
+plugin_gen_mem_callbacks(TCGv_i64 copy_addr, TCGv orig_addr, MemOpIdx oi,
enum qemu_plugin_mem_rw rw)
{
#ifdef CONFIG_PLUGIN
if (tcg_ctx->plugin_insn != NULL) {
qemu_plugin_meminfo_t info = make_plugin_meminfo(oi, rw);
- plugin_gen_empty_mem_callback(copy_addr ? : orig_addr, info);
+
+#if TARGET_LONG_BITS == 64
if (copy_addr) {
- tcg_temp_free(copy_addr);
+ plugin_gen_empty_mem_callback(copy_addr, info);
+ tcg_temp_free_i64(copy_addr);
+ } else {
+ plugin_gen_empty_mem_callback(orig_addr, info);
}
+#else
+ if (!copy_addr) {
+ copy_addr = tcg_temp_ebb_new_i64();
+ tcg_gen_extu_tl_i64(copy_addr, orig_addr);
+ }
+ plugin_gen_empty_mem_callback(copy_addr, info);
+ tcg_temp_free_i64(copy_addr);
+#endif
}
#endif
}
@@ -147,7 +159,7 @@ void tcg_gen_qemu_ld_i32(TCGv_i32 val, TCGv addr, TCGArg idx, MemOp memop)
{
MemOp orig_memop;
MemOpIdx oi;
- TCGv copy_addr;
+ TCGv_i64 copy_addr;
tcg_gen_req_mo(TCG_MO_LD_LD | TCG_MO_ST_LD);
memop = tcg_canonicalize_memop(memop, 0, 0);
@@ -223,7 +235,7 @@ void tcg_gen_qemu_ld_i64(TCGv_i64 val, TCGv addr, TCGArg idx, MemOp memop)
{
MemOp orig_memop;
MemOpIdx oi;
- TCGv copy_addr;
+ TCGv_i64 copy_addr;
if (TCG_TARGET_REG_BITS == 32 && (memop & MO_SIZE) < MO_64) {
tcg_gen_qemu_ld_i32(TCGV_LOW(val), addr, idx, memop);
--
2.34.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PULL v2 10/12] tcg/loongarch64: Conditionalize tcg_out_exts_i32_i64
2023-05-03 7:20 [PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
` (18 preceding siblings ...)
2023-05-03 7:21 ` [PATCH 10/84] accel/tcg: Widen plugin_gen_empty_mem_callback to i64 Richard Henderson
@ 2023-05-03 7:21 ` Richard Henderson
19 siblings, 0 replies; 24+ messages in thread
From: Richard Henderson @ 2023-05-03 7:21 UTC (permalink / raw)
To: qemu-devel; +Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo
Since TCG_TYPE_I32 values are kept sign-extended in registers,
via ".w" instructions, we need not extend if the register matches.
This is already relied upon by comparisons.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
tcg/loongarch64/tcg-target.c.inc | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc
index 21c2fc9e98..0940788c6f 100644
--- a/tcg/loongarch64/tcg-target.c.inc
+++ b/tcg/loongarch64/tcg-target.c.inc
@@ -463,7 +463,9 @@ static void tcg_out_ext32s(TCGContext *s, TCGReg ret, TCGReg arg)
static void tcg_out_exts_i32_i64(TCGContext *s, TCGReg ret, TCGReg arg)
{
- tcg_out_ext32s(s, ret, arg);
+ if (ret != arg) {
+ tcg_out_ext32s(s, ret, arg);
+ }
}
static void tcg_out_extu_i32_i64(TCGContext *s, TCGReg ret, TCGReg arg)
--
2.34.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH 02/84] tcg: Widen gen_insn_data to uint64_t
2023-05-03 7:22 [RESEND PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
@ 2023-05-03 7:22 ` Richard Henderson
2023-05-04 12:40 ` Anton Johansson via
2023-05-11 15:22 ` Alex Bennée
0 siblings, 2 replies; 24+ messages in thread
From: Richard Henderson @ 2023-05-03 7:22 UTC (permalink / raw)
To: qemu-devel; +Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo
We already pass uint64_t to restore_state_to_opc; this changes all
of the other uses from insn_start through the encoding to decoding.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
include/tcg/tcg-op.h | 39 +++++++++------------------------------
include/tcg/tcg-opc.h | 2 +-
include/tcg/tcg.h | 30 +++++++++++++++---------------
accel/tcg/translate-all.c | 28 ++++++++++++++++------------
tcg/tcg.c | 18 ++++--------------
5 files changed, 45 insertions(+), 72 deletions(-)
diff --git a/include/tcg/tcg-op.h b/include/tcg/tcg-op.h
index 4401fa493c..de3b70aa84 100644
--- a/include/tcg/tcg-op.h
+++ b/include/tcg/tcg-op.h
@@ -723,48 +723,27 @@ static inline void tcg_gen_concat32_i64(TCGv_i64 ret, TCGv_i64 lo, TCGv_i64 hi)
#endif
#if TARGET_INSN_START_WORDS == 1
-# if TARGET_LONG_BITS <= TCG_TARGET_REG_BITS
static inline void tcg_gen_insn_start(target_ulong pc)
{
- tcg_gen_op1(INDEX_op_insn_start, pc);
+ TCGOp *op = tcg_emit_op(INDEX_op_insn_start, 64 / TCG_TARGET_REG_BITS);
+ tcg_set_insn_start_param(op, 0, pc);
}
-# else
-static inline void tcg_gen_insn_start(target_ulong pc)
-{
- tcg_gen_op2(INDEX_op_insn_start, (uint32_t)pc, (uint32_t)(pc >> 32));
-}
-# endif
#elif TARGET_INSN_START_WORDS == 2
-# if TARGET_LONG_BITS <= TCG_TARGET_REG_BITS
static inline void tcg_gen_insn_start(target_ulong pc, target_ulong a1)
{
- tcg_gen_op2(INDEX_op_insn_start, pc, a1);
+ TCGOp *op = tcg_emit_op(INDEX_op_insn_start, 2 * 64 / TCG_TARGET_REG_BITS);
+ tcg_set_insn_start_param(op, 0, pc);
+ tcg_set_insn_start_param(op, 1, a1);
}
-# else
-static inline void tcg_gen_insn_start(target_ulong pc, target_ulong a1)
-{
- tcg_gen_op4(INDEX_op_insn_start,
- (uint32_t)pc, (uint32_t)(pc >> 32),
- (uint32_t)a1, (uint32_t)(a1 >> 32));
-}
-# endif
#elif TARGET_INSN_START_WORDS == 3
-# if TARGET_LONG_BITS <= TCG_TARGET_REG_BITS
static inline void tcg_gen_insn_start(target_ulong pc, target_ulong a1,
target_ulong a2)
{
- tcg_gen_op3(INDEX_op_insn_start, pc, a1, a2);
+ TCGOp *op = tcg_emit_op(INDEX_op_insn_start, 3 * 64 / TCG_TARGET_REG_BITS);
+ tcg_set_insn_start_param(op, 0, pc);
+ tcg_set_insn_start_param(op, 1, a1);
+ tcg_set_insn_start_param(op, 2, a2);
}
-# else
-static inline void tcg_gen_insn_start(target_ulong pc, target_ulong a1,
- target_ulong a2)
-{
- tcg_gen_op6(INDEX_op_insn_start,
- (uint32_t)pc, (uint32_t)(pc >> 32),
- (uint32_t)a1, (uint32_t)(a1 >> 32),
- (uint32_t)a2, (uint32_t)(a2 >> 32));
-}
-# endif
#else
# error "Unhandled number of operands to insn_start"
#endif
diff --git a/include/tcg/tcg-opc.h b/include/tcg/tcg-opc.h
index 94cf7c5d6a..29216366d2 100644
--- a/include/tcg/tcg-opc.h
+++ b/include/tcg/tcg-opc.h
@@ -190,7 +190,7 @@ DEF(mulsh_i64, 1, 2, 0, IMPL64 | IMPL(TCG_TARGET_HAS_mulsh_i64))
#define DATA64_ARGS (TCG_TARGET_REG_BITS == 64 ? 1 : 2)
/* QEMU specific */
-DEF(insn_start, 0, 0, TLADDR_ARGS * TARGET_INSN_START_WORDS,
+DEF(insn_start, 0, 0, DATA64_ARGS * TARGET_INSN_START_WORDS,
TCG_OPF_NOT_PRESENT)
DEF(exit_tb, 0, 0, 1, TCG_OPF_BB_EXIT | TCG_OPF_BB_END)
DEF(goto_tb, 0, 0, 1, TCG_OPF_BB_EXIT | TCG_OPF_BB_END)
diff --git a/include/tcg/tcg.h b/include/tcg/tcg.h
index efbd891f87..7c6a613364 100644
--- a/include/tcg/tcg.h
+++ b/include/tcg/tcg.h
@@ -629,7 +629,7 @@ struct TCGContext {
TCGTemp *reg_to_temp[TCG_TARGET_NB_REGS];
uint16_t gen_insn_end_off[TCG_MAX_INSNS];
- target_ulong gen_insn_data[TCG_MAX_INSNS][TARGET_INSN_START_WORDS];
+ uint64_t gen_insn_data[TCG_MAX_INSNS][TARGET_INSN_START_WORDS];
/* Exit to translator on overflow. */
sigjmp_buf jmp_trans;
@@ -771,24 +771,24 @@ static inline void tcg_set_insn_param(TCGOp *op, int arg, TCGArg v)
op->args[arg] = v;
}
-static inline target_ulong tcg_get_insn_start_param(TCGOp *op, int arg)
+static inline uint64_t tcg_get_insn_start_param(TCGOp *op, int arg)
{
-#if TARGET_LONG_BITS <= TCG_TARGET_REG_BITS
- return tcg_get_insn_param(op, arg);
-#else
- return tcg_get_insn_param(op, arg * 2) |
- ((uint64_t)tcg_get_insn_param(op, arg * 2 + 1) << 32);
-#endif
+ if (TCG_TARGET_REG_BITS == 64) {
+ return tcg_get_insn_param(op, arg);
+ } else {
+ return deposit64(tcg_get_insn_param(op, arg * 2), 32, 32,
+ tcg_get_insn_param(op, arg * 2 + 1));
+ }
}
-static inline void tcg_set_insn_start_param(TCGOp *op, int arg, target_ulong v)
+static inline void tcg_set_insn_start_param(TCGOp *op, int arg, uint64_t v)
{
-#if TARGET_LONG_BITS <= TCG_TARGET_REG_BITS
- tcg_set_insn_param(op, arg, v);
-#else
- tcg_set_insn_param(op, arg * 2, v);
- tcg_set_insn_param(op, arg * 2 + 1, v >> 32);
-#endif
+ if (TCG_TARGET_REG_BITS == 64) {
+ tcg_set_insn_param(op, arg, v);
+ } else {
+ tcg_set_insn_param(op, arg * 2, v);
+ tcg_set_insn_param(op, arg * 2 + 1, v >> 32);
+ }
}
/* The last op that was emitted. */
diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
index 5b13281119..7b7d9a5fff 100644
--- a/accel/tcg/translate-all.c
+++ b/accel/tcg/translate-all.c
@@ -72,9 +72,11 @@ QEMU_BUILD_BUG_ON(CPU_TRACE_DSTATE_MAX_EVENTS >
TBContext tb_ctx;
-/* Encode VAL as a signed leb128 sequence at P.
- Return P incremented past the encoded value. */
-static uint8_t *encode_sleb128(uint8_t *p, target_long val)
+/*
+ * Encode VAL as a signed leb128 sequence at P.
+ * Return P incremented past the encoded value.
+ */
+static uint8_t *encode_sleb128(uint8_t *p, int64_t val)
{
int more, byte;
@@ -92,21 +94,23 @@ static uint8_t *encode_sleb128(uint8_t *p, target_long val)
return p;
}
-/* Decode a signed leb128 sequence at *PP; increment *PP past the
- decoded value. Return the decoded value. */
-static target_long decode_sleb128(const uint8_t **pp)
+/*
+ * Decode a signed leb128 sequence at *PP; increment *PP past the
+ * decoded value. Return the decoded value.
+ */
+static int64_t decode_sleb128(const uint8_t **pp)
{
const uint8_t *p = *pp;
- target_long val = 0;
+ int64_t val = 0;
int byte, shift = 0;
do {
byte = *p++;
- val |= (target_ulong)(byte & 0x7f) << shift;
+ val |= (int64_t)(byte & 0x7f) << shift;
shift += 7;
} while (byte & 0x80);
if (shift < TARGET_LONG_BITS && (byte & 0x40)) {
- val |= -(target_ulong)1 << shift;
+ val |= -(int64_t)1 << shift;
}
*pp = p;
@@ -132,7 +136,7 @@ static int encode_search(TranslationBlock *tb, uint8_t *block)
int i, j, n;
for (i = 0, n = tb->icount; i < n; ++i) {
- target_ulong prev;
+ uint64_t prev;
for (j = 0; j < TARGET_INSN_START_WORDS; ++j) {
if (i == 0) {
@@ -444,7 +448,7 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
/* Dump header and the first instruction */
fprintf(logfile, "OUT: [size=%d]\n", gen_code_size);
fprintf(logfile,
- " -- guest addr 0x" TARGET_FMT_lx " + tb prologue\n",
+ " -- guest addr 0x%016" PRIx64 " + tb prologue\n",
tcg_ctx->gen_insn_data[insn][0]);
chunk_start = tcg_ctx->gen_insn_end_off[insn];
disas(logfile, tb->tc.ptr, chunk_start);
@@ -457,7 +461,7 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
while (insn < tb->icount) {
size_t chunk_end = tcg_ctx->gen_insn_end_off[insn];
if (chunk_end > chunk_start) {
- fprintf(logfile, " -- guest addr 0x" TARGET_FMT_lx "\n",
+ fprintf(logfile, " -- guest addr 0x%016" PRIx64 "\n",
tcg_ctx->gen_insn_data[insn][0]);
disas(logfile, tb->tc.ptr + chunk_start,
chunk_end - chunk_start);
diff --git a/tcg/tcg.c b/tcg/tcg.c
index 45f8758fda..a3b8c82073 100644
--- a/tcg/tcg.c
+++ b/tcg/tcg.c
@@ -2276,13 +2276,8 @@ static void tcg_dump_ops(TCGContext *s, FILE *f, bool have_prefs)
col += ne_fprintf(f, "\n ----");
for (i = 0; i < TARGET_INSN_START_WORDS; ++i) {
- target_ulong a;
-#if TARGET_LONG_BITS > TCG_TARGET_REG_BITS
- a = deposit64(op->args[i * 2], 32, 32, op->args[i * 2 + 1]);
-#else
- a = op->args[i];
-#endif
- col += ne_fprintf(f, " " TARGET_FMT_lx, a);
+ col += ne_fprintf(f, " %016" PRIx64,
+ tcg_get_insn_start_param(op, i));
}
} else if (c == INDEX_op_call) {
const TCGHelperInfo *info = tcg_call_info(op);
@@ -5940,13 +5935,8 @@ int tcg_gen_code(TCGContext *s, TranslationBlock *tb, target_ulong pc_start)
}
num_insns++;
for (i = 0; i < TARGET_INSN_START_WORDS; ++i) {
- target_ulong a;
-#if TARGET_LONG_BITS > TCG_TARGET_REG_BITS
- a = deposit64(op->args[i * 2], 32, 32, op->args[i * 2 + 1]);
-#else
- a = op->args[i];
-#endif
- s->gen_insn_data[num_insns][i] = a;
+ s->gen_insn_data[num_insns][i] =
+ tcg_get_insn_start_param(op, i);
}
break;
case INDEX_op_discard:
--
2.34.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [PATCH 02/84] tcg: Widen gen_insn_data to uint64_t
2023-05-03 7:22 ` [PATCH 02/84] tcg: Widen gen_insn_data to uint64_t Richard Henderson
@ 2023-05-04 12:40 ` Anton Johansson via
2023-05-11 15:22 ` Alex Bennée
1 sibling, 0 replies; 24+ messages in thread
From: Anton Johansson via @ 2023-05-04 12:40 UTC (permalink / raw)
To: Richard Henderson, qemu-devel; +Cc: ale, philmd, marcel.apfelbaum, wangyanan55
On 5/3/23 09:22, Richard Henderson wrote:
> We already pass uint64_t to restore_state_to_opc; this changes all
> of the other uses from insn_start through the encoding to decoding.
>
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
> include/tcg/tcg-op.h | 39 +++++++++------------------------------
> include/tcg/tcg-opc.h | 2 +-
> include/tcg/tcg.h | 30 +++++++++++++++---------------
> accel/tcg/translate-all.c | 28 ++++++++++++++++------------
> tcg/tcg.c | 18 ++++--------------
> 5 files changed, 45 insertions(+), 72 deletions(-)
>
Reviewed-by: Anton Johansson <anjo@rev.ng>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 02/84] tcg: Widen gen_insn_data to uint64_t
2023-05-03 7:22 ` [PATCH 02/84] tcg: Widen gen_insn_data to uint64_t Richard Henderson
2023-05-04 12:40 ` Anton Johansson via
@ 2023-05-11 15:22 ` Alex Bennée
1 sibling, 0 replies; 24+ messages in thread
From: Alex Bennée @ 2023-05-11 15:22 UTC (permalink / raw)
To: Richard Henderson
Cc: ale, philmd, marcel.apfelbaum, wangyanan55, anjo, qemu-devel
Richard Henderson <richard.henderson@linaro.org> writes:
> We already pass uint64_t to restore_state_to_opc; this changes all
> of the other uses from insn_start through the encoding to decoding.
>
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2023-05-11 15:24 UTC | newest]
Thread overview: 24+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-05-03 7:20 [PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
2023-05-03 7:20 ` [PULL v2 01/12] softmmu: Tidy dirtylimit_dirty_ring_full_time Richard Henderson
2023-05-03 7:20 ` [PATCH 01/84] tcg: Split out memory ops to tcg-op-ldst.c Richard Henderson
2023-05-03 7:20 ` [PULL v2 02/12] accel/tcg: Uncache the host address for instruction fetch when tlb size < 1 Richard Henderson
2023-05-03 7:20 ` [PATCH 02/84] tcg: Widen gen_insn_data to uint64_t Richard Henderson
2023-05-03 7:20 ` [PATCH 03/84] accel/tcg: Widen tcg-ldst.h addresses " Richard Henderson
2023-05-03 7:20 ` [PULL v2 03/12] qemu/bitops.h: Limit rotate amounts Richard Henderson
2023-05-03 7:20 ` [PULL v2 04/12] qemu/host-utils.h: Add clz and ctz functions for lower-bit integers Richard Henderson
2023-05-03 7:20 ` [PATCH 04/84] tcg: Widen helper_{ld,st}_i128 addresses to uint64_t Richard Henderson
2023-05-03 7:20 ` [PULL v2 05/12] tcg: Add tcg_gen_gvec_andcs Richard Henderson
2023-05-03 7:20 ` [PATCH 05/84] tcg: Widen helper_atomic_* addresses to uint64_t Richard Henderson
2023-05-03 7:20 ` [PULL v2 06/12] tcg: Add tcg_gen_gvec_rotrs Richard Henderson
2023-05-03 7:20 ` [PATCH 06/84] tcg: Widen tcg_gen_code pc_start argument to uint64_t Richard Henderson
2023-05-03 7:20 ` [PATCH 07/84] accel/tcg: Merge gen_mem_wrapped with plugin_gen_empty_mem_callback Richard Henderson
2023-05-03 7:20 ` [PULL v2 07/12] qemu/int128: Re-shuffle Int128Alias members Richard Henderson
2023-05-03 7:21 ` [PATCH 08/84] accel/tcg: Merge do_gen_mem_cb into caller Richard Henderson
2023-05-03 7:21 ` [PULL v2 08/12] migration/xbzrle: Use __attribute__((target)) for avx512 Richard Henderson
2023-05-03 7:21 ` [PULL v2 09/12] accel/tcg: Add cpu_ld*_code_mmu Richard Henderson
2023-05-03 7:21 ` [PATCH 09/84] tcg: Reduce copies for plugin_gen_mem_callbacks Richard Henderson
2023-05-03 7:21 ` [PATCH 10/84] accel/tcg: Widen plugin_gen_empty_mem_callback to i64 Richard Henderson
2023-05-03 7:21 ` [PULL v2 10/12] tcg/loongarch64: Conditionalize tcg_out_exts_i32_i64 Richard Henderson
-- strict thread matches above, loose matches on Subject: below --
2023-05-03 7:22 [RESEND PATCH 00/84] tcg: Build once for system, once for user Richard Henderson
2023-05-03 7:22 ` [PATCH 02/84] tcg: Widen gen_insn_data to uint64_t Richard Henderson
2023-05-04 12:40 ` Anton Johansson via
2023-05-11 15:22 ` Alex Bennée
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).