* [PATCH 00/27] accel/tcg: Improvements to atomic128.h
@ 2023-05-20 16:26 Richard Henderson
2023-05-20 16:26 ` [PATCH 01/27] util: Introduce host-specific cpuinfo.h Richard Henderson
` (26 more replies)
0 siblings, 27 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel
Cc: qemu-arm, qemu-ppc, Daniel Henrique Barboza,
Cédric Le Goater, David Gibson, Greg Kurz, qemu-s390x,
David Hildenbrand, Ilya Leoshkevich
Peter raised a good point about it not being ideal to mix inline assembly
into the middle of accel/tcg/ldst_atomicity.c. We now have a host-specific
structure in which to put those.
Additionally, Peter noticed that clang will incorrectly use a read-write
sequence for __atomic_load_16 on AArch64, which might fault for our usage
in user-only emulation.
Fixing both of these simultaneously splits atomic16_read into
atomic16_read_{ro,rw}, because there is in fact room for both
in the emulation -- we currently use cmpxchg directly where we
can allow a read with write side-effect.
Additionally, prepare for runtime detection. Both x86_64 and aarch64
have architecture extensions that *do* allow 128-bit load and store
without using cmpxchg.
To make runtime detection work, we need to remove preprocessor use
of HAVE_ATOMIC128*. It turns out this was only used for the legacy
helper_atomic_{ld,st}o_{be,le}_mmu functions. These uses within
ppc64 and s390x can now be updated to tcg_gen_qemu_{ld,st}_i128 and
cpu_{ld,st}16_mmu. After doing that, we can remove the problematic
#if's entirely.
r~
Cc: qemu-ppc@nongnu.org
Cc: Daniel Henrique Barboza <danielhb413@gmail.com>
Cc: "Cédric Le Goater" <clg@kaod.org>
Cc: David Gibson <david@gibson.dropbear.id.au>
Cc: Greg Kurz <groug@kaod.org>
Cc: qemu-s390x@nongnu.org
Cc: David Hildenbrand <david@redhat.com>
Cc: Ilya Leoshkevich <iii@linux.ibm.com>
Richard Henderson (27):
util: Introduce host-specific cpuinfo.h
util: Add cpuinfo-i386.c
util: Add i386 CPUINFO_ATOMIC_VMOVDQU
tcg/i386: Use host/cpuinfo.h
util/bufferiszero: Use i386 host/cpuinfo.h
migration/xbzrle: Shuffle function order
migration/xbzrle: Use i386 host/cpuinfo.h
migration: Build migration_files once
util: Add cpuinfo-aarch64.c
include/host: Split out atomic128-cas.h
include/host: Split out atomic128-ldst.h
meson: Fix detect atomic128 support with optimization
include/qemu: Move CONFIG_ATOMIC128_OPT handling to atomic128.h
target/ppc: Use tcg_gen_qemu_{ld,st}_i128 for LQARX, LQ, STQ
target/s390x: Use tcg_gen_qemu_{ld,st}_i128 for LPQ, STPQ
accel/tcg: Unify cpu_{ld,st}*_{be,le}_mmu
target/s390x: Use cpu_{ld,st}*_mmu in do_csst
target/s390x: Always use cpu_atomic_cmpxchgl_be_mmu in do_csst
accel/tcg: Remove cpu_atomic_{ld,st}o_*_mmu
accel/tcg: Remove prot argument to atomic_mmu_lookup
accel/tcg: Eliminate #if on HAVE_ATOMIC128 and HAVE_CMPXCHG128
qemu/atomic128: Split atomic16_read
accel/tcg: Correctly use atomic128.h in ldst_atomicity.c.inc
tcg: Split out tcg/debug-assert.h
qemu/atomic128: Improve cmpxchg fallback for atomic16_set
qemu/atomic128: Add runtime test for FEAT_LSE2
qemu/atomic128: Add x86_64 atomic128-ldst.h
accel/tcg/atomic_template.h | 93 +---
host/include/aarch64/host/atomic128-cas.h | 45 ++
host/include/aarch64/host/atomic128-ldst.h | 79 ++++
host/include/aarch64/host/cpuinfo.h | 22 +
host/include/generic/host/atomic128-cas.h | 47 +++
host/include/generic/host/atomic128-ldst.h | 81 ++++
host/include/generic/host/cpuinfo.h | 4 +
host/include/i386/host/cpuinfo.h | 39 ++
host/include/x86_64/host/atomic128-ldst.h | 54 +++
host/include/x86_64/host/cpuinfo.h | 1 +
include/exec/cpu_ldst.h | 67 +--
include/qemu/atomic128.h | 146 +------
include/tcg/debug-assert.h | 17 +
include/tcg/tcg.h | 9 +-
migration/xbzrle.h | 5 +-
target/ppc/cpu.h | 1 -
target/ppc/helper.h | 9 -
target/s390x/cpu.h | 3 -
target/s390x/helper.h | 4 -
tcg/aarch64/tcg-target.h | 6 +-
tcg/i386/tcg-target.h | 28 +-
accel/tcg/cputlb.c | 211 +++------
accel/tcg/user-exec.c | 332 ++++-----------
migration/ram.c | 34 +-
migration/xbzrle.c | 268 ++++++------
target/arm/tcg/m_helper.c | 4 +-
target/ppc/mem_helper.c | 48 ---
target/ppc/translate.c | 34 +-
target/s390x/tcg/mem_helper.c | 136 ++----
target/s390x/tcg/translate.c | 30 +-
target/sparc/ldst_helper.c | 18 +-
tests/bench/xbzrle-bench.c | 469 ---------------------
tests/unit/test-xbzrle.c | 49 +--
util/bufferiszero.c | 126 ++----
util/cpuinfo-aarch64.c | 67 +++
util/cpuinfo-i386.c | 99 +++++
accel/tcg/atomic_common.c.inc | 14 -
accel/tcg/ldst_atomicity.c.inc | 135 +-----
accel/tcg/ldst_common.c.inc | 24 +-
meson.build | 10 +-
migration/meson.build | 1 -
target/ppc/translate/fixedpoint-impl.c.inc | 51 +--
target/s390x/tcg/insn-data.h.inc | 2 +-
tcg/aarch64/tcg-target.c.inc | 40 --
tcg/i386/tcg-target.c.inc | 123 +-----
tests/bench/meson.build | 6 -
util/meson.build | 6 +
47 files changed, 1081 insertions(+), 2016 deletions(-)
create mode 100644 host/include/aarch64/host/atomic128-cas.h
create mode 100644 host/include/aarch64/host/atomic128-ldst.h
create mode 100644 host/include/aarch64/host/cpuinfo.h
create mode 100644 host/include/generic/host/atomic128-cas.h
create mode 100644 host/include/generic/host/atomic128-ldst.h
create mode 100644 host/include/generic/host/cpuinfo.h
create mode 100644 host/include/i386/host/cpuinfo.h
create mode 100644 host/include/x86_64/host/atomic128-ldst.h
create mode 100644 host/include/x86_64/host/cpuinfo.h
create mode 100644 include/tcg/debug-assert.h
delete mode 100644 tests/bench/xbzrle-bench.c
create mode 100644 util/cpuinfo-aarch64.c
create mode 100644 util/cpuinfo-i386.c
--
2.34.1
^ permalink raw reply [flat|nested] 46+ messages in thread
* [PATCH 01/27] util: Introduce host-specific cpuinfo.h
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-21 10:47 ` Philippe Mathieu-Daudé
2023-05-23 15:56 ` Alex Bennée
2023-05-20 16:26 ` [PATCH 02/27] util: Add cpuinfo-i386.c Richard Henderson
` (25 subsequent siblings)
26 siblings, 2 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm, Juan Quintela
The entire contents of the header is host-specific, but the
existence of such a header is not, which could prevent some
host specific ifdefs at the top of the file for the include.
Add host/include/{arch,generic} to the project arguments.
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
host/include/generic/host/cpuinfo.h | 4 ++++
meson.build | 8 ++++++++
2 files changed, 12 insertions(+)
create mode 100644 host/include/generic/host/cpuinfo.h
diff --git a/host/include/generic/host/cpuinfo.h b/host/include/generic/host/cpuinfo.h
new file mode 100644
index 0000000000..eca672064a
--- /dev/null
+++ b/host/include/generic/host/cpuinfo.h
@@ -0,0 +1,4 @@
+/*
+ * No host specific cpu indentification.
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
diff --git a/meson.build b/meson.build
index 0a5cdefd4d..4ffc0d3e59 100644
--- a/meson.build
+++ b/meson.build
@@ -512,6 +512,14 @@ add_project_arguments('-iquote', '.',
'-iquote', meson.current_source_dir() / 'include',
language: all_languages)
+host_include = meson.current_source_dir() / 'host/include/'
+if fs.is_dir(host_include / host_arch)
+ add_project_arguments('-iquote', host_include / host_arch,
+ language: all_languages)
+endif
+add_project_arguments('-iquote', host_include / 'generic',
+ language: all_languages)
+
sparse = find_program('cgcc', required: get_option('sparse'))
if sparse.found()
run_target('sparse',
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 02/27] util: Add cpuinfo-i386.c
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
2023-05-20 16:26 ` [PATCH 01/27] util: Introduce host-specific cpuinfo.h Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-21 11:28 ` Philippe Mathieu-Daudé
2023-05-20 16:26 ` [PATCH 03/27] util: Add i386 CPUINFO_ATOMIC_VMOVDQU Richard Henderson
` (24 subsequent siblings)
26 siblings, 1 reply; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm, Juan Quintela
Add cpuinfo.h for i386 and x86_64, and the initialization
for that in util/. Populate that with a slightly altered
copy of the tcg host probing code. Other uses of cpuid.h
will be adjusted one patch at a time.
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
host/include/i386/host/cpuinfo.h | 38 ++++++++++++
host/include/x86_64/host/cpuinfo.h | 1 +
util/cpuinfo-i386.c | 97 ++++++++++++++++++++++++++++++
util/meson.build | 4 ++
4 files changed, 140 insertions(+)
create mode 100644 host/include/i386/host/cpuinfo.h
create mode 100644 host/include/x86_64/host/cpuinfo.h
create mode 100644 util/cpuinfo-i386.c
diff --git a/host/include/i386/host/cpuinfo.h b/host/include/i386/host/cpuinfo.h
new file mode 100644
index 0000000000..e6f7461378
--- /dev/null
+++ b/host/include/i386/host/cpuinfo.h
@@ -0,0 +1,38 @@
+/*
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ * Host specific cpu indentification for x86.
+ */
+
+#ifndef HOST_CPUINFO_H
+#define HOST_CPUINFO_H
+
+/* Digested version of <cpuid.h> */
+
+#define CPUINFO_ALWAYS (1u << 0) /* so cpuinfo is nonzero */
+#define CPUINFO_CMOV (1u << 1)
+#define CPUINFO_MOVBE (1u << 2)
+#define CPUINFO_LZCNT (1u << 3)
+#define CPUINFO_POPCNT (1u << 4)
+#define CPUINFO_BMI1 (1u << 5)
+#define CPUINFO_BMI2 (1u << 6)
+#define CPUINFO_SSE2 (1u << 7)
+#define CPUINFO_SSE4 (1u << 8)
+#define CPUINFO_AVX1 (1u << 9)
+#define CPUINFO_AVX2 (1u << 10)
+#define CPUINFO_AVX512F (1u << 11)
+#define CPUINFO_AVX512VL (1u << 12)
+#define CPUINFO_AVX512BW (1u << 13)
+#define CPUINFO_AVX512DQ (1u << 14)
+#define CPUINFO_AVX512VBMI2 (1u << 15)
+#define CPUINFO_ATOMIC_VMOVDQA (1u << 16)
+
+/* Initialized with a constructor. */
+extern unsigned cpuinfo;
+
+/*
+ * We cannot rely on constructor ordering, so other constructors must
+ * use the function interface rather than the variable above.
+ */
+unsigned cpuinfo_init(void);
+
+#endif /* HOST_CPUINFO_H */
diff --git a/host/include/x86_64/host/cpuinfo.h b/host/include/x86_64/host/cpuinfo.h
new file mode 100644
index 0000000000..67debab9a0
--- /dev/null
+++ b/host/include/x86_64/host/cpuinfo.h
@@ -0,0 +1 @@
+#include "host/include/i386/host/cpuinfo.h"
diff --git a/util/cpuinfo-i386.c b/util/cpuinfo-i386.c
new file mode 100644
index 0000000000..434319aa71
--- /dev/null
+++ b/util/cpuinfo-i386.c
@@ -0,0 +1,97 @@
+/*
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ * Host specific cpu indentification for x86.
+ */
+
+#include "qemu/osdep.h"
+#include "host/cpuinfo.h"
+#ifdef CONFIG_CPUID_H
+# include "qemu/cpuid.h"
+#endif
+
+unsigned cpuinfo;
+
+/* Called both as constructor and (possibly) via other constructors. */
+unsigned __attribute__((constructor)) cpuinfo_init(void)
+{
+ unsigned info = cpuinfo;
+
+ if (info) {
+ return info;
+ }
+
+#ifdef CONFIG_CPUID_H
+ unsigned max, a, b, c, d, b7 = 0, c7 = 0;
+
+ max = __get_cpuid_max(0, 0);
+
+ if (max >= 7) {
+ __cpuid_count(7, 0, a, b7, c7, d);
+ info |= (b7 & bit_BMI ? CPUINFO_BMI1 : 0);
+ info |= (b7 & bit_BMI2 ? CPUINFO_BMI2 : 0);
+ }
+
+ if (max >= 1) {
+ __cpuid(1, a, b, c, d);
+
+ info |= (d & bit_CMOV ? CPUINFO_CMOV : 0);
+ info |= (d & bit_SSE2 ? CPUINFO_SSE2 : 0);
+ info |= (c & bit_SSE4_1 ? CPUINFO_SSE4 : 0);
+ info |= (c & bit_MOVBE ? CPUINFO_MOVBE : 0);
+ info |= (c & bit_POPCNT ? CPUINFO_POPCNT : 0);
+
+ /* For AVX features, we must check available and usable. */
+ if ((c & bit_AVX) && (c & bit_OSXSAVE)) {
+ unsigned bv = xgetbv_low(0);
+
+ if ((bv & 6) == 6) {
+ info |= CPUINFO_AVX1;
+ info |= (b7 & bit_AVX2 ? CPUINFO_AVX2 : 0);
+
+ if ((bv & 0xe0) == 0xe0) {
+ info |= (b7 & bit_AVX512F ? CPUINFO_AVX512F : 0);
+ info |= (b7 & bit_AVX512VL ? CPUINFO_AVX512VL : 0);
+ info |= (b7 & bit_AVX512BW ? CPUINFO_AVX512BW : 0);
+ info |= (b7 & bit_AVX512DQ ? CPUINFO_AVX512DQ : 0);
+ info |= (c7 & bit_AVX512VBMI2 ? CPUINFO_AVX512VBMI2 : 0);
+ }
+
+ /*
+ * The Intel SDM has added:
+ * Processors that enumerate support for Intel® AVX
+ * (by setting the feature flag CPUID.01H:ECX.AVX[bit 28])
+ * guarantee that the 16-byte memory operations performed
+ * by the following instructions will always be carried
+ * out atomically:
+ * - MOVAPD, MOVAPS, and MOVDQA.
+ * - VMOVAPD, VMOVAPS, and VMOVDQA when encoded with VEX.128.
+ * - VMOVAPD, VMOVAPS, VMOVDQA32, and VMOVDQA64 when encoded
+ * with EVEX.128 and k0 (masking disabled).
+ * Note that these instructions require the linear addresses
+ * of their memory operands to be 16-byte aligned.
+ *
+ * AMD has provided an even stronger guarantee that processors
+ * with AVX provide 16-byte atomicity for all cachable,
+ * naturally aligned single loads and stores, e.g. MOVDQU.
+ *
+ * See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104688
+ */
+ __cpuid(0, a, b, c, d);
+ if (c == signature_INTEL_ecx || c == signature_AMD_ecx) {
+ info |= CPUINFO_ATOMIC_VMOVDQA;
+ }
+ }
+ }
+ }
+
+ max = __get_cpuid_max(0x8000000, 0);
+ if (max >= 1) {
+ __cpuid(0x80000001, a, b, c, d);
+ info |= (c & bit_LZCNT ? CPUINFO_LZCNT : 0);
+ }
+#endif
+
+ info |= CPUINFO_ALWAYS;
+ cpuinfo = info;
+ return info;
+}
diff --git a/util/meson.build b/util/meson.build
index e1f1c39e10..b3be9fad5d 100644
--- a/util/meson.build
+++ b/util/meson.build
@@ -108,3 +108,7 @@ if have_block
endif
util_ss.add(when: 'CONFIG_LINUX', if_true: files('vfio-helpers.c'))
endif
+
+if cpu in ['x86', 'x86_64']
+ util_ss.add(files('cpuinfo-i386.c'))
+endif
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 03/27] util: Add i386 CPUINFO_ATOMIC_VMOVDQU
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
2023-05-20 16:26 ` [PATCH 01/27] util: Introduce host-specific cpuinfo.h Richard Henderson
2023-05-20 16:26 ` [PATCH 02/27] util: Add cpuinfo-i386.c Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-20 16:26 ` [PATCH 04/27] tcg/i386: Use host/cpuinfo.h Richard Henderson
` (23 subsequent siblings)
26 siblings, 0 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm, Peter Maydell
Add a bit to indicate when VMOVDQU is also atomic if aligned.
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
host/include/i386/host/cpuinfo.h | 1 +
util/cpuinfo-i386.c | 4 +++-
2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/host/include/i386/host/cpuinfo.h b/host/include/i386/host/cpuinfo.h
index e6f7461378..a6537123cf 100644
--- a/host/include/i386/host/cpuinfo.h
+++ b/host/include/i386/host/cpuinfo.h
@@ -25,6 +25,7 @@
#define CPUINFO_AVX512DQ (1u << 14)
#define CPUINFO_AVX512VBMI2 (1u << 15)
#define CPUINFO_ATOMIC_VMOVDQA (1u << 16)
+#define CPUINFO_ATOMIC_VMOVDQU (1u << 17)
/* Initialized with a constructor. */
extern unsigned cpuinfo;
diff --git a/util/cpuinfo-i386.c b/util/cpuinfo-i386.c
index 434319aa71..ab6143d9e7 100644
--- a/util/cpuinfo-i386.c
+++ b/util/cpuinfo-i386.c
@@ -77,8 +77,10 @@ unsigned __attribute__((constructor)) cpuinfo_init(void)
* See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104688
*/
__cpuid(0, a, b, c, d);
- if (c == signature_INTEL_ecx || c == signature_AMD_ecx) {
+ if (c == signature_INTEL_ecx) {
info |= CPUINFO_ATOMIC_VMOVDQA;
+ } else if (c == signature_AMD_ecx) {
+ info |= CPUINFO_ATOMIC_VMOVDQA | CPUINFO_ATOMIC_VMOVDQU;
}
}
}
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 04/27] tcg/i386: Use host/cpuinfo.h
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (2 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 03/27] util: Add i386 CPUINFO_ATOMIC_VMOVDQU Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-20 16:26 ` [PATCH 05/27] util/bufferiszero: Use i386 host/cpuinfo.h Richard Henderson
` (22 subsequent siblings)
26 siblings, 0 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm, Peter Maydell
Use the CPUINFO_* bits instead of the individual boolean
variables that we had been using. Remove all of the init
code that was moved over to cpuinfo-i386.c.
Note that have_avx512* check both AVX512{F,VL}, as we had
previously done during tcg_target_init.
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
tcg/i386/tcg-target.h | 28 +++++----
tcg/i386/tcg-target.c.inc | 123 ++------------------------------------
2 files changed, 22 insertions(+), 129 deletions(-)
diff --git a/tcg/i386/tcg-target.h b/tcg/i386/tcg-target.h
index 0b5a2c68c5..0106946996 100644
--- a/tcg/i386/tcg-target.h
+++ b/tcg/i386/tcg-target.h
@@ -25,6 +25,8 @@
#ifndef I386_TCG_TARGET_H
#define I386_TCG_TARGET_H
+#include "host/cpuinfo.h"
+
#define TCG_TARGET_INSN_UNIT_SIZE 1
#define TCG_TARGET_TLB_DISPLACEMENT_BITS 31
@@ -111,16 +113,22 @@ typedef enum {
# define TCG_TARGET_CALL_RET_I128 TCG_CALL_RET_BY_REF
#endif
-extern bool have_bmi1;
-extern bool have_popcnt;
-extern bool have_avx1;
-extern bool have_avx2;
-extern bool have_avx512bw;
-extern bool have_avx512dq;
-extern bool have_avx512vbmi2;
-extern bool have_avx512vl;
-extern bool have_movbe;
-extern bool have_atomic16;
+#define have_bmi1 (cpuinfo & CPUINFO_BMI1)
+#define have_popcnt (cpuinfo & CPUINFO_POPCNT)
+#define have_avx1 (cpuinfo & CPUINFO_AVX1)
+#define have_avx2 (cpuinfo & CPUINFO_AVX2)
+#define have_movbe (cpuinfo & CPUINFO_MOVBE)
+#define have_atomic16 (cpuinfo & CPUINFO_ATOMIC_VMOVDQA)
+
+/*
+ * There are interesting instructions in AVX512, so long as we have AVX512VL,
+ * which indicates support for EVEX on sizes smaller than 512 bits.
+ */
+#define have_avx512vl ((cpuinfo & CPUINFO_AVX512VL) && \
+ (cpuinfo & CPUINFO_AVX512F))
+#define have_avx512bw ((cpuinfo & CPUINFO_AVX512BW) && have_avx512vl)
+#define have_avx512dq ((cpuinfo & CPUINFO_AVX512DQ) && have_avx512vl)
+#define have_avx512vbmi2 ((cpuinfo & CPUINFO_AVX512VBMI2) && have_avx512vl)
/* optional instructions */
#define TCG_TARGET_HAS_div2_i32 1
diff --git a/tcg/i386/tcg-target.c.inc b/tcg/i386/tcg-target.c.inc
index 8b9a5f00e5..bfe9d98b7e 100644
--- a/tcg/i386/tcg-target.c.inc
+++ b/tcg/i386/tcg-target.c.inc
@@ -158,42 +158,14 @@ static TCGReg tcg_target_call_oarg_reg(TCGCallReturnKind kind, int slot)
# define SOFTMMU_RESERVE_REGS 0
#endif
-/* The host compiler should supply <cpuid.h> to enable runtime features
- detection, as we're not going to go so far as our own inline assembly.
- If not available, default values will be assumed. */
-#if defined(CONFIG_CPUID_H)
-#include "qemu/cpuid.h"
-#endif
-
/* For 64-bit, we always know that CMOV is available. */
#if TCG_TARGET_REG_BITS == 64
-# define have_cmov 1
-#elif defined(CONFIG_CPUID_H)
-static bool have_cmov;
+# define have_cmov true
#else
-# define have_cmov 0
-#endif
-
-/* We need these symbols in tcg-target.h, and we can't properly conditionalize
- it there. Therefore we always define the variable. */
-bool have_bmi1;
-bool have_popcnt;
-bool have_avx1;
-bool have_avx2;
-bool have_avx512bw;
-bool have_avx512dq;
-bool have_avx512vbmi2;
-bool have_avx512vl;
-bool have_movbe;
-bool have_atomic16;
-
-#ifdef CONFIG_CPUID_H
-static bool have_bmi2;
-static bool have_lzcnt;
-#else
-# define have_bmi2 0
-# define have_lzcnt 0
+# define have_cmov (cpuinfo & CPUINFO_CMOV)
#endif
+#define have_bmi2 (cpuinfo & CPUINFO_BMI2)
+#define have_lzcnt (cpuinfo & CPUINFO_LZCNT)
static const tcg_insn_unit *tb_ret_addr;
@@ -3961,93 +3933,6 @@ static void tcg_out_nop_fill(tcg_insn_unit *p, int count)
static void tcg_target_init(TCGContext *s)
{
-#ifdef CONFIG_CPUID_H
- unsigned a, b, c, d, b7 = 0, c7 = 0;
- unsigned max = __get_cpuid_max(0, 0);
-
- if (max >= 7) {
- /* BMI1 is available on AMD Piledriver and Intel Haswell CPUs. */
- __cpuid_count(7, 0, a, b7, c7, d);
- have_bmi1 = (b7 & bit_BMI) != 0;
- have_bmi2 = (b7 & bit_BMI2) != 0;
- }
-
- if (max >= 1) {
- __cpuid(1, a, b, c, d);
-#ifndef have_cmov
- /* For 32-bit, 99% certainty that we're running on hardware that
- supports cmov, but we still need to check. In case cmov is not
- available, we'll use a small forward branch. */
- have_cmov = (d & bit_CMOV) != 0;
-#endif
-
- /* MOVBE is only available on Intel Atom and Haswell CPUs, so we
- need to probe for it. */
- have_movbe = (c & bit_MOVBE) != 0;
- have_popcnt = (c & bit_POPCNT) != 0;
-
- /* There are a number of things we must check before we can be
- sure of not hitting invalid opcode. */
- if (c & bit_OSXSAVE) {
- unsigned bv = xgetbv_low(0);
-
- if ((bv & 6) == 6) {
- have_avx1 = (c & bit_AVX) != 0;
- have_avx2 = (b7 & bit_AVX2) != 0;
-
- /*
- * There are interesting instructions in AVX512, so long
- * as we have AVX512VL, which indicates support for EVEX
- * on sizes smaller than 512 bits. We are required to
- * check that OPMASK and all extended ZMM state are enabled
- * even if we're not using them -- the insns will fault.
- */
- if ((bv & 0xe0) == 0xe0
- && (b7 & bit_AVX512F)
- && (b7 & bit_AVX512VL)) {
- have_avx512vl = true;
- have_avx512bw = (b7 & bit_AVX512BW) != 0;
- have_avx512dq = (b7 & bit_AVX512DQ) != 0;
- have_avx512vbmi2 = (c7 & bit_AVX512VBMI2) != 0;
- }
-
- /*
- * The Intel SDM has added:
- * Processors that enumerate support for Intel® AVX
- * (by setting the feature flag CPUID.01H:ECX.AVX[bit 28])
- * guarantee that the 16-byte memory operations performed
- * by the following instructions will always be carried
- * out atomically:
- * - MOVAPD, MOVAPS, and MOVDQA.
- * - VMOVAPD, VMOVAPS, and VMOVDQA when encoded with VEX.128.
- * - VMOVAPD, VMOVAPS, VMOVDQA32, and VMOVDQA64 when encoded
- * with EVEX.128 and k0 (masking disabled).
- * Note that these instructions require the linear addresses
- * of their memory operands to be 16-byte aligned.
- *
- * AMD has provided an even stronger guarantee that processors
- * with AVX provide 16-byte atomicity for all cachable,
- * naturally aligned single loads and stores, e.g. MOVDQU.
- *
- * See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104688
- */
- if (have_avx1) {
- __cpuid(0, a, b, c, d);
- have_atomic16 = (c == signature_INTEL_ecx ||
- c == signature_AMD_ecx);
- }
- }
- }
- }
-
- max = __get_cpuid_max(0x8000000, 0);
- if (max >= 1) {
- __cpuid(0x80000001, a, b, c, d);
- /* LZCNT was introduced with AMD Barcelona and Intel Haswell CPUs. */
- have_lzcnt = (c & bit_LZCNT) != 0;
- }
-#endif /* CONFIG_CPUID_H */
-
tcg_target_available_regs[TCG_TYPE_I32] = ALL_GENERAL_REGS;
if (TCG_TARGET_REG_BITS == 64) {
tcg_target_available_regs[TCG_TYPE_I64] = ALL_GENERAL_REGS;
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 05/27] util/bufferiszero: Use i386 host/cpuinfo.h
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (3 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 04/27] tcg/i386: Use host/cpuinfo.h Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-20 16:26 ` [PATCH 06/27] migration/xbzrle: Shuffle function order Richard Henderson
` (21 subsequent siblings)
26 siblings, 0 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm
Use cpuinfo_init() during init_accel(), and the variable cpuinfo
during test_buffer_is_zero_next_accel(). Adjust the logic that
cycles through the set of accelerators for testing.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
util/bufferiszero.c | 126 ++++++++++++++++----------------------------
1 file changed, 45 insertions(+), 81 deletions(-)
diff --git a/util/bufferiszero.c b/util/bufferiszero.c
index 1886bc5ba4..d3c14320ef 100644
--- a/util/bufferiszero.c
+++ b/util/bufferiszero.c
@@ -24,6 +24,7 @@
#include "qemu/osdep.h"
#include "qemu/cutils.h"
#include "qemu/bswap.h"
+#include "host/cpuinfo.h"
static bool
buffer_zero_int(const void *buf, size_t len)
@@ -184,111 +185,74 @@ buffer_zero_avx512(const void *buf, size_t len)
}
#endif /* CONFIG_AVX512F_OPT */
-
-/* Note that for test_buffer_is_zero_next_accel, the most preferred
- * ISA must have the least significant bit.
- */
-#define CACHE_AVX512F 1
-#define CACHE_AVX2 2
-#define CACHE_SSE4 4
-#define CACHE_SSE2 8
-
-/* Make sure that these variables are appropriately initialized when
+/*
+ * Make sure that these variables are appropriately initialized when
* SSE2 is enabled on the compiler command-line, but the compiler is
* too old to support CONFIG_AVX2_OPT.
*/
#if defined(CONFIG_AVX512F_OPT) || defined(CONFIG_AVX2_OPT)
-# define INIT_CACHE 0
-# define INIT_ACCEL buffer_zero_int
+# define INIT_USED 0
+# define INIT_LENGTH 0
+# define INIT_ACCEL buffer_zero_int
#else
# ifndef __SSE2__
# error "ISA selection confusion"
# endif
-# define INIT_CACHE CACHE_SSE2
-# define INIT_ACCEL buffer_zero_sse2
+# define INIT_USED CPUINFO_SSE2
+# define INIT_LENGTH 64
+# define INIT_ACCEL buffer_zero_sse2
#endif
-static unsigned cpuid_cache = INIT_CACHE;
+static unsigned used_accel = INIT_USED;
+static unsigned length_to_accel = INIT_LENGTH;
static bool (*buffer_accel)(const void *, size_t) = INIT_ACCEL;
-static int length_to_accel = 64;
-static void init_accel(unsigned cache)
+static unsigned __attribute__((noinline))
+select_accel_cpuinfo(unsigned info)
{
- bool (*fn)(const void *, size_t) = buffer_zero_int;
- if (cache & CACHE_SSE2) {
- fn = buffer_zero_sse2;
- length_to_accel = 64;
- }
-#ifdef CONFIG_AVX2_OPT
- if (cache & CACHE_SSE4) {
- fn = buffer_zero_sse4;
- length_to_accel = 64;
- }
- if (cache & CACHE_AVX2) {
- fn = buffer_zero_avx2;
- length_to_accel = 128;
- }
-#endif
+ static const struct {
+ unsigned bit;
+ unsigned len;
+ bool (*fn)(const void *, size_t);
+ } all[] = {
#ifdef CONFIG_AVX512F_OPT
- if (cache & CACHE_AVX512F) {
- fn = buffer_zero_avx512;
- length_to_accel = 256;
- }
+ { CPUINFO_AVX512F, 256, buffer_zero_avx512 },
#endif
- buffer_accel = fn;
+#ifdef CONFIG_AVX2_OPT
+ { CPUINFO_AVX2, 128, buffer_zero_avx2 },
+ { CPUINFO_SSE4, 64, buffer_zero_sse4 },
+#endif
+ { CPUINFO_SSE2, 64, buffer_zero_sse2 },
+ { CPUINFO_ALWAYS, 0, buffer_zero_int },
+ };
+
+ for (unsigned i = 0; i < ARRAY_SIZE(all); ++i) {
+ if (info & all[i].bit) {
+ length_to_accel = all[i].len;
+ buffer_accel = all[i].fn;
+ return all[i].bit;
+ }
+ }
+ return 0;
}
#if defined(CONFIG_AVX512F_OPT) || defined(CONFIG_AVX2_OPT)
-#include "qemu/cpuid.h"
-
-static void __attribute__((constructor)) init_cpuid_cache(void)
+static void __attribute__((constructor)) init_accel(void)
{
- unsigned max = __get_cpuid_max(0, NULL);
- int a, b, c, d;
- unsigned cache = 0;
-
- if (max >= 1) {
- __cpuid(1, a, b, c, d);
- if (d & bit_SSE2) {
- cache |= CACHE_SSE2;
- }
- if (c & bit_SSE4_1) {
- cache |= CACHE_SSE4;
- }
-
- /* We must check that AVX is not just available, but usable. */
- if ((c & bit_OSXSAVE) && (c & bit_AVX) && max >= 7) {
- unsigned bv = xgetbv_low(0);
- __cpuid_count(7, 0, a, b, c, d);
- if ((bv & 0x6) == 0x6 && (b & bit_AVX2)) {
- cache |= CACHE_AVX2;
- }
- /* 0xe6:
- * XCR0[7:5] = 111b (OPMASK state, upper 256-bit of ZMM0-ZMM15
- * and ZMM16-ZMM31 state are enabled by OS)
- * XCR0[2:1] = 11b (XMM state and YMM state are enabled by OS)
- */
- if ((bv & 0xe6) == 0xe6 && (b & bit_AVX512F)) {
- cache |= CACHE_AVX512F;
- }
- }
- }
- cpuid_cache = cache;
- init_accel(cache);
+ used_accel = select_accel_cpuinfo(cpuinfo_init());
}
#endif /* CONFIG_AVX2_OPT */
bool test_buffer_is_zero_next_accel(void)
{
- /* If no bits set, we just tested buffer_zero_int, and there
- are no more acceleration options to test. */
- if (cpuid_cache == 0) {
- return false;
- }
- /* Disable the accelerator we used before and select a new one. */
- cpuid_cache &= cpuid_cache - 1;
- init_accel(cpuid_cache);
- return true;
+ /*
+ * Accumulate the accelerators that we've already tested, and
+ * remove them from the set to test this round. We'll get back
+ * a zero from select_accel_cpuinfo when there are no more.
+ */
+ unsigned used = select_accel_cpuinfo(cpuinfo & ~used_accel);
+ used_accel |= used;
+ return used;
}
static bool select_accel_fn(const void *buf, size_t len)
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 06/27] migration/xbzrle: Shuffle function order
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (4 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 05/27] util/bufferiszero: Use i386 host/cpuinfo.h Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-20 16:26 ` [PATCH 07/27] migration/xbzrle: Use i386 host/cpuinfo.h Richard Henderson
` (20 subsequent siblings)
26 siblings, 0 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm, Juan Quintela
Place the CONFIG_AVX512BW_OPT block at the top,
which will aid function selection in the next patch.
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
migration/xbzrle.c | 244 ++++++++++++++++++++++-----------------------
1 file changed, 122 insertions(+), 122 deletions(-)
diff --git a/migration/xbzrle.c b/migration/xbzrle.c
index 258e4959c9..751b5428f7 100644
--- a/migration/xbzrle.c
+++ b/migration/xbzrle.c
@@ -15,6 +15,128 @@
#include "qemu/host-utils.h"
#include "xbzrle.h"
+#if defined(CONFIG_AVX512BW_OPT)
+#include <immintrin.h>
+
+int __attribute__((target("avx512bw")))
+xbzrle_encode_buffer_avx512(uint8_t *old_buf, uint8_t *new_buf, int slen,
+ uint8_t *dst, int dlen)
+{
+ uint32_t zrun_len = 0, nzrun_len = 0;
+ int d = 0, i = 0, num = 0;
+ uint8_t *nzrun_start = NULL;
+ /* add 1 to include residual part in main loop */
+ uint32_t count512s = (slen >> 6) + 1;
+ /* countResidual is tail of data, i.e., countResidual = slen % 64 */
+ uint32_t count_residual = slen & 0b111111;
+ bool never_same = true;
+ uint64_t mask_residual = 1;
+ mask_residual <<= count_residual;
+ mask_residual -= 1;
+ __m512i r = _mm512_set1_epi32(0);
+
+ while (count512s) {
+ int bytes_to_check = 64;
+ uint64_t mask = 0xffffffffffffffff;
+ if (count512s == 1) {
+ bytes_to_check = count_residual;
+ mask = mask_residual;
+ }
+ __m512i old_data = _mm512_mask_loadu_epi8(r,
+ mask, old_buf + i);
+ __m512i new_data = _mm512_mask_loadu_epi8(r,
+ mask, new_buf + i);
+ uint64_t comp = _mm512_cmpeq_epi8_mask(old_data, new_data);
+ count512s--;
+
+ bool is_same = (comp & 0x1);
+ while (bytes_to_check) {
+ if (d + 2 > dlen) {
+ return -1;
+ }
+ if (is_same) {
+ if (nzrun_len) {
+ d += uleb128_encode_small(dst + d, nzrun_len);
+ if (d + nzrun_len > dlen) {
+ return -1;
+ }
+ nzrun_start = new_buf + i - nzrun_len;
+ memcpy(dst + d, nzrun_start, nzrun_len);
+ d += nzrun_len;
+ nzrun_len = 0;
+ }
+ /* 64 data at a time for speed */
+ if (count512s && (comp == 0xffffffffffffffff)) {
+ i += 64;
+ zrun_len += 64;
+ break;
+ }
+ never_same = false;
+ num = ctz64(~comp);
+ num = (num < bytes_to_check) ? num : bytes_to_check;
+ zrun_len += num;
+ bytes_to_check -= num;
+ comp >>= num;
+ i += num;
+ if (bytes_to_check) {
+ /* still has different data after same data */
+ d += uleb128_encode_small(dst + d, zrun_len);
+ zrun_len = 0;
+ } else {
+ break;
+ }
+ }
+ if (never_same || zrun_len) {
+ /*
+ * never_same only acts if
+ * data begins with diff in first count512s
+ */
+ d += uleb128_encode_small(dst + d, zrun_len);
+ zrun_len = 0;
+ never_same = false;
+ }
+ /* has diff, 64 data at a time for speed */
+ if ((bytes_to_check == 64) && (comp == 0x0)) {
+ i += 64;
+ nzrun_len += 64;
+ break;
+ }
+ num = ctz64(comp);
+ num = (num < bytes_to_check) ? num : bytes_to_check;
+ nzrun_len += num;
+ bytes_to_check -= num;
+ comp >>= num;
+ i += num;
+ if (bytes_to_check) {
+ /* mask like 111000 */
+ d += uleb128_encode_small(dst + d, nzrun_len);
+ /* overflow */
+ if (d + nzrun_len > dlen) {
+ return -1;
+ }
+ nzrun_start = new_buf + i - nzrun_len;
+ memcpy(dst + d, nzrun_start, nzrun_len);
+ d += nzrun_len;
+ nzrun_len = 0;
+ is_same = true;
+ }
+ }
+ }
+
+ if (nzrun_len != 0) {
+ d += uleb128_encode_small(dst + d, nzrun_len);
+ /* overflow */
+ if (d + nzrun_len > dlen) {
+ return -1;
+ }
+ nzrun_start = new_buf + i - nzrun_len;
+ memcpy(dst + d, nzrun_start, nzrun_len);
+ d += nzrun_len;
+ }
+ return d;
+}
+#endif
+
/*
page = zrun nzrun
| zrun nzrun page
@@ -175,125 +297,3 @@ int xbzrle_decode_buffer(uint8_t *src, int slen, uint8_t *dst, int dlen)
return d;
}
-
-#if defined(CONFIG_AVX512BW_OPT)
-#include <immintrin.h>
-
-int __attribute__((target("avx512bw")))
-xbzrle_encode_buffer_avx512(uint8_t *old_buf, uint8_t *new_buf, int slen,
- uint8_t *dst, int dlen)
-{
- uint32_t zrun_len = 0, nzrun_len = 0;
- int d = 0, i = 0, num = 0;
- uint8_t *nzrun_start = NULL;
- /* add 1 to include residual part in main loop */
- uint32_t count512s = (slen >> 6) + 1;
- /* countResidual is tail of data, i.e., countResidual = slen % 64 */
- uint32_t count_residual = slen & 0b111111;
- bool never_same = true;
- uint64_t mask_residual = 1;
- mask_residual <<= count_residual;
- mask_residual -= 1;
- __m512i r = _mm512_set1_epi32(0);
-
- while (count512s) {
- int bytes_to_check = 64;
- uint64_t mask = 0xffffffffffffffff;
- if (count512s == 1) {
- bytes_to_check = count_residual;
- mask = mask_residual;
- }
- __m512i old_data = _mm512_mask_loadu_epi8(r,
- mask, old_buf + i);
- __m512i new_data = _mm512_mask_loadu_epi8(r,
- mask, new_buf + i);
- uint64_t comp = _mm512_cmpeq_epi8_mask(old_data, new_data);
- count512s--;
-
- bool is_same = (comp & 0x1);
- while (bytes_to_check) {
- if (d + 2 > dlen) {
- return -1;
- }
- if (is_same) {
- if (nzrun_len) {
- d += uleb128_encode_small(dst + d, nzrun_len);
- if (d + nzrun_len > dlen) {
- return -1;
- }
- nzrun_start = new_buf + i - nzrun_len;
- memcpy(dst + d, nzrun_start, nzrun_len);
- d += nzrun_len;
- nzrun_len = 0;
- }
- /* 64 data at a time for speed */
- if (count512s && (comp == 0xffffffffffffffff)) {
- i += 64;
- zrun_len += 64;
- break;
- }
- never_same = false;
- num = ctz64(~comp);
- num = (num < bytes_to_check) ? num : bytes_to_check;
- zrun_len += num;
- bytes_to_check -= num;
- comp >>= num;
- i += num;
- if (bytes_to_check) {
- /* still has different data after same data */
- d += uleb128_encode_small(dst + d, zrun_len);
- zrun_len = 0;
- } else {
- break;
- }
- }
- if (never_same || zrun_len) {
- /*
- * never_same only acts if
- * data begins with diff in first count512s
- */
- d += uleb128_encode_small(dst + d, zrun_len);
- zrun_len = 0;
- never_same = false;
- }
- /* has diff, 64 data at a time for speed */
- if ((bytes_to_check == 64) && (comp == 0x0)) {
- i += 64;
- nzrun_len += 64;
- break;
- }
- num = ctz64(comp);
- num = (num < bytes_to_check) ? num : bytes_to_check;
- nzrun_len += num;
- bytes_to_check -= num;
- comp >>= num;
- i += num;
- if (bytes_to_check) {
- /* mask like 111000 */
- d += uleb128_encode_small(dst + d, nzrun_len);
- /* overflow */
- if (d + nzrun_len > dlen) {
- return -1;
- }
- nzrun_start = new_buf + i - nzrun_len;
- memcpy(dst + d, nzrun_start, nzrun_len);
- d += nzrun_len;
- nzrun_len = 0;
- is_same = true;
- }
- }
- }
-
- if (nzrun_len != 0) {
- d += uleb128_encode_small(dst + d, nzrun_len);
- /* overflow */
- if (d + nzrun_len > dlen) {
- return -1;
- }
- nzrun_start = new_buf + i - nzrun_len;
- memcpy(dst + d, nzrun_start, nzrun_len);
- d += nzrun_len;
- }
- return d;
-}
-#endif
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 07/27] migration/xbzrle: Use i386 host/cpuinfo.h
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (5 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 06/27] migration/xbzrle: Shuffle function order Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-20 16:26 ` [PATCH 08/27] migration: Build migration_files once Richard Henderson
` (19 subsequent siblings)
26 siblings, 0 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm, Juan Quintela
Perform the function selection once, and only if CONFIG_AVX512_OPT
is enabled. Centralize the selection to xbzrle.c, instead of
spreading the init across 3 files.
Remove xbzrle-bench.c. The benefit of being able to benchmark
the different implementations is less important than not peeking
into the internals of the implementation.
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
migration/xbzrle.h | 5 +-
migration/ram.c | 34 +--
migration/xbzrle.c | 26 +-
tests/bench/xbzrle-bench.c | 469 -------------------------------------
tests/unit/test-xbzrle.c | 49 +---
tests/bench/meson.build | 6 -
6 files changed, 39 insertions(+), 550 deletions(-)
delete mode 100644 tests/bench/xbzrle-bench.c
diff --git a/migration/xbzrle.h b/migration/xbzrle.h
index 6feb49160a..39e651b9ec 100644
--- a/migration/xbzrle.h
+++ b/migration/xbzrle.h
@@ -18,8 +18,5 @@ int xbzrle_encode_buffer(uint8_t *old_buf, uint8_t *new_buf, int slen,
uint8_t *dst, int dlen);
int xbzrle_decode_buffer(uint8_t *src, int slen, uint8_t *dst, int dlen);
-#if defined(CONFIG_AVX512BW_OPT)
-int xbzrle_encode_buffer_avx512(uint8_t *old_buf, uint8_t *new_buf, int slen,
- uint8_t *dst, int dlen);
-#endif
+
#endif
diff --git a/migration/ram.c b/migration/ram.c
index 9fb076fa58..88a6c82e63 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -90,34 +90,6 @@
#define RAM_SAVE_FLAG_MULTIFD_FLUSH 0x200
/* We can't use any flag that is bigger than 0x200 */
-int (*xbzrle_encode_buffer_func)(uint8_t *, uint8_t *, int,
- uint8_t *, int) = xbzrle_encode_buffer;
-#if defined(CONFIG_AVX512BW_OPT)
-#include "qemu/cpuid.h"
-static void __attribute__((constructor)) init_cpu_flag(void)
-{
- unsigned max = __get_cpuid_max(0, NULL);
- int a, b, c, d;
- if (max >= 1) {
- __cpuid(1, a, b, c, d);
- /* We must check that AVX is not just available, but usable. */
- if ((c & bit_OSXSAVE) && (c & bit_AVX) && max >= 7) {
- int bv;
- __asm("xgetbv" : "=a"(bv), "=d"(d) : "c"(0));
- __cpuid_count(7, 0, a, b, c, d);
- /* 0xe6:
- * XCR0[7:5] = 111b (OPMASK state, upper 256-bit of ZMM0-ZMM15
- * and ZMM16-ZMM31 state are enabled by OS)
- * XCR0[2:1] = 11b (XMM state and YMM state are enabled by OS)
- */
- if ((bv & 0xe6) == 0xe6 && (b & bit_AVX512BW)) {
- xbzrle_encode_buffer_func = xbzrle_encode_buffer_avx512;
- }
- }
- }
-}
-#endif
-
XBZRLECacheStats xbzrle_counters;
/* used by the search for pages to send */
@@ -660,9 +632,9 @@ static int save_xbzrle_page(RAMState *rs, PageSearchStatus *pss,
memcpy(XBZRLE.current_buf, *current_data, TARGET_PAGE_SIZE);
/* XBZRLE encoding (if there is no overflow) */
- encoded_len = xbzrle_encode_buffer_func(prev_cached_page, XBZRLE.current_buf,
- TARGET_PAGE_SIZE, XBZRLE.encoded_buf,
- TARGET_PAGE_SIZE);
+ encoded_len = xbzrle_encode_buffer(prev_cached_page, XBZRLE.current_buf,
+ TARGET_PAGE_SIZE, XBZRLE.encoded_buf,
+ TARGET_PAGE_SIZE);
/*
* Update the cache contents, so that it corresponds to the data
diff --git a/migration/xbzrle.c b/migration/xbzrle.c
index 751b5428f7..3eddcf249b 100644
--- a/migration/xbzrle.c
+++ b/migration/xbzrle.c
@@ -17,8 +17,9 @@
#if defined(CONFIG_AVX512BW_OPT)
#include <immintrin.h>
+#include "host/cpuinfo.h"
-int __attribute__((target("avx512bw")))
+static int __attribute__((target("avx512bw")))
xbzrle_encode_buffer_avx512(uint8_t *old_buf, uint8_t *new_buf, int slen,
uint8_t *dst, int dlen)
{
@@ -135,6 +136,29 @@ xbzrle_encode_buffer_avx512(uint8_t *old_buf, uint8_t *new_buf, int slen,
}
return d;
}
+
+static int xbzrle_encode_buffer_int(uint8_t *old_buf, uint8_t *new_buf,
+ int slen, uint8_t *dst, int dlen);
+
+static int (*accel_func)(uint8_t *, uint8_t *, int, uint8_t *, int);
+
+static void __attribute__((constructor)) init_accel(void)
+{
+ unsigned info = cpuinfo_init();
+ if (info & CPUINFO_AVX512BW) {
+ accel_func = xbzrle_encode_buffer_avx512;
+ } else {
+ accel_func = xbzrle_encode_buffer_int;
+ }
+}
+
+int xbzrle_encode_buffer(uint8_t *old_buf, uint8_t *new_buf, int slen,
+ uint8_t *dst, int dlen)
+{
+ return accel_func(old_buf, new_buf, slen, dst, dlen);
+}
+
+#define xbzrle_encode_buffer xbzrle_encode_buffer_int
#endif
/*
diff --git a/tests/bench/xbzrle-bench.c b/tests/bench/xbzrle-bench.c
deleted file mode 100644
index 8848a3a32d..0000000000
--- a/tests/bench/xbzrle-bench.c
+++ /dev/null
@@ -1,469 +0,0 @@
-/*
- * Xor Based Zero Run Length Encoding unit tests.
- *
- * Copyright 2013 Red Hat, Inc. and/or its affiliates
- *
- * Authors:
- * Orit Wasserman <owasserm@redhat.com>
- *
- * This work is licensed under the terms of the GNU GPL, version 2 or later.
- * See the COPYING file in the top-level directory.
- *
- */
-#include "qemu/osdep.h"
-#include "qemu/cutils.h"
-#include "../migration/xbzrle.h"
-
-#if defined(CONFIG_AVX512BW_OPT)
-#define XBZRLE_PAGE_SIZE 4096
-static bool is_cpu_support_avx512bw;
-#include "qemu/cpuid.h"
-static void __attribute__((constructor)) init_cpu_flag(void)
-{
- unsigned max = __get_cpuid_max(0, NULL);
- int a, b, c, d;
- is_cpu_support_avx512bw = false;
- if (max >= 1) {
- __cpuid(1, a, b, c, d);
- /* We must check that AVX is not just available, but usable. */
- if ((c & bit_OSXSAVE) && (c & bit_AVX) && max >= 7) {
- int bv;
- __asm("xgetbv" : "=a"(bv), "=d"(d) : "c"(0));
- __cpuid_count(7, 0, a, b, c, d);
- /* 0xe6:
- * XCR0[7:5] = 111b (OPMASK state, upper 256-bit of ZMM0-ZMM15
- * and ZMM16-ZMM31 state are enabled by OS)
- * XCR0[2:1] = 11b (XMM state and YMM state are enabled by OS)
- */
- if ((bv & 0xe6) == 0xe6 && (b & bit_AVX512BW)) {
- is_cpu_support_avx512bw = true;
- }
- }
- }
- return ;
-}
-
-struct ResTime {
- float t_raw;
- float t_512;
-};
-
-
-/* Function prototypes
-int xbzrle_encode_buffer_avx512(uint8_t *old_buf, uint8_t *new_buf, int slen,
- uint8_t *dst, int dlen);
-*/
-static void encode_decode_zero(struct ResTime *res)
-{
- uint8_t *buffer = g_malloc0(XBZRLE_PAGE_SIZE);
- uint8_t *compressed = g_malloc0(XBZRLE_PAGE_SIZE);
- uint8_t *buffer512 = g_malloc0(XBZRLE_PAGE_SIZE);
- uint8_t *compressed512 = g_malloc0(XBZRLE_PAGE_SIZE);
- int i = 0;
- int dlen = 0, dlen512 = 0;
- int diff_len = g_test_rand_int_range(0, XBZRLE_PAGE_SIZE - 1006);
-
- for (i = diff_len; i > 0; i--) {
- buffer[1000 + i] = i;
- buffer512[1000 + i] = i;
- }
-
- buffer[1000 + diff_len + 3] = 103;
- buffer[1000 + diff_len + 5] = 105;
-
- buffer512[1000 + diff_len + 3] = 103;
- buffer512[1000 + diff_len + 5] = 105;
-
- /* encode zero page */
- time_t t_start, t_end, t_start512, t_end512;
- t_start = clock();
- dlen = xbzrle_encode_buffer(buffer, buffer, XBZRLE_PAGE_SIZE, compressed,
- XBZRLE_PAGE_SIZE);
- t_end = clock();
- float time_val = difftime(t_end, t_start);
- g_assert(dlen == 0);
-
- t_start512 = clock();
- dlen512 = xbzrle_encode_buffer_avx512(buffer512, buffer512, XBZRLE_PAGE_SIZE,
- compressed512, XBZRLE_PAGE_SIZE);
- t_end512 = clock();
- float time_val512 = difftime(t_end512, t_start512);
- g_assert(dlen512 == 0);
-
- res->t_raw = time_val;
- res->t_512 = time_val512;
-
- g_free(buffer);
- g_free(compressed);
- g_free(buffer512);
- g_free(compressed512);
-
-}
-
-static void test_encode_decode_zero_avx512(void)
-{
- int i;
- float time_raw = 0.0, time_512 = 0.0;
- struct ResTime res;
- for (i = 0; i < 10000; i++) {
- encode_decode_zero(&res);
- time_raw += res.t_raw;
- time_512 += res.t_512;
- }
- printf("Zero test:\n");
- printf("Raw xbzrle_encode time is %f ms\n", time_raw);
- printf("512 xbzrle_encode time is %f ms\n", time_512);
-}
-
-static void encode_decode_unchanged(struct ResTime *res)
-{
- uint8_t *compressed = g_malloc0(XBZRLE_PAGE_SIZE);
- uint8_t *test = g_malloc0(XBZRLE_PAGE_SIZE);
- uint8_t *compressed512 = g_malloc0(XBZRLE_PAGE_SIZE);
- uint8_t *test512 = g_malloc0(XBZRLE_PAGE_SIZE);
- int i = 0;
- int dlen = 0, dlen512 = 0;
- int diff_len = g_test_rand_int_range(0, XBZRLE_PAGE_SIZE - 1006);
-
- for (i = diff_len; i > 0; i--) {
- test[1000 + i] = i + 4;
- test512[1000 + i] = i + 4;
- }
-
- test[1000 + diff_len + 3] = 107;
- test[1000 + diff_len + 5] = 109;
-
- test512[1000 + diff_len + 3] = 107;
- test512[1000 + diff_len + 5] = 109;
-
- /* test unchanged buffer */
- time_t t_start, t_end, t_start512, t_end512;
- t_start = clock();
- dlen = xbzrle_encode_buffer(test, test, XBZRLE_PAGE_SIZE, compressed,
- XBZRLE_PAGE_SIZE);
- t_end = clock();
- float time_val = difftime(t_end, t_start);
- g_assert(dlen == 0);
-
- t_start512 = clock();
- dlen512 = xbzrle_encode_buffer_avx512(test512, test512, XBZRLE_PAGE_SIZE,
- compressed512, XBZRLE_PAGE_SIZE);
- t_end512 = clock();
- float time_val512 = difftime(t_end512, t_start512);
- g_assert(dlen512 == 0);
-
- res->t_raw = time_val;
- res->t_512 = time_val512;
-
- g_free(test);
- g_free(compressed);
- g_free(test512);
- g_free(compressed512);
-
-}
-
-static void test_encode_decode_unchanged_avx512(void)
-{
- int i;
- float time_raw = 0.0, time_512 = 0.0;
- struct ResTime res;
- for (i = 0; i < 10000; i++) {
- encode_decode_unchanged(&res);
- time_raw += res.t_raw;
- time_512 += res.t_512;
- }
- printf("Unchanged test:\n");
- printf("Raw xbzrle_encode time is %f ms\n", time_raw);
- printf("512 xbzrle_encode time is %f ms\n", time_512);
-}
-
-static void encode_decode_1_byte(struct ResTime *res)
-{
- uint8_t *buffer = g_malloc0(XBZRLE_PAGE_SIZE);
- uint8_t *test = g_malloc0(XBZRLE_PAGE_SIZE);
- uint8_t *compressed = g_malloc(XBZRLE_PAGE_SIZE);
- uint8_t *buffer512 = g_malloc0(XBZRLE_PAGE_SIZE);
- uint8_t *test512 = g_malloc0(XBZRLE_PAGE_SIZE);
- uint8_t *compressed512 = g_malloc(XBZRLE_PAGE_SIZE);
- int dlen = 0, rc = 0, dlen512 = 0, rc512 = 0;
- uint8_t buf[2];
- uint8_t buf512[2];
-
- test[XBZRLE_PAGE_SIZE - 1] = 1;
- test512[XBZRLE_PAGE_SIZE - 1] = 1;
-
- time_t t_start, t_end, t_start512, t_end512;
- t_start = clock();
- dlen = xbzrle_encode_buffer(buffer, test, XBZRLE_PAGE_SIZE, compressed,
- XBZRLE_PAGE_SIZE);
- t_end = clock();
- float time_val = difftime(t_end, t_start);
- g_assert(dlen == (uleb128_encode_small(&buf[0], 4095) + 2));
-
- rc = xbzrle_decode_buffer(compressed, dlen, buffer, XBZRLE_PAGE_SIZE);
- g_assert(rc == XBZRLE_PAGE_SIZE);
- g_assert(memcmp(test, buffer, XBZRLE_PAGE_SIZE) == 0);
-
- t_start512 = clock();
- dlen512 = xbzrle_encode_buffer_avx512(buffer512, test512, XBZRLE_PAGE_SIZE,
- compressed512, XBZRLE_PAGE_SIZE);
- t_end512 = clock();
- float time_val512 = difftime(t_end512, t_start512);
- g_assert(dlen512 == (uleb128_encode_small(&buf512[0], 4095) + 2));
-
- rc512 = xbzrle_decode_buffer(compressed512, dlen512, buffer512,
- XBZRLE_PAGE_SIZE);
- g_assert(rc512 == XBZRLE_PAGE_SIZE);
- g_assert(memcmp(test512, buffer512, XBZRLE_PAGE_SIZE) == 0);
-
- res->t_raw = time_val;
- res->t_512 = time_val512;
-
- g_free(buffer);
- g_free(compressed);
- g_free(test);
- g_free(buffer512);
- g_free(compressed512);
- g_free(test512);
-
-}
-
-static void test_encode_decode_1_byte_avx512(void)
-{
- int i;
- float time_raw = 0.0, time_512 = 0.0;
- struct ResTime res;
- for (i = 0; i < 10000; i++) {
- encode_decode_1_byte(&res);
- time_raw += res.t_raw;
- time_512 += res.t_512;
- }
- printf("1 byte test:\n");
- printf("Raw xbzrle_encode time is %f ms\n", time_raw);
- printf("512 xbzrle_encode time is %f ms\n", time_512);
-}
-
-static void encode_decode_overflow(struct ResTime *res)
-{
- uint8_t *compressed = g_malloc0(XBZRLE_PAGE_SIZE);
- uint8_t *test = g_malloc0(XBZRLE_PAGE_SIZE);
- uint8_t *buffer = g_malloc0(XBZRLE_PAGE_SIZE);
- uint8_t *compressed512 = g_malloc0(XBZRLE_PAGE_SIZE);
- uint8_t *test512 = g_malloc0(XBZRLE_PAGE_SIZE);
- uint8_t *buffer512 = g_malloc0(XBZRLE_PAGE_SIZE);
- int i = 0, rc = 0, rc512 = 0;
-
- for (i = 0; i < XBZRLE_PAGE_SIZE / 2 - 1; i++) {
- test[i * 2] = 1;
- test512[i * 2] = 1;
- }
-
- /* encode overflow */
- time_t t_start, t_end, t_start512, t_end512;
- t_start = clock();
- rc = xbzrle_encode_buffer(buffer, test, XBZRLE_PAGE_SIZE, compressed,
- XBZRLE_PAGE_SIZE);
- t_end = clock();
- float time_val = difftime(t_end, t_start);
- g_assert(rc == -1);
-
- t_start512 = clock();
- rc512 = xbzrle_encode_buffer_avx512(buffer512, test512, XBZRLE_PAGE_SIZE,
- compressed512, XBZRLE_PAGE_SIZE);
- t_end512 = clock();
- float time_val512 = difftime(t_end512, t_start512);
- g_assert(rc512 == -1);
-
- res->t_raw = time_val;
- res->t_512 = time_val512;
-
- g_free(buffer);
- g_free(compressed);
- g_free(test);
- g_free(buffer512);
- g_free(compressed512);
- g_free(test512);
-
-}
-
-static void test_encode_decode_overflow_avx512(void)
-{
- int i;
- float time_raw = 0.0, time_512 = 0.0;
- struct ResTime res;
- for (i = 0; i < 10000; i++) {
- encode_decode_overflow(&res);
- time_raw += res.t_raw;
- time_512 += res.t_512;
- }
- printf("Overflow test:\n");
- printf("Raw xbzrle_encode time is %f ms\n", time_raw);
- printf("512 xbzrle_encode time is %f ms\n", time_512);
-}
-
-static void encode_decode_range_avx512(struct ResTime *res)
-{
- uint8_t *buffer = g_malloc0(XBZRLE_PAGE_SIZE);
- uint8_t *compressed = g_malloc(XBZRLE_PAGE_SIZE);
- uint8_t *test = g_malloc0(XBZRLE_PAGE_SIZE);
- uint8_t *buffer512 = g_malloc0(XBZRLE_PAGE_SIZE);
- uint8_t *compressed512 = g_malloc(XBZRLE_PAGE_SIZE);
- uint8_t *test512 = g_malloc0(XBZRLE_PAGE_SIZE);
- int i = 0, rc = 0, rc512 = 0;
- int dlen = 0, dlen512 = 0;
-
- int diff_len = g_test_rand_int_range(0, XBZRLE_PAGE_SIZE - 1006);
-
- for (i = diff_len; i > 0; i--) {
- buffer[1000 + i] = i;
- test[1000 + i] = i + 4;
- buffer512[1000 + i] = i;
- test512[1000 + i] = i + 4;
- }
-
- buffer[1000 + diff_len + 3] = 103;
- test[1000 + diff_len + 3] = 107;
-
- buffer[1000 + diff_len + 5] = 105;
- test[1000 + diff_len + 5] = 109;
-
- buffer512[1000 + diff_len + 3] = 103;
- test512[1000 + diff_len + 3] = 107;
-
- buffer512[1000 + diff_len + 5] = 105;
- test512[1000 + diff_len + 5] = 109;
-
- /* test encode/decode */
- time_t t_start, t_end, t_start512, t_end512;
- t_start = clock();
- dlen = xbzrle_encode_buffer(test, buffer, XBZRLE_PAGE_SIZE, compressed,
- XBZRLE_PAGE_SIZE);
- t_end = clock();
- float time_val = difftime(t_end, t_start);
- rc = xbzrle_decode_buffer(compressed, dlen, test, XBZRLE_PAGE_SIZE);
- g_assert(rc < XBZRLE_PAGE_SIZE);
- g_assert(memcmp(test, buffer, XBZRLE_PAGE_SIZE) == 0);
-
- t_start512 = clock();
- dlen512 = xbzrle_encode_buffer_avx512(test512, buffer512, XBZRLE_PAGE_SIZE,
- compressed512, XBZRLE_PAGE_SIZE);
- t_end512 = clock();
- float time_val512 = difftime(t_end512, t_start512);
- rc512 = xbzrle_decode_buffer(compressed512, dlen512, test512, XBZRLE_PAGE_SIZE);
- g_assert(rc512 < XBZRLE_PAGE_SIZE);
- g_assert(memcmp(test512, buffer512, XBZRLE_PAGE_SIZE) == 0);
-
- res->t_raw = time_val;
- res->t_512 = time_val512;
-
- g_free(buffer);
- g_free(compressed);
- g_free(test);
- g_free(buffer512);
- g_free(compressed512);
- g_free(test512);
-
-}
-
-static void test_encode_decode_avx512(void)
-{
- int i;
- float time_raw = 0.0, time_512 = 0.0;
- struct ResTime res;
- for (i = 0; i < 10000; i++) {
- encode_decode_range_avx512(&res);
- time_raw += res.t_raw;
- time_512 += res.t_512;
- }
- printf("Encode decode test:\n");
- printf("Raw xbzrle_encode time is %f ms\n", time_raw);
- printf("512 xbzrle_encode time is %f ms\n", time_512);
-}
-
-static void encode_decode_random(struct ResTime *res)
-{
- uint8_t *buffer = g_malloc0(XBZRLE_PAGE_SIZE);
- uint8_t *compressed = g_malloc(XBZRLE_PAGE_SIZE);
- uint8_t *test = g_malloc0(XBZRLE_PAGE_SIZE);
- uint8_t *buffer512 = g_malloc0(XBZRLE_PAGE_SIZE);
- uint8_t *compressed512 = g_malloc(XBZRLE_PAGE_SIZE);
- uint8_t *test512 = g_malloc0(XBZRLE_PAGE_SIZE);
- int i = 0, rc = 0, rc512 = 0;
- int dlen = 0, dlen512 = 0;
-
- int diff_len = g_test_rand_int_range(0, XBZRLE_PAGE_SIZE - 1);
- /* store the index of diff */
- int dirty_index[diff_len];
- for (int j = 0; j < diff_len; j++) {
- dirty_index[j] = g_test_rand_int_range(0, XBZRLE_PAGE_SIZE - 1);
- }
- for (i = diff_len - 1; i >= 0; i--) {
- buffer[dirty_index[i]] = i;
- test[dirty_index[i]] = i + 4;
- buffer512[dirty_index[i]] = i;
- test512[dirty_index[i]] = i + 4;
- }
-
- time_t t_start, t_end, t_start512, t_end512;
- t_start = clock();
- dlen = xbzrle_encode_buffer(test, buffer, XBZRLE_PAGE_SIZE, compressed,
- XBZRLE_PAGE_SIZE);
- t_end = clock();
- float time_val = difftime(t_end, t_start);
- rc = xbzrle_decode_buffer(compressed, dlen, test, XBZRLE_PAGE_SIZE);
- g_assert(rc < XBZRLE_PAGE_SIZE);
-
- t_start512 = clock();
- dlen512 = xbzrle_encode_buffer_avx512(test512, buffer512, XBZRLE_PAGE_SIZE,
- compressed512, XBZRLE_PAGE_SIZE);
- t_end512 = clock();
- float time_val512 = difftime(t_end512, t_start512);
- rc512 = xbzrle_decode_buffer(compressed512, dlen512, test512, XBZRLE_PAGE_SIZE);
- g_assert(rc512 < XBZRLE_PAGE_SIZE);
-
- res->t_raw = time_val;
- res->t_512 = time_val512;
-
- g_free(buffer);
- g_free(compressed);
- g_free(test);
- g_free(buffer512);
- g_free(compressed512);
- g_free(test512);
-
-}
-
-static void test_encode_decode_random_avx512(void)
-{
- int i;
- float time_raw = 0.0, time_512 = 0.0;
- struct ResTime res;
- for (i = 0; i < 10000; i++) {
- encode_decode_random(&res);
- time_raw += res.t_raw;
- time_512 += res.t_512;
- }
- printf("Random test:\n");
- printf("Raw xbzrle_encode time is %f ms\n", time_raw);
- printf("512 xbzrle_encode time is %f ms\n", time_512);
-}
-#endif
-
-int main(int argc, char **argv)
-{
- g_test_init(&argc, &argv, NULL);
- g_test_rand_int();
- #if defined(CONFIG_AVX512BW_OPT)
- if (likely(is_cpu_support_avx512bw)) {
- g_test_add_func("/xbzrle/encode_decode_zero", test_encode_decode_zero_avx512);
- g_test_add_func("/xbzrle/encode_decode_unchanged",
- test_encode_decode_unchanged_avx512);
- g_test_add_func("/xbzrle/encode_decode_1_byte", test_encode_decode_1_byte_avx512);
- g_test_add_func("/xbzrle/encode_decode_overflow",
- test_encode_decode_overflow_avx512);
- g_test_add_func("/xbzrle/encode_decode", test_encode_decode_avx512);
- g_test_add_func("/xbzrle/encode_decode_random", test_encode_decode_random_avx512);
- }
- #endif
- return g_test_run();
-}
diff --git a/tests/unit/test-xbzrle.c b/tests/unit/test-xbzrle.c
index 547046d093..b6996de69a 100644
--- a/tests/unit/test-xbzrle.c
+++ b/tests/unit/test-xbzrle.c
@@ -16,35 +16,6 @@
#define XBZRLE_PAGE_SIZE 4096
-int (*xbzrle_encode_buffer_func)(uint8_t *, uint8_t *, int,
- uint8_t *, int) = xbzrle_encode_buffer;
-#if defined(CONFIG_AVX512BW_OPT)
-#include "qemu/cpuid.h"
-static void __attribute__((constructor)) init_cpu_flag(void)
-{
- unsigned max = __get_cpuid_max(0, NULL);
- int a, b, c, d;
- if (max >= 1) {
- __cpuid(1, a, b, c, d);
- /* We must check that AVX is not just available, but usable. */
- if ((c & bit_OSXSAVE) && (c & bit_AVX) && max >= 7) {
- int bv;
- __asm("xgetbv" : "=a"(bv), "=d"(d) : "c"(0));
- __cpuid_count(7, 0, a, b, c, d);
- /* 0xe6:
- * XCR0[7:5] = 111b (OPMASK state, upper 256-bit of ZMM0-ZMM15
- * and ZMM16-ZMM31 state are enabled by OS)
- * XCR0[2:1] = 11b (XMM state and YMM state are enabled by OS)
- */
- if ((bv & 0xe6) == 0xe6 && (b & bit_AVX512BW)) {
- xbzrle_encode_buffer_func = xbzrle_encode_buffer_avx512;
- }
- }
- }
- return ;
-}
-#endif
-
static void test_uleb(void)
{
uint32_t i, val;
@@ -83,8 +54,8 @@ static void test_encode_decode_zero(void)
buffer[1000 + diff_len + 5] = 105;
/* encode zero page */
- dlen = xbzrle_encode_buffer_func(buffer, buffer, XBZRLE_PAGE_SIZE, compressed,
- XBZRLE_PAGE_SIZE);
+ dlen = xbzrle_encode_buffer(buffer, buffer, XBZRLE_PAGE_SIZE,
+ compressed, XBZRLE_PAGE_SIZE);
g_assert(dlen == 0);
g_free(buffer);
@@ -107,8 +78,8 @@ static void test_encode_decode_unchanged(void)
test[1000 + diff_len + 5] = 109;
/* test unchanged buffer */
- dlen = xbzrle_encode_buffer_func(test, test, XBZRLE_PAGE_SIZE, compressed,
- XBZRLE_PAGE_SIZE);
+ dlen = xbzrle_encode_buffer(test, test, XBZRLE_PAGE_SIZE,
+ compressed, XBZRLE_PAGE_SIZE);
g_assert(dlen == 0);
g_free(test);
@@ -125,8 +96,8 @@ static void test_encode_decode_1_byte(void)
test[XBZRLE_PAGE_SIZE - 1] = 1;
- dlen = xbzrle_encode_buffer_func(buffer, test, XBZRLE_PAGE_SIZE, compressed,
- XBZRLE_PAGE_SIZE);
+ dlen = xbzrle_encode_buffer(buffer, test, XBZRLE_PAGE_SIZE,
+ compressed, XBZRLE_PAGE_SIZE);
g_assert(dlen == (uleb128_encode_small(&buf[0], 4095) + 2));
rc = xbzrle_decode_buffer(compressed, dlen, buffer, XBZRLE_PAGE_SIZE);
@@ -150,8 +121,8 @@ static void test_encode_decode_overflow(void)
}
/* encode overflow */
- rc = xbzrle_encode_buffer_func(buffer, test, XBZRLE_PAGE_SIZE, compressed,
- XBZRLE_PAGE_SIZE);
+ rc = xbzrle_encode_buffer(buffer, test, XBZRLE_PAGE_SIZE,
+ compressed, XBZRLE_PAGE_SIZE);
g_assert(rc == -1);
g_free(buffer);
@@ -181,8 +152,8 @@ static void encode_decode_range(void)
test[1000 + diff_len + 5] = 109;
/* test encode/decode */
- dlen = xbzrle_encode_buffer_func(test, buffer, XBZRLE_PAGE_SIZE, compressed,
- XBZRLE_PAGE_SIZE);
+ dlen = xbzrle_encode_buffer(test, buffer, XBZRLE_PAGE_SIZE,
+ compressed, XBZRLE_PAGE_SIZE);
rc = xbzrle_decode_buffer(compressed, dlen, test, XBZRLE_PAGE_SIZE);
g_assert(rc < XBZRLE_PAGE_SIZE);
diff --git a/tests/bench/meson.build b/tests/bench/meson.build
index 4e6b469066..3c799dbd98 100644
--- a/tests/bench/meson.build
+++ b/tests/bench/meson.build
@@ -3,12 +3,6 @@ qht_bench = executable('qht-bench',
sources: 'qht-bench.c',
dependencies: [qemuutil])
-if have_system
-xbzrle_bench = executable('xbzrle-bench',
- sources: 'xbzrle-bench.c',
- dependencies: [qemuutil,migration])
-endif
-
qtree_bench = executable('qtree-bench',
sources: 'qtree-bench.c',
dependencies: [qemuutil])
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 08/27] migration: Build migration_files once
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (6 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 07/27] migration/xbzrle: Use i386 host/cpuinfo.h Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-20 16:26 ` [PATCH 09/27] util: Add cpuinfo-aarch64.c Richard Henderson
` (18 subsequent siblings)
26 siblings, 0 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm, Juan Quintela
The items in migration_files are built for libmigration and included
info softmmu_ss from there; no need to also include them directly.
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
migration/meson.build | 1 -
1 file changed, 1 deletion(-)
diff --git a/migration/meson.build b/migration/meson.build
index a8e01e70ae..8ba6e420fe 100644
--- a/migration/meson.build
+++ b/migration/meson.build
@@ -8,7 +8,6 @@ migration_files = files(
'qemu-file.c',
'yank_functions.c',
)
-softmmu_ss.add(migration_files)
softmmu_ss.add(files(
'block-dirty-bitmap.c',
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 09/27] util: Add cpuinfo-aarch64.c
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (7 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 08/27] migration: Build migration_files once Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-20 16:26 ` [PATCH 10/27] include/host: Split out atomic128-cas.h Richard Henderson
` (17 subsequent siblings)
26 siblings, 0 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm, Peter Maydell
Move the code from tcg/. The only use of these bits so far
is with respect to the atomicity of tcg operations.
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
host/include/aarch64/host/cpuinfo.h | 22 ++++++++++
tcg/aarch64/tcg-target.h | 6 ++-
util/cpuinfo-aarch64.c | 67 +++++++++++++++++++++++++++++
tcg/aarch64/tcg-target.c.inc | 40 -----------------
util/meson.build | 4 +-
5 files changed, 96 insertions(+), 43 deletions(-)
create mode 100644 host/include/aarch64/host/cpuinfo.h
create mode 100644 util/cpuinfo-aarch64.c
diff --git a/host/include/aarch64/host/cpuinfo.h b/host/include/aarch64/host/cpuinfo.h
new file mode 100644
index 0000000000..82227890b4
--- /dev/null
+++ b/host/include/aarch64/host/cpuinfo.h
@@ -0,0 +1,22 @@
+/*
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ * Host specific cpu indentification for AArch64.
+ */
+
+#ifndef HOST_CPUINFO_H
+#define HOST_CPUINFO_H
+
+#define CPUINFO_ALWAYS (1u << 0) /* so cpuinfo is nonzero */
+#define CPUINFO_LSE (1u << 1)
+#define CPUINFO_LSE2 (1u << 2)
+
+/* Initialized with a constructor. */
+extern unsigned cpuinfo;
+
+/*
+ * We cannot rely on constructor ordering, so other constructors must
+ * use the function interface rather than the variable above.
+ */
+unsigned cpuinfo_init(void);
+
+#endif /* HOST_CPUINFO_H */
diff --git a/tcg/aarch64/tcg-target.h b/tcg/aarch64/tcg-target.h
index 74ee2ed255..d5f7614880 100644
--- a/tcg/aarch64/tcg-target.h
+++ b/tcg/aarch64/tcg-target.h
@@ -13,6 +13,8 @@
#ifndef AARCH64_TCG_TARGET_H
#define AARCH64_TCG_TARGET_H
+#include "host/cpuinfo.h"
+
#define TCG_TARGET_INSN_UNIT_SIZE 4
#define TCG_TARGET_TLB_DISPLACEMENT_BITS 24
#define MAX_CODE_GEN_BUFFER_SIZE ((size_t)-1)
@@ -57,8 +59,8 @@ typedef enum {
#define TCG_TARGET_CALL_ARG_I128 TCG_CALL_ARG_EVEN
#define TCG_TARGET_CALL_RET_I128 TCG_CALL_RET_NORMAL
-extern bool have_lse;
-extern bool have_lse2;
+#define have_lse (cpuinfo & CPUINFO_LSE)
+#define have_lse2 (cpuinfo & CPUINFO_LSE2)
/* optional instructions */
#define TCG_TARGET_HAS_div_i32 1
diff --git a/util/cpuinfo-aarch64.c b/util/cpuinfo-aarch64.c
new file mode 100644
index 0000000000..f99acb7884
--- /dev/null
+++ b/util/cpuinfo-aarch64.c
@@ -0,0 +1,67 @@
+/*
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ * Host specific cpu indentification for AArch64.
+ */
+
+#include "qemu/osdep.h"
+#include "host/cpuinfo.h"
+
+#ifdef CONFIG_LINUX
+# ifdef CONFIG_GETAUXVAL
+# include <sys/auxv.h>
+# else
+# include <asm/hwcap.h>
+# include "elf.h"
+# endif
+#endif
+#ifdef CONFIG_DARWIN
+# include <sys/sysctl.h>
+#endif
+
+unsigned cpuinfo;
+
+#ifdef CONFIG_DARWIN
+static bool sysctl_for_bool(const char *name)
+{
+ int val = 0;
+ size_t len = sizeof(val);
+
+ if (sysctlbyname(name, &val, &len, NULL, 0) == 0) {
+ return val != 0;
+ }
+
+ /*
+ * We might in the future ask for properties not present in older kernels,
+ * but we're only asking about static properties, all of which should be
+ * 'int'. So we shouln't see ENOMEM (val too small), or any of the other
+ * more exotic errors.
+ */
+ assert(errno == ENOENT);
+ return false;
+}
+#endif
+
+/* Called both as constructor and (possibly) via other constructors. */
+unsigned __attribute__((constructor)) cpuinfo_init(void)
+{
+ unsigned info = cpuinfo;
+
+ if (info) {
+ return info;
+ }
+
+ info = CPUINFO_ALWAYS;
+
+#ifdef CONFIG_LINUX
+ unsigned long hwcap = qemu_getauxval(AT_HWCAP);
+ info |= (hwcap & HWCAP_ATOMICS ? CPUINFO_LSE : 0);
+ info |= (hwcap & HWCAP_USCAT ? CPUINFO_LSE2 : 0);
+#endif
+#ifdef CONFIG_DARWIN
+ info |= sysctl_for_bool("hw.optional.arm.FEAT_LSE") * CPUINFO_LSE;
+ info |= sysctl_for_bool("hw.optional.arm.FEAT_LSE2") * CPUINFO_LSE2;
+#endif
+
+ cpuinfo = info;
+ return info;
+}
diff --git a/tcg/aarch64/tcg-target.c.inc b/tcg/aarch64/tcg-target.c.inc
index bc6b99a1bd..84283665e7 100644
--- a/tcg/aarch64/tcg-target.c.inc
+++ b/tcg/aarch64/tcg-target.c.inc
@@ -13,12 +13,6 @@
#include "../tcg-ldst.c.inc"
#include "../tcg-pool.c.inc"
#include "qemu/bitops.h"
-#ifdef __linux__
-#include <asm/hwcap.h>
-#endif
-#ifdef CONFIG_DARWIN
-#include <sys/sysctl.h>
-#endif
/* We're going to re-use TCGType in setting of the SF bit, which controls
the size of the operation performed. If we know the values match, it
@@ -77,9 +71,6 @@ static TCGReg tcg_target_call_oarg_reg(TCGCallReturnKind kind, int slot)
return TCG_REG_X0 + slot;
}
-bool have_lse;
-bool have_lse2;
-
#define TCG_REG_TMP TCG_REG_X30
#define TCG_VEC_TMP TCG_REG_V31
@@ -2878,39 +2869,8 @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op)
}
}
-#ifdef CONFIG_DARWIN
-static bool sysctl_for_bool(const char *name)
-{
- int val = 0;
- size_t len = sizeof(val);
-
- if (sysctlbyname(name, &val, &len, NULL, 0) == 0) {
- return val != 0;
- }
-
- /*
- * We might in the future ask for properties not present in older kernels,
- * but we're only asking about static properties, all of which should be
- * 'int'. So we shouln't see ENOMEM (val too small), or any of the other
- * more exotic errors.
- */
- assert(errno == ENOENT);
- return false;
-}
-#endif
-
static void tcg_target_init(TCGContext *s)
{
-#ifdef __linux__
- unsigned long hwcap = qemu_getauxval(AT_HWCAP);
- have_lse = hwcap & HWCAP_ATOMICS;
- have_lse2 = hwcap & HWCAP_USCAT;
-#endif
-#ifdef CONFIG_DARWIN
- have_lse = sysctl_for_bool("hw.optional.arm.FEAT_LSE");
- have_lse2 = sysctl_for_bool("hw.optional.arm.FEAT_LSE2");
-#endif
-
tcg_target_available_regs[TCG_TYPE_I32] = 0xffffffffu;
tcg_target_available_regs[TCG_TYPE_I64] = 0xffffffffu;
tcg_target_available_regs[TCG_TYPE_V64] = 0xffffffff00000000ull;
diff --git a/util/meson.build b/util/meson.build
index b3be9fad5d..3a93071d27 100644
--- a/util/meson.build
+++ b/util/meson.build
@@ -109,6 +109,8 @@ if have_block
util_ss.add(when: 'CONFIG_LINUX', if_true: files('vfio-helpers.c'))
endif
-if cpu in ['x86', 'x86_64']
+if cpu == 'aarch64'
+ util_ss.add(files('cpuinfo-aarch64.c'))
+elif cpu in ['x86', 'x86_64']
util_ss.add(files('cpuinfo-i386.c'))
endif
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 10/27] include/host: Split out atomic128-cas.h
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (8 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 09/27] util: Add cpuinfo-aarch64.c Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-21 10:44 ` Philippe Mathieu-Daudé
2023-05-20 16:26 ` [PATCH 11/27] include/host: Split out atomic128-ldst.h Richard Henderson
` (16 subsequent siblings)
26 siblings, 1 reply; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm
Separates the aarch64-specific portion into its own file.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
host/include/aarch64/host/atomic128-cas.h | 43 ++++++++++++++++++
host/include/generic/host/atomic128-cas.h | 43 ++++++++++++++++++
include/qemu/atomic128.h | 55 +----------------------
3 files changed, 87 insertions(+), 54 deletions(-)
create mode 100644 host/include/aarch64/host/atomic128-cas.h
create mode 100644 host/include/generic/host/atomic128-cas.h
diff --git a/host/include/aarch64/host/atomic128-cas.h b/host/include/aarch64/host/atomic128-cas.h
new file mode 100644
index 0000000000..1247995419
--- /dev/null
+++ b/host/include/aarch64/host/atomic128-cas.h
@@ -0,0 +1,43 @@
+/*
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ * Compare-and-swap for 128-bit atomic operations, generic version.
+ *
+ * Copyright (C) 2018, 2023 Linaro, Ltd.
+ *
+ * See docs/devel/atomics.rst for discussion about the guarantees each
+ * atomic primitive is meant to provide.
+ */
+
+#ifndef AARCH64_ATOMIC128_CAS_H
+#define AARCH64_ATOMIC128_CAS_H
+
+/* Through gcc 10, aarch64 has no support for 128-bit atomics. */
+#if defined(CONFIG_ATOMIC128) || defined(CONFIG_CMPXCHG128)
+#include "host/include/generic/host/atomic128-cas.h"
+#else
+static inline Int128 atomic16_cmpxchg(Int128 *ptr, Int128 cmp, Int128 new)
+{
+ uint64_t cmpl = int128_getlo(cmp), cmph = int128_gethi(cmp);
+ uint64_t newl = int128_getlo(new), newh = int128_gethi(new);
+ uint64_t oldl, oldh;
+ uint32_t tmp;
+
+ asm("0: ldaxp %[oldl], %[oldh], %[mem]\n\t"
+ "cmp %[oldl], %[cmpl]\n\t"
+ "ccmp %[oldh], %[cmph], #0, eq\n\t"
+ "b.ne 1f\n\t"
+ "stlxp %w[tmp], %[newl], %[newh], %[mem]\n\t"
+ "cbnz %w[tmp], 0b\n"
+ "1:"
+ : [mem] "+m"(*ptr), [tmp] "=&r"(tmp),
+ [oldl] "=&r"(oldl), [oldh] "=&r"(oldh)
+ : [cmpl] "r"(cmpl), [cmph] "r"(cmph),
+ [newl] "r"(newl), [newh] "r"(newh)
+ : "memory", "cc");
+
+ return int128_make128(oldl, oldh);
+}
+# define HAVE_CMPXCHG128 1
+#endif
+
+#endif /* AARCH64_ATOMIC128_CAS_H */
diff --git a/host/include/generic/host/atomic128-cas.h b/host/include/generic/host/atomic128-cas.h
new file mode 100644
index 0000000000..513622fe34
--- /dev/null
+++ b/host/include/generic/host/atomic128-cas.h
@@ -0,0 +1,43 @@
+/*
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ * Compare-and-swap for 128-bit atomic operations, generic version.
+ *
+ * Copyright (C) 2018, 2023 Linaro, Ltd.
+ *
+ * See docs/devel/atomics.rst for discussion about the guarantees each
+ * atomic primitive is meant to provide.
+ */
+
+#ifndef HOST_ATOMIC128_CAS_H
+#define HOST_ATOMIC128_CAS_H
+
+#if defined(CONFIG_ATOMIC128)
+static inline Int128 atomic16_cmpxchg(Int128 *ptr, Int128 cmp, Int128 new)
+{
+ Int128Alias r, c, n;
+
+ c.s = cmp;
+ n.s = new;
+ r.i = qatomic_cmpxchg__nocheck((__int128_t *)ptr, c.i, n.i);
+ return r.s;
+}
+# define HAVE_CMPXCHG128 1
+#elif defined(CONFIG_CMPXCHG128)
+static inline Int128 atomic16_cmpxchg(Int128 *ptr, Int128 cmp, Int128 new)
+{
+ Int128Alias r, c, n;
+
+ c.s = cmp;
+ n.s = new;
+ r.i = __sync_val_compare_and_swap_16((__int128_t *)ptr, c.i, n.i);
+ return r.s;
+}
+# define HAVE_CMPXCHG128 1
+#else
+/* Fallback definition that must be optimized away, or error. */
+Int128 QEMU_ERROR("unsupported atomic")
+ atomic16_cmpxchg(Int128 *ptr, Int128 cmp, Int128 new);
+# define HAVE_CMPXCHG128 0
+#endif
+
+#endif /* HOST_ATOMIC128_CAS_H */
diff --git a/include/qemu/atomic128.h b/include/qemu/atomic128.h
index d0ba0b9c65..10a2322c44 100644
--- a/include/qemu/atomic128.h
+++ b/include/qemu/atomic128.h
@@ -41,60 +41,7 @@
* Therefore, special case each platform.
*/
-#if defined(CONFIG_ATOMIC128)
-static inline Int128 atomic16_cmpxchg(Int128 *ptr, Int128 cmp, Int128 new)
-{
- Int128Alias r, c, n;
-
- c.s = cmp;
- n.s = new;
- r.i = qatomic_cmpxchg__nocheck((__int128_t *)ptr, c.i, n.i);
- return r.s;
-}
-# define HAVE_CMPXCHG128 1
-#elif defined(CONFIG_CMPXCHG128)
-static inline Int128 atomic16_cmpxchg(Int128 *ptr, Int128 cmp, Int128 new)
-{
- Int128Alias r, c, n;
-
- c.s = cmp;
- n.s = new;
- r.i = __sync_val_compare_and_swap_16((__int128_t *)ptr, c.i, n.i);
- return r.s;
-}
-# define HAVE_CMPXCHG128 1
-#elif defined(__aarch64__)
-/* Through gcc 8, aarch64 has no support for 128-bit at all. */
-static inline Int128 atomic16_cmpxchg(Int128 *ptr, Int128 cmp, Int128 new)
-{
- uint64_t cmpl = int128_getlo(cmp), cmph = int128_gethi(cmp);
- uint64_t newl = int128_getlo(new), newh = int128_gethi(new);
- uint64_t oldl, oldh;
- uint32_t tmp;
-
- asm("0: ldaxp %[oldl], %[oldh], %[mem]\n\t"
- "cmp %[oldl], %[cmpl]\n\t"
- "ccmp %[oldh], %[cmph], #0, eq\n\t"
- "b.ne 1f\n\t"
- "stlxp %w[tmp], %[newl], %[newh], %[mem]\n\t"
- "cbnz %w[tmp], 0b\n"
- "1:"
- : [mem] "+m"(*ptr), [tmp] "=&r"(tmp),
- [oldl] "=&r"(oldl), [oldh] "=&r"(oldh)
- : [cmpl] "r"(cmpl), [cmph] "r"(cmph),
- [newl] "r"(newl), [newh] "r"(newh)
- : "memory", "cc");
-
- return int128_make128(oldl, oldh);
-}
-# define HAVE_CMPXCHG128 1
-#else
-/* Fallback definition that must be optimized away, or error. */
-Int128 QEMU_ERROR("unsupported atomic")
- atomic16_cmpxchg(Int128 *ptr, Int128 cmp, Int128 new);
-# define HAVE_CMPXCHG128 0
-#endif /* Some definition for HAVE_CMPXCHG128 */
-
+#include "host/atomic128-cas.h"
#if defined(CONFIG_ATOMIC128)
static inline Int128 atomic16_read(Int128 *ptr)
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 11/27] include/host: Split out atomic128-ldst.h
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (9 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 10/27] include/host: Split out atomic128-cas.h Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-20 16:26 ` [PATCH 12/27] meson: Fix detect atomic128 support with optimization Richard Henderson
` (15 subsequent siblings)
26 siblings, 0 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm
Separates the aarch64-specific portion into its own file.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
host/include/aarch64/host/atomic128-cas.h | 2 +-
host/include/aarch64/host/atomic128-ldst.h | 49 ++++++++++++++
host/include/generic/host/atomic128-ldst.h | 57 +++++++++++++++++
include/qemu/atomic128.h | 74 +---------------------
4 files changed, 108 insertions(+), 74 deletions(-)
create mode 100644 host/include/aarch64/host/atomic128-ldst.h
create mode 100644 host/include/generic/host/atomic128-ldst.h
diff --git a/host/include/aarch64/host/atomic128-cas.h b/host/include/aarch64/host/atomic128-cas.h
index 1247995419..33f365ce67 100644
--- a/host/include/aarch64/host/atomic128-cas.h
+++ b/host/include/aarch64/host/atomic128-cas.h
@@ -1,6 +1,6 @@
/*
* SPDX-License-Identifier: GPL-2.0-or-later
- * Compare-and-swap for 128-bit atomic operations, generic version.
+ * Compare-and-swap for 128-bit atomic operations, aarch64 version.
*
* Copyright (C) 2018, 2023 Linaro, Ltd.
*
diff --git a/host/include/aarch64/host/atomic128-ldst.h b/host/include/aarch64/host/atomic128-ldst.h
new file mode 100644
index 0000000000..c2e7b44bc5
--- /dev/null
+++ b/host/include/aarch64/host/atomic128-ldst.h
@@ -0,0 +1,49 @@
+/*
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ * Load/store for 128-bit atomic operations, aarch64 version.
+ *
+ * Copyright (C) 2018, 2023 Linaro, Ltd.
+ *
+ * See docs/devel/atomics.rst for discussion about the guarantees each
+ * atomic primitive is meant to provide.
+ */
+
+#ifndef AARCH64_ATOMIC128_LDST_H
+#define AARCH64_ATOMIC128_LDST_H
+
+/* Through gcc 10, aarch64 has no support for 128-bit atomics. */
+#if !defined(CONFIG_ATOMIC128) && !defined(CONFIG_USER_ONLY)
+/* We can do better than cmpxchg for AArch64. */
+static inline Int128 atomic16_read(Int128 *ptr)
+{
+ uint64_t l, h;
+ uint32_t tmp;
+
+ /* The load must be paired with the store to guarantee not tearing. */
+ asm("0: ldxp %[l], %[h], %[mem]\n\t"
+ "stxp %w[tmp], %[l], %[h], %[mem]\n\t"
+ "cbnz %w[tmp], 0b"
+ : [mem] "+m"(*ptr), [tmp] "=r"(tmp), [l] "=r"(l), [h] "=r"(h));
+
+ return int128_make128(l, h);
+}
+
+static inline void atomic16_set(Int128 *ptr, Int128 val)
+{
+ uint64_t l = int128_getlo(val), h = int128_gethi(val);
+ uint64_t t1, t2;
+
+ /* Load into temporaries to acquire the exclusive access lock. */
+ asm("0: ldxp %[t1], %[t2], %[mem]\n\t"
+ "stxp %w[t1], %[l], %[h], %[mem]\n\t"
+ "cbnz %w[t1], 0b"
+ : [mem] "+m"(*ptr), [t1] "=&r"(t1), [t2] "=&r"(t2)
+ : [l] "r"(l), [h] "r"(h));
+}
+
+# define HAVE_ATOMIC128 1
+#else
+#include "host/include/generic/host/atomic128-ldst.h"
+#endif
+
+#endif /* AARCH64_ATOMIC128_LDST_H */
diff --git a/host/include/generic/host/atomic128-ldst.h b/host/include/generic/host/atomic128-ldst.h
new file mode 100644
index 0000000000..e7354a9255
--- /dev/null
+++ b/host/include/generic/host/atomic128-ldst.h
@@ -0,0 +1,57 @@
+/*
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ * Load/store for 128-bit atomic operations, generic version.
+ *
+ * Copyright (C) 2018, 2023 Linaro, Ltd.
+ *
+ * See docs/devel/atomics.rst for discussion about the guarantees each
+ * atomic primitive is meant to provide.
+ */
+
+#ifndef HOST_ATOMIC128_LDST_H
+#define HOST_ATOMIC128_LDST_H
+
+#if defined(CONFIG_ATOMIC128)
+static inline Int128 atomic16_read(Int128 *ptr)
+{
+ Int128Alias r;
+
+ r.i = qatomic_read__nocheck((__int128_t *)ptr);
+ return r.s;
+}
+
+static inline void atomic16_set(Int128 *ptr, Int128 val)
+{
+ Int128Alias v;
+
+ v.s = val;
+ qatomic_set__nocheck((__int128_t *)ptr, v.i);
+}
+
+# define HAVE_ATOMIC128 1
+#elif !defined(CONFIG_USER_ONLY) && HAVE_CMPXCHG128
+static inline Int128 atomic16_read(Int128 *ptr)
+{
+ /* Maybe replace 0 with 0, returning the old value. */
+ Int128 z = int128_make64(0);
+ return atomic16_cmpxchg(ptr, z, z);
+}
+
+static inline void atomic16_set(Int128 *ptr, Int128 val)
+{
+ Int128 old = *ptr, cmp;
+ do {
+ cmp = old;
+ old = atomic16_cmpxchg(ptr, cmp, val);
+ } while (int128_ne(old, cmp));
+}
+
+# define HAVE_ATOMIC128 1
+#else
+/* Fallback definitions that must be optimized away, or error. */
+Int128 QEMU_ERROR("unsupported atomic") atomic16_read(Int128 *ptr);
+void QEMU_ERROR("unsupported atomic") atomic16_set(Int128 *ptr, Int128 val);
+# define HAVE_ATOMIC128 0
+#endif
+
+#endif /* HOST_ATOMIC128_LDST_H */
diff --git a/include/qemu/atomic128.h b/include/qemu/atomic128.h
index 10a2322c44..3a8adb4d47 100644
--- a/include/qemu/atomic128.h
+++ b/include/qemu/atomic128.h
@@ -42,78 +42,6 @@
*/
#include "host/atomic128-cas.h"
-
-#if defined(CONFIG_ATOMIC128)
-static inline Int128 atomic16_read(Int128 *ptr)
-{
- Int128Alias r;
-
- r.i = qatomic_read__nocheck((__int128_t *)ptr);
- return r.s;
-}
-
-static inline void atomic16_set(Int128 *ptr, Int128 val)
-{
- Int128Alias v;
-
- v.s = val;
- qatomic_set__nocheck((__int128_t *)ptr, v.i);
-}
-
-# define HAVE_ATOMIC128 1
-#elif !defined(CONFIG_USER_ONLY) && defined(__aarch64__)
-/* We can do better than cmpxchg for AArch64. */
-static inline Int128 atomic16_read(Int128 *ptr)
-{
- uint64_t l, h;
- uint32_t tmp;
-
- /* The load must be paired with the store to guarantee not tearing. */
- asm("0: ldxp %[l], %[h], %[mem]\n\t"
- "stxp %w[tmp], %[l], %[h], %[mem]\n\t"
- "cbnz %w[tmp], 0b"
- : [mem] "+m"(*ptr), [tmp] "=r"(tmp), [l] "=r"(l), [h] "=r"(h));
-
- return int128_make128(l, h);
-}
-
-static inline void atomic16_set(Int128 *ptr, Int128 val)
-{
- uint64_t l = int128_getlo(val), h = int128_gethi(val);
- uint64_t t1, t2;
-
- /* Load into temporaries to acquire the exclusive access lock. */
- asm("0: ldxp %[t1], %[t2], %[mem]\n\t"
- "stxp %w[t1], %[l], %[h], %[mem]\n\t"
- "cbnz %w[t1], 0b"
- : [mem] "+m"(*ptr), [t1] "=&r"(t1), [t2] "=&r"(t2)
- : [l] "r"(l), [h] "r"(h));
-}
-
-# define HAVE_ATOMIC128 1
-#elif !defined(CONFIG_USER_ONLY) && HAVE_CMPXCHG128
-static inline Int128 atomic16_read(Int128 *ptr)
-{
- /* Maybe replace 0 with 0, returning the old value. */
- Int128 z = int128_make64(0);
- return atomic16_cmpxchg(ptr, z, z);
-}
-
-static inline void atomic16_set(Int128 *ptr, Int128 val)
-{
- Int128 old = *ptr, cmp;
- do {
- cmp = old;
- old = atomic16_cmpxchg(ptr, cmp, val);
- } while (int128_ne(old, cmp));
-}
-
-# define HAVE_ATOMIC128 1
-#else
-/* Fallback definitions that must be optimized away, or error. */
-Int128 QEMU_ERROR("unsupported atomic") atomic16_read(Int128 *ptr);
-void QEMU_ERROR("unsupported atomic") atomic16_set(Int128 *ptr, Int128 val);
-# define HAVE_ATOMIC128 0
-#endif /* Some definition for HAVE_ATOMIC128 */
+#include "host/atomic128-ldst.h"
#endif /* QEMU_ATOMIC128_H */
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 12/27] meson: Fix detect atomic128 support with optimization
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (10 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 11/27] include/host: Split out atomic128-ldst.h Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-21 10:54 ` Philippe Mathieu-Daudé
2023-05-20 16:26 ` [PATCH 13/27] include/qemu: Move CONFIG_ATOMIC128_OPT handling to atomic128.h Richard Henderson
` (14 subsequent siblings)
26 siblings, 1 reply; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm
Silly typo: sizeof(16) != 16.
Fixes: e61f1efeb730 ("meson: Detect atomic128 support with optimization")
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
meson.build | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meson.build b/meson.build
index 4ffc0d3e59..5e7fc6345f 100644
--- a/meson.build
+++ b/meson.build
@@ -2555,7 +2555,7 @@ if has_int128
# __alignof(unsigned __int128) for the host.
atomic_test_128 = '''
int main(int ac, char **av) {
- unsigned __int128 *p = __builtin_assume_aligned(av[ac - 1], sizeof(16));
+ unsigned __int128 *p = __builtin_assume_aligned(av[ac - 1], 16);
p[1] = __atomic_load_n(&p[0], __ATOMIC_RELAXED);
__atomic_store_n(&p[2], p[3], __ATOMIC_RELAXED);
__atomic_compare_exchange_n(&p[4], &p[5], p[6], 0, __ATOMIC_RELAXED, __ATOMIC_RELAXED);
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 13/27] include/qemu: Move CONFIG_ATOMIC128_OPT handling to atomic128.h
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (11 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 12/27] meson: Fix detect atomic128 support with optimization Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-20 16:26 ` [PATCH 14/27] target/ppc: Use tcg_gen_qemu_{ld, st}_i128 for LQARX, LQ, STQ Richard Henderson
` (13 subsequent siblings)
26 siblings, 0 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm
Not only the routines in ldst_atomicity.c.inc need markup,
but also the ones in the headers.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
host/include/generic/host/atomic128-cas.h | 12 ++++++++----
host/include/generic/host/atomic128-ldst.h | 18 ++++++++++++------
include/qemu/atomic128.h | 17 +++++++++++++++++
accel/tcg/ldst_atomicity.c.inc | 17 -----------------
4 files changed, 37 insertions(+), 27 deletions(-)
diff --git a/host/include/generic/host/atomic128-cas.h b/host/include/generic/host/atomic128-cas.h
index 513622fe34..991d3da082 100644
--- a/host/include/generic/host/atomic128-cas.h
+++ b/host/include/generic/host/atomic128-cas.h
@@ -12,24 +12,28 @@
#define HOST_ATOMIC128_CAS_H
#if defined(CONFIG_ATOMIC128)
-static inline Int128 atomic16_cmpxchg(Int128 *ptr, Int128 cmp, Int128 new)
+static inline Int128 ATTRIBUTE_ATOMIC128_OPT
+atomic16_cmpxchg(Int128 *ptr, Int128 cmp, Int128 new)
{
+ __int128_t *ptr_align = __builtin_assume_aligned(ptr, 16);
Int128Alias r, c, n;
c.s = cmp;
n.s = new;
- r.i = qatomic_cmpxchg__nocheck((__int128_t *)ptr, c.i, n.i);
+ r.i = qatomic_cmpxchg__nocheck(ptr_align, c.i, n.i);
return r.s;
}
# define HAVE_CMPXCHG128 1
#elif defined(CONFIG_CMPXCHG128)
-static inline Int128 atomic16_cmpxchg(Int128 *ptr, Int128 cmp, Int128 new)
+static inline Int128 ATTRIBUTE_ATOMIC128_OPT
+atomic16_cmpxchg(Int128 *ptr, Int128 cmp, Int128 new)
{
+ __int128_t *ptr_align = __builtin_assume_aligned(ptr, 16);
Int128Alias r, c, n;
c.s = cmp;
n.s = new;
- r.i = __sync_val_compare_and_swap_16((__int128_t *)ptr, c.i, n.i);
+ r.i = __sync_val_compare_and_swap_16(ptr_align, c.i, n.i);
return r.s;
}
# define HAVE_CMPXCHG128 1
diff --git a/host/include/generic/host/atomic128-ldst.h b/host/include/generic/host/atomic128-ldst.h
index e7354a9255..46911dfb61 100644
--- a/host/include/generic/host/atomic128-ldst.h
+++ b/host/include/generic/host/atomic128-ldst.h
@@ -12,32 +12,38 @@
#define HOST_ATOMIC128_LDST_H
#if defined(CONFIG_ATOMIC128)
-static inline Int128 atomic16_read(Int128 *ptr)
+static inline Int128 ATTRIBUTE_ATOMIC128_OPT
+atomic16_read(Int128 *ptr)
{
+ __int128_t *ptr_align = __builtin_assume_aligned(ptr, 16);
Int128Alias r;
- r.i = qatomic_read__nocheck((__int128_t *)ptr);
+ r.i = qatomic_read__nocheck(ptr_align);
return r.s;
}
-static inline void atomic16_set(Int128 *ptr, Int128 val)
+static inline void ATTRIBUTE_ATOMIC128_OPT
+atomic16_set(Int128 *ptr, Int128 val)
{
+ __int128_t *ptr_align = __builtin_assume_aligned(ptr, 16);
Int128Alias v;
v.s = val;
- qatomic_set__nocheck((__int128_t *)ptr, v.i);
+ qatomic_set__nocheck(ptr_align, v.i);
}
# define HAVE_ATOMIC128 1
#elif !defined(CONFIG_USER_ONLY) && HAVE_CMPXCHG128
-static inline Int128 atomic16_read(Int128 *ptr)
+static inline Int128 ATTRIBUTE_ATOMIC128_OPT
+atomic16_read(Int128 *ptr)
{
/* Maybe replace 0 with 0, returning the old value. */
Int128 z = int128_make64(0);
return atomic16_cmpxchg(ptr, z, z);
}
-static inline void atomic16_set(Int128 *ptr, Int128 val)
+static inline void ATTRIBUTE_ATOMIC128_OPT
+atomic16_set(Int128 *ptr, Int128 val)
{
Int128 old = *ptr, cmp;
do {
diff --git a/include/qemu/atomic128.h b/include/qemu/atomic128.h
index 3a8adb4d47..34554bf0ac 100644
--- a/include/qemu/atomic128.h
+++ b/include/qemu/atomic128.h
@@ -15,6 +15,23 @@
#include "qemu/int128.h"
+/*
+ * If __alignof(unsigned __int128) < 16, GCC may refuse to inline atomics
+ * that are supported by the host, e.g. s390x. We can force the pointer to
+ * have our known alignment with __builtin_assume_aligned, however prior to
+ * GCC 13 that was only reliable with optimization enabled. See
+ * https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107389
+ */
+#if defined(CONFIG_ATOMIC128_OPT)
+# if !defined(__OPTIMIZE__)
+# define ATTRIBUTE_ATOMIC128_OPT __attribute__((optimize("O1")))
+# endif
+# define CONFIG_ATOMIC128
+#endif
+#ifndef ATTRIBUTE_ATOMIC128_OPT
+# define ATTRIBUTE_ATOMIC128_OPT
+#endif
+
/*
* GCC is a house divided about supporting large atomic operations.
*
diff --git a/accel/tcg/ldst_atomicity.c.inc b/accel/tcg/ldst_atomicity.c.inc
index ba5db7c366..b89631bbef 100644
--- a/accel/tcg/ldst_atomicity.c.inc
+++ b/accel/tcg/ldst_atomicity.c.inc
@@ -16,23 +16,6 @@
#endif
#define HAVE_al8_fast (ATOMIC_REG_SIZE >= 8)
-/*
- * If __alignof(unsigned __int128) < 16, GCC may refuse to inline atomics
- * that are supported by the host, e.g. s390x. We can force the pointer to
- * have our known alignment with __builtin_assume_aligned, however prior to
- * GCC 13 that was only reliable with optimization enabled. See
- * https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107389
- */
-#if defined(CONFIG_ATOMIC128_OPT)
-# if !defined(__OPTIMIZE__)
-# define ATTRIBUTE_ATOMIC128_OPT __attribute__((optimize("O1")))
-# endif
-# define CONFIG_ATOMIC128
-#endif
-#ifndef ATTRIBUTE_ATOMIC128_OPT
-# define ATTRIBUTE_ATOMIC128_OPT
-#endif
-
#if defined(CONFIG_ATOMIC128)
# define HAVE_al16_fast true
#else
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 14/27] target/ppc: Use tcg_gen_qemu_{ld, st}_i128 for LQARX, LQ, STQ
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (12 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 13/27] include/qemu: Move CONFIG_ATOMIC128_OPT handling to atomic128.h Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-20 16:26 ` [PATCH 15/27] target/s390x: Use tcg_gen_qemu_{ld, st}_i128 for LPQ, STPQ Richard Henderson
` (12 subsequent siblings)
26 siblings, 0 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel
Cc: qemu-arm, qemu-ppc, Daniel Henrique Barboza,
Cédric Le Goater, David Gibson, Greg Kurz
No need to roll our own, as this is now provided by tcg.
This was the last use of retxl, so remove that too.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
Cc: qemu-ppc@nongnu.org
Cc: Daniel Henrique Barboza <danielhb413@gmail.com>
Cc: "Cédric Le Goater" <clg@kaod.org>
Cc: David Gibson <david@gibson.dropbear.id.au>
Cc: Greg Kurz <groug@kaod.org>
---
target/ppc/cpu.h | 1 -
target/ppc/helper.h | 9 ----
target/ppc/mem_helper.c | 48 --------------------
target/ppc/translate.c | 34 ++-------------
target/ppc/translate/fixedpoint-impl.c.inc | 51 +++-------------------
5 files changed, 11 insertions(+), 132 deletions(-)
diff --git a/target/ppc/cpu.h b/target/ppc/cpu.h
index 1c02596d9f..0f9f2e1a0c 100644
--- a/target/ppc/cpu.h
+++ b/target/ppc/cpu.h
@@ -1124,7 +1124,6 @@ struct CPUArchState {
/* used to speed-up TLB assist handlers */
target_ulong nip; /* next instruction pointer */
- uint64_t retxh; /* high part of 128-bit helper return */
/* when a memory exception occurs, the access type is stored here */
int access_type;
diff --git a/target/ppc/helper.h b/target/ppc/helper.h
index 0beaca5c7a..38efbc351c 100644
--- a/target/ppc/helper.h
+++ b/target/ppc/helper.h
@@ -810,12 +810,3 @@ DEF_HELPER_4(DSCLIQ, void, env, fprp, fprp, i32)
DEF_HELPER_1(tbegin, void, env)
DEF_HELPER_FLAGS_1(fixup_thrm, TCG_CALL_NO_RWG, void, env)
-
-#ifdef TARGET_PPC64
-DEF_HELPER_FLAGS_3(lq_le_parallel, TCG_CALL_NO_WG, i64, env, tl, i32)
-DEF_HELPER_FLAGS_3(lq_be_parallel, TCG_CALL_NO_WG, i64, env, tl, i32)
-DEF_HELPER_FLAGS_5(stq_le_parallel, TCG_CALL_NO_WG,
- void, env, tl, i64, i64, i32)
-DEF_HELPER_FLAGS_5(stq_be_parallel, TCG_CALL_NO_WG,
- void, env, tl, i64, i64, i32)
-#endif
diff --git a/target/ppc/mem_helper.c b/target/ppc/mem_helper.c
index 1578887a8f..46eae65819 100644
--- a/target/ppc/mem_helper.c
+++ b/target/ppc/mem_helper.c
@@ -367,54 +367,6 @@ target_ulong helper_lscbx(CPUPPCState *env, target_ulong addr, uint32_t reg,
return i;
}
-#ifdef TARGET_PPC64
-uint64_t helper_lq_le_parallel(CPUPPCState *env, target_ulong addr,
- uint32_t opidx)
-{
- Int128 ret;
-
- /* We will have raised EXCP_ATOMIC from the translator. */
- assert(HAVE_ATOMIC128);
- ret = cpu_atomic_ldo_le_mmu(env, addr, opidx, GETPC());
- env->retxh = int128_gethi(ret);
- return int128_getlo(ret);
-}
-
-uint64_t helper_lq_be_parallel(CPUPPCState *env, target_ulong addr,
- uint32_t opidx)
-{
- Int128 ret;
-
- /* We will have raised EXCP_ATOMIC from the translator. */
- assert(HAVE_ATOMIC128);
- ret = cpu_atomic_ldo_be_mmu(env, addr, opidx, GETPC());
- env->retxh = int128_gethi(ret);
- return int128_getlo(ret);
-}
-
-void helper_stq_le_parallel(CPUPPCState *env, target_ulong addr,
- uint64_t lo, uint64_t hi, uint32_t opidx)
-{
- Int128 val;
-
- /* We will have raised EXCP_ATOMIC from the translator. */
- assert(HAVE_ATOMIC128);
- val = int128_make128(lo, hi);
- cpu_atomic_sto_le_mmu(env, addr, val, opidx, GETPC());
-}
-
-void helper_stq_be_parallel(CPUPPCState *env, target_ulong addr,
- uint64_t lo, uint64_t hi, uint32_t opidx)
-{
- Int128 val;
-
- /* We will have raised EXCP_ATOMIC from the translator. */
- assert(HAVE_ATOMIC128);
- val = int128_make128(lo, hi);
- cpu_atomic_sto_be_mmu(env, addr, val, opidx, GETPC());
-}
-#endif
-
/*****************************************************************************/
/* Altivec extension helpers */
#if HOST_BIG_ENDIAN
diff --git a/target/ppc/translate.c b/target/ppc/translate.c
index f603f1a939..1720570b9b 100644
--- a/target/ppc/translate.c
+++ b/target/ppc/translate.c
@@ -3757,6 +3757,7 @@ static void gen_lqarx(DisasContext *ctx)
{
int rd = rD(ctx->opcode);
TCGv EA, hi, lo;
+ TCGv_i128 t16;
if (unlikely((rd & 1) || (rd == rA(ctx->opcode)) ||
(rd == rB(ctx->opcode)))) {
@@ -3772,36 +3773,9 @@ static void gen_lqarx(DisasContext *ctx)
lo = cpu_gpr[rd + 1];
hi = cpu_gpr[rd];
- if (tb_cflags(ctx->base.tb) & CF_PARALLEL) {
- if (HAVE_ATOMIC128) {
- TCGv_i32 oi = tcg_temp_new_i32();
- if (ctx->le_mode) {
- tcg_gen_movi_i32(oi, make_memop_idx(MO_LE | MO_128 | MO_ALIGN,
- ctx->mem_idx));
- gen_helper_lq_le_parallel(lo, cpu_env, EA, oi);
- } else {
- tcg_gen_movi_i32(oi, make_memop_idx(MO_BE | MO_128 | MO_ALIGN,
- ctx->mem_idx));
- gen_helper_lq_be_parallel(lo, cpu_env, EA, oi);
- }
- tcg_gen_ld_i64(hi, cpu_env, offsetof(CPUPPCState, retxh));
- } else {
- /* Restart with exclusive lock. */
- gen_helper_exit_atomic(cpu_env);
- ctx->base.is_jmp = DISAS_NORETURN;
- return;
- }
- } else if (ctx->le_mode) {
- tcg_gen_qemu_ld_i64(lo, EA, ctx->mem_idx, MO_LEUQ | MO_ALIGN_16);
- tcg_gen_mov_tl(cpu_reserve, EA);
- gen_addr_add(ctx, EA, EA, 8);
- tcg_gen_qemu_ld_i64(hi, EA, ctx->mem_idx, MO_LEUQ);
- } else {
- tcg_gen_qemu_ld_i64(hi, EA, ctx->mem_idx, MO_BEUQ | MO_ALIGN_16);
- tcg_gen_mov_tl(cpu_reserve, EA);
- gen_addr_add(ctx, EA, EA, 8);
- tcg_gen_qemu_ld_i64(lo, EA, ctx->mem_idx, MO_BEUQ);
- }
+ t16 = tcg_temp_new_i128();
+ tcg_gen_qemu_ld_i128(t16, EA, ctx->mem_idx, DEF_MEMOP(MO_128 | MO_ALIGN));
+ tcg_gen_extr_i128_i64(lo, hi, t16);
tcg_gen_st_tl(hi, cpu_env, offsetof(CPUPPCState, reserve_val));
tcg_gen_st_tl(lo, cpu_env, offsetof(CPUPPCState, reserve_val2));
diff --git a/target/ppc/translate/fixedpoint-impl.c.inc b/target/ppc/translate/fixedpoint-impl.c.inc
index 02d86b77a8..f47f1a50e8 100644
--- a/target/ppc/translate/fixedpoint-impl.c.inc
+++ b/target/ppc/translate/fixedpoint-impl.c.inc
@@ -72,7 +72,7 @@ static bool do_ldst_quad(DisasContext *ctx, arg_D *a, bool store, bool prefixed)
#if defined(TARGET_PPC64)
TCGv ea;
TCGv_i64 low_addr_gpr, high_addr_gpr;
- MemOp mop;
+ TCGv_i128 t16;
REQUIRE_INSNS_FLAGS(ctx, 64BX);
@@ -101,51 +101,14 @@ static bool do_ldst_quad(DisasContext *ctx, arg_D *a, bool store, bool prefixed)
low_addr_gpr = cpu_gpr[a->rt + 1];
high_addr_gpr = cpu_gpr[a->rt];
}
+ t16 = tcg_temp_new_i128();
- if (tb_cflags(ctx->base.tb) & CF_PARALLEL) {
- if (HAVE_ATOMIC128) {
- mop = DEF_MEMOP(MO_128);
- TCGv_i32 oi = tcg_constant_i32(make_memop_idx(mop, ctx->mem_idx));
- if (store) {
- if (ctx->le_mode) {
- gen_helper_stq_le_parallel(cpu_env, ea, low_addr_gpr,
- high_addr_gpr, oi);
- } else {
- gen_helper_stq_be_parallel(cpu_env, ea, high_addr_gpr,
- low_addr_gpr, oi);
-
- }
- } else {
- if (ctx->le_mode) {
- gen_helper_lq_le_parallel(low_addr_gpr, cpu_env, ea, oi);
- tcg_gen_ld_i64(high_addr_gpr, cpu_env,
- offsetof(CPUPPCState, retxh));
- } else {
- gen_helper_lq_be_parallel(high_addr_gpr, cpu_env, ea, oi);
- tcg_gen_ld_i64(low_addr_gpr, cpu_env,
- offsetof(CPUPPCState, retxh));
- }
- }
- } else {
- /* Restart with exclusive lock. */
- gen_helper_exit_atomic(cpu_env);
- ctx->base.is_jmp = DISAS_NORETURN;
- }
+ if (store) {
+ tcg_gen_concat_i64_i128(t16, low_addr_gpr, high_addr_gpr);
+ tcg_gen_qemu_st_i128(t16, ea, ctx->mem_idx, DEF_MEMOP(MO_128));
} else {
- mop = DEF_MEMOP(MO_UQ);
- if (store) {
- tcg_gen_qemu_st_i64(low_addr_gpr, ea, ctx->mem_idx, mop);
- } else {
- tcg_gen_qemu_ld_i64(low_addr_gpr, ea, ctx->mem_idx, mop);
- }
-
- gen_addr_add(ctx, ea, ea, 8);
-
- if (store) {
- tcg_gen_qemu_st_i64(high_addr_gpr, ea, ctx->mem_idx, mop);
- } else {
- tcg_gen_qemu_ld_i64(high_addr_gpr, ea, ctx->mem_idx, mop);
- }
+ tcg_gen_qemu_ld_i128(t16, ea, ctx->mem_idx, DEF_MEMOP(MO_128));
+ tcg_gen_extr_i128_i64(low_addr_gpr, high_addr_gpr, t16);
}
#else
qemu_build_not_reached();
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 15/27] target/s390x: Use tcg_gen_qemu_{ld, st}_i128 for LPQ, STPQ
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (13 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 14/27] target/ppc: Use tcg_gen_qemu_{ld, st}_i128 for LQARX, LQ, STQ Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-22 8:35 ` [PATCH 15/27] target/s390x: Use tcg_gen_qemu_{ld,st}_i128 " David Hildenbrand
2023-05-20 16:26 ` [PATCH 16/27] accel/tcg: Unify cpu_{ld,st}*_{be,le}_mmu Richard Henderson
` (11 subsequent siblings)
26 siblings, 1 reply; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm, qemu-s390x, David Hildenbrand, Ilya Leoshkevich
No need to roll our own, as this is now provided by tcg.
This was the last use of retxl, so remove that too.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
Cc: qemu-s390x@nongnu.org
Cc: David Hildenbrand <david@redhat.com>
Cc: Ilya Leoshkevich <iii@linux.ibm.com>
---
target/s390x/cpu.h | 3 --
target/s390x/helper.h | 4 ---
target/s390x/tcg/mem_helper.c | 61 --------------------------------
target/s390x/tcg/translate.c | 30 +++++-----------
target/s390x/tcg/insn-data.h.inc | 2 +-
5 files changed, 9 insertions(+), 91 deletions(-)
diff --git a/target/s390x/cpu.h b/target/s390x/cpu.h
index c47e7adcb1..f130c29f83 100644
--- a/target/s390x/cpu.h
+++ b/target/s390x/cpu.h
@@ -76,9 +76,6 @@ struct CPUArchState {
float_status fpu_status; /* passed to softfloat lib */
- /* The low part of a 128-bit return, or remainder of a divide. */
- uint64_t retxl;
-
PSW psw;
S390CrashReason crash_reason;
diff --git a/target/s390x/helper.h b/target/s390x/helper.h
index 341bc51ec2..7529e725f2 100644
--- a/target/s390x/helper.h
+++ b/target/s390x/helper.h
@@ -108,10 +108,6 @@ DEF_HELPER_FLAGS_2(sfas, TCG_CALL_NO_WG, void, env, i64)
DEF_HELPER_FLAGS_2(srnm, TCG_CALL_NO_WG, void, env, i64)
DEF_HELPER_FLAGS_1(popcnt, TCG_CALL_NO_RWG_SE, i64, i64)
DEF_HELPER_2(stfle, i32, env, i64)
-DEF_HELPER_FLAGS_2(lpq, TCG_CALL_NO_WG, i64, env, i64)
-DEF_HELPER_FLAGS_2(lpq_parallel, TCG_CALL_NO_WG, i64, env, i64)
-DEF_HELPER_FLAGS_4(stpq, TCG_CALL_NO_WG, void, env, i64, i64, i64)
-DEF_HELPER_FLAGS_4(stpq_parallel, TCG_CALL_NO_WG, void, env, i64, i64, i64)
DEF_HELPER_4(mvcos, i32, env, i64, i64, i64)
DEF_HELPER_4(cu12, i32, env, i32, i32, i32)
DEF_HELPER_4(cu14, i32, env, i32, i32, i32)
diff --git a/target/s390x/tcg/mem_helper.c b/target/s390x/tcg/mem_helper.c
index 8b58b8d88d..0e0d66b3b6 100644
--- a/target/s390x/tcg/mem_helper.c
+++ b/target/s390x/tcg/mem_helper.c
@@ -2398,67 +2398,6 @@ uint64_t HELPER(lra)(CPUS390XState *env, uint64_t addr)
}
#endif
-/* load pair from quadword */
-uint64_t HELPER(lpq)(CPUS390XState *env, uint64_t addr)
-{
- uintptr_t ra = GETPC();
- uint64_t hi, lo;
-
- check_alignment(env, addr, 16, ra);
- hi = cpu_ldq_data_ra(env, addr + 0, ra);
- lo = cpu_ldq_data_ra(env, addr + 8, ra);
-
- env->retxl = lo;
- return hi;
-}
-
-uint64_t HELPER(lpq_parallel)(CPUS390XState *env, uint64_t addr)
-{
- uintptr_t ra = GETPC();
- uint64_t hi, lo;
- int mem_idx;
- MemOpIdx oi;
- Int128 v;
-
- assert(HAVE_ATOMIC128);
-
- mem_idx = cpu_mmu_index(env, false);
- oi = make_memop_idx(MO_TEUQ | MO_ALIGN_16, mem_idx);
- v = cpu_atomic_ldo_be_mmu(env, addr, oi, ra);
- hi = int128_gethi(v);
- lo = int128_getlo(v);
-
- env->retxl = lo;
- return hi;
-}
-
-/* store pair to quadword */
-void HELPER(stpq)(CPUS390XState *env, uint64_t addr,
- uint64_t low, uint64_t high)
-{
- uintptr_t ra = GETPC();
-
- check_alignment(env, addr, 16, ra);
- cpu_stq_data_ra(env, addr + 0, high, ra);
- cpu_stq_data_ra(env, addr + 8, low, ra);
-}
-
-void HELPER(stpq_parallel)(CPUS390XState *env, uint64_t addr,
- uint64_t low, uint64_t high)
-{
- uintptr_t ra = GETPC();
- int mem_idx;
- MemOpIdx oi;
- Int128 v;
-
- assert(HAVE_ATOMIC128);
-
- mem_idx = cpu_mmu_index(env, false);
- oi = make_memop_idx(MO_TEUQ | MO_ALIGN_16, mem_idx);
- v = int128_make128(low, high);
- cpu_atomic_sto_be_mmu(env, addr, v, oi, ra);
-}
-
/* Execute instruction. This instruction executes an insn modified with
the contents of r1. It does not change the executed instruction in memory;
it does not change the program counter.
diff --git a/target/s390x/tcg/translate.c b/target/s390x/tcg/translate.c
index d6670e6a87..3eb3708d55 100644
--- a/target/s390x/tcg/translate.c
+++ b/target/s390x/tcg/translate.c
@@ -335,11 +335,6 @@ static void store_freg32_i64(int reg, TCGv_i64 v)
tcg_gen_st32_i64(v, cpu_env, freg32_offset(reg));
}
-static void return_low128(TCGv_i64 dest)
-{
- tcg_gen_ld_i64(dest, cpu_env, offsetof(CPUS390XState, retxl));
-}
-
static void update_psw_addr(DisasContext *s)
{
/* psw.addr */
@@ -3130,15 +3125,9 @@ static DisasJumpType op_lpd(DisasContext *s, DisasOps *o)
static DisasJumpType op_lpq(DisasContext *s, DisasOps *o)
{
- if (!(tb_cflags(s->base.tb) & CF_PARALLEL)) {
- gen_helper_lpq(o->out, cpu_env, o->in2);
- } else if (HAVE_ATOMIC128) {
- gen_helper_lpq_parallel(o->out, cpu_env, o->in2);
- } else {
- gen_helper_exit_atomic(cpu_env);
- return DISAS_NORETURN;
- }
- return_low128(o->out2);
+ o->out_128 = tcg_temp_new_i128();
+ tcg_gen_qemu_ld_i128(o->out_128, o->in2, get_mem_index(s),
+ MO_TE | MO_128 | MO_ALIGN);
return DISAS_NEXT;
}
@@ -4533,14 +4522,11 @@ static DisasJumpType op_stmh(DisasContext *s, DisasOps *o)
static DisasJumpType op_stpq(DisasContext *s, DisasOps *o)
{
- if (!(tb_cflags(s->base.tb) & CF_PARALLEL)) {
- gen_helper_stpq(cpu_env, o->in2, o->out2, o->out);
- } else if (HAVE_ATOMIC128) {
- gen_helper_stpq_parallel(cpu_env, o->in2, o->out2, o->out);
- } else {
- gen_helper_exit_atomic(cpu_env);
- return DISAS_NORETURN;
- }
+ TCGv_i128 t16 = tcg_temp_new_i128();
+
+ tcg_gen_concat_i64_i128(t16, o->out2, o->out);
+ tcg_gen_qemu_st_i128(t16, o->in2, get_mem_index(s),
+ MO_TE | MO_128 | MO_ALIGN);
return DISAS_NEXT;
}
diff --git a/target/s390x/tcg/insn-data.h.inc b/target/s390x/tcg/insn-data.h.inc
index 1f1ac742a9..bcc70d99ba 100644
--- a/target/s390x/tcg/insn-data.h.inc
+++ b/target/s390x/tcg/insn-data.h.inc
@@ -570,7 +570,7 @@
D(0xc804, LPD, SSF, ILA, 0, 0, new_P, r3_P32, lpd, 0, MO_TEUL)
D(0xc805, LPDG, SSF, ILA, 0, 0, new_P, r3_P64, lpd, 0, MO_TEUQ)
/* LOAD PAIR FROM QUADWORD */
- C(0xe38f, LPQ, RXY_a, Z, 0, a2, r1_P, 0, lpq, 0)
+ C(0xe38f, LPQ, RXY_a, Z, 0, a2, 0, r1_D64, lpq, 0)
/* LOAD POSITIVE */
C(0x1000, LPR, RR_a, Z, 0, r2_32s, new, r1_32, abs, abs32)
C(0xb900, LPGR, RRE, Z, 0, r2, r1, 0, abs, abs64)
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 16/27] accel/tcg: Unify cpu_{ld,st}*_{be,le}_mmu
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (14 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 15/27] target/s390x: Use tcg_gen_qemu_{ld, st}_i128 for LPQ, STPQ Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-21 11:15 ` Philippe Mathieu-Daudé
2023-05-20 16:26 ` [PATCH 17/27] target/s390x: Use cpu_{ld,st}*_mmu in do_csst Richard Henderson
` (10 subsequent siblings)
26 siblings, 1 reply; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm
With the current structure of cputlb.c, there is no difference
between the little-endian and big-endian entry points, aside
from the assert. Unify the pairs of functions.
The only use of the functions with explicit endianness was in
target/sparc64, and that was only to satisfy the assert.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
include/exec/cpu_ldst.h | 58 ++-----
accel/tcg/cputlb.c | 122 +++-----------
accel/tcg/user-exec.c | 322 ++++++++++--------------------------
target/arm/tcg/m_helper.c | 4 +-
target/sparc/ldst_helper.c | 18 +-
accel/tcg/ldst_common.c.inc | 24 +--
6 files changed, 137 insertions(+), 411 deletions(-)
diff --git a/include/exec/cpu_ldst.h b/include/exec/cpu_ldst.h
index 7c867c94c3..fc1d3d9301 100644
--- a/include/exec/cpu_ldst.h
+++ b/include/exec/cpu_ldst.h
@@ -207,43 +207,21 @@ void cpu_stq_le_mmuidx_ra(CPUArchState *env, abi_ptr ptr, uint64_t val,
int mmu_idx, uintptr_t ra);
uint8_t cpu_ldb_mmu(CPUArchState *env, abi_ptr ptr, MemOpIdx oi, uintptr_t ra);
-uint16_t cpu_ldw_be_mmu(CPUArchState *env, abi_ptr ptr,
- MemOpIdx oi, uintptr_t ra);
-uint32_t cpu_ldl_be_mmu(CPUArchState *env, abi_ptr ptr,
- MemOpIdx oi, uintptr_t ra);
-uint64_t cpu_ldq_be_mmu(CPUArchState *env, abi_ptr ptr,
- MemOpIdx oi, uintptr_t ra);
-uint16_t cpu_ldw_le_mmu(CPUArchState *env, abi_ptr ptr,
- MemOpIdx oi, uintptr_t ra);
-uint32_t cpu_ldl_le_mmu(CPUArchState *env, abi_ptr ptr,
- MemOpIdx oi, uintptr_t ra);
-uint64_t cpu_ldq_le_mmu(CPUArchState *env, abi_ptr ptr,
- MemOpIdx oi, uintptr_t ra);
-
-Int128 cpu_ld16_be_mmu(CPUArchState *env, abi_ptr addr,
- MemOpIdx oi, uintptr_t ra);
-Int128 cpu_ld16_le_mmu(CPUArchState *env, abi_ptr addr,
- MemOpIdx oi, uintptr_t ra);
+uint16_t cpu_ldw_mmu(CPUArchState *env, abi_ptr ptr, MemOpIdx oi, uintptr_t ra);
+uint32_t cpu_ldl_mmu(CPUArchState *env, abi_ptr ptr, MemOpIdx oi, uintptr_t ra);
+uint64_t cpu_ldq_mmu(CPUArchState *env, abi_ptr ptr, MemOpIdx oi, uintptr_t ra);
+Int128 cpu_ld16_mmu(CPUArchState *env, abi_ptr addr, MemOpIdx oi, uintptr_t ra);
void cpu_stb_mmu(CPUArchState *env, abi_ptr ptr, uint8_t val,
MemOpIdx oi, uintptr_t ra);
-void cpu_stw_be_mmu(CPUArchState *env, abi_ptr ptr, uint16_t val,
- MemOpIdx oi, uintptr_t ra);
-void cpu_stl_be_mmu(CPUArchState *env, abi_ptr ptr, uint32_t val,
- MemOpIdx oi, uintptr_t ra);
-void cpu_stq_be_mmu(CPUArchState *env, abi_ptr ptr, uint64_t val,
- MemOpIdx oi, uintptr_t ra);
-void cpu_stw_le_mmu(CPUArchState *env, abi_ptr ptr, uint16_t val,
- MemOpIdx oi, uintptr_t ra);
-void cpu_stl_le_mmu(CPUArchState *env, abi_ptr ptr, uint32_t val,
- MemOpIdx oi, uintptr_t ra);
-void cpu_stq_le_mmu(CPUArchState *env, abi_ptr ptr, uint64_t val,
- MemOpIdx oi, uintptr_t ra);
-
-void cpu_st16_be_mmu(CPUArchState *env, abi_ptr addr, Int128 val,
- MemOpIdx oi, uintptr_t ra);
-void cpu_st16_le_mmu(CPUArchState *env, abi_ptr addr, Int128 val,
- MemOpIdx oi, uintptr_t ra);
+void cpu_stw_mmu(CPUArchState *env, abi_ptr ptr, uint16_t val,
+ MemOpIdx oi, uintptr_t ra);
+void cpu_stl_mmu(CPUArchState *env, abi_ptr ptr, uint32_t val,
+ MemOpIdx oi, uintptr_t ra);
+void cpu_stq_mmu(CPUArchState *env, abi_ptr ptr, uint64_t val,
+ MemOpIdx oi, uintptr_t ra);
+void cpu_st16_mmu(CPUArchState *env, abi_ptr addr, Int128 val,
+ MemOpIdx oi, uintptr_t ra);
uint32_t cpu_atomic_cmpxchgb_mmu(CPUArchState *env, target_ulong addr,
uint32_t cmpv, uint32_t newv,
@@ -416,9 +394,6 @@ static inline CPUTLBEntry *tlb_entry(CPUArchState *env, uintptr_t mmu_idx,
# define cpu_ldsw_mmuidx_ra cpu_ldsw_be_mmuidx_ra
# define cpu_ldl_mmuidx_ra cpu_ldl_be_mmuidx_ra
# define cpu_ldq_mmuidx_ra cpu_ldq_be_mmuidx_ra
-# define cpu_ldw_mmu cpu_ldw_be_mmu
-# define cpu_ldl_mmu cpu_ldl_be_mmu
-# define cpu_ldq_mmu cpu_ldq_be_mmu
# define cpu_stw_data cpu_stw_be_data
# define cpu_stl_data cpu_stl_be_data
# define cpu_stq_data cpu_stq_be_data
@@ -428,9 +403,6 @@ static inline CPUTLBEntry *tlb_entry(CPUArchState *env, uintptr_t mmu_idx,
# define cpu_stw_mmuidx_ra cpu_stw_be_mmuidx_ra
# define cpu_stl_mmuidx_ra cpu_stl_be_mmuidx_ra
# define cpu_stq_mmuidx_ra cpu_stq_be_mmuidx_ra
-# define cpu_stw_mmu cpu_stw_be_mmu
-# define cpu_stl_mmu cpu_stl_be_mmu
-# define cpu_stq_mmu cpu_stq_be_mmu
#else
# define cpu_lduw_data cpu_lduw_le_data
# define cpu_ldsw_data cpu_ldsw_le_data
@@ -444,9 +416,6 @@ static inline CPUTLBEntry *tlb_entry(CPUArchState *env, uintptr_t mmu_idx,
# define cpu_ldsw_mmuidx_ra cpu_ldsw_le_mmuidx_ra
# define cpu_ldl_mmuidx_ra cpu_ldl_le_mmuidx_ra
# define cpu_ldq_mmuidx_ra cpu_ldq_le_mmuidx_ra
-# define cpu_ldw_mmu cpu_ldw_le_mmu
-# define cpu_ldl_mmu cpu_ldl_le_mmu
-# define cpu_ldq_mmu cpu_ldq_le_mmu
# define cpu_stw_data cpu_stw_le_data
# define cpu_stl_data cpu_stl_le_data
# define cpu_stq_data cpu_stq_le_data
@@ -456,9 +425,6 @@ static inline CPUTLBEntry *tlb_entry(CPUArchState *env, uintptr_t mmu_idx,
# define cpu_stw_mmuidx_ra cpu_stw_le_mmuidx_ra
# define cpu_stl_mmuidx_ra cpu_stl_le_mmuidx_ra
# define cpu_stq_mmuidx_ra cpu_stq_le_mmuidx_ra
-# define cpu_stw_mmu cpu_stw_le_mmu
-# define cpu_stl_mmu cpu_stl_le_mmu
-# define cpu_stq_mmu cpu_stq_le_mmu
#endif
uint8_t cpu_ldb_code_mmu(CPUArchState *env, abi_ptr addr,
diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index ae0fbcdee2..b1e13d165c 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -2575,89 +2575,45 @@ uint8_t cpu_ldb_mmu(CPUArchState *env, abi_ptr addr, MemOpIdx oi, uintptr_t ra)
return ret;
}
-uint16_t cpu_ldw_be_mmu(CPUArchState *env, abi_ptr addr,
- MemOpIdx oi, uintptr_t ra)
+uint16_t cpu_ldw_mmu(CPUArchState *env, abi_ptr addr,
+ MemOpIdx oi, uintptr_t ra)
{
uint16_t ret;
- tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_BEUW);
+ tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_16);
ret = do_ld2_mmu(env, addr, oi, ra, MMU_DATA_LOAD);
plugin_load_cb(env, addr, oi);
return ret;
}
-uint32_t cpu_ldl_be_mmu(CPUArchState *env, abi_ptr addr,
- MemOpIdx oi, uintptr_t ra)
+uint32_t cpu_ldl_mmu(CPUArchState *env, abi_ptr addr,
+ MemOpIdx oi, uintptr_t ra)
{
uint32_t ret;
- tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_BEUL);
+ tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_32);
ret = do_ld4_mmu(env, addr, oi, ra, MMU_DATA_LOAD);
plugin_load_cb(env, addr, oi);
return ret;
}
-uint64_t cpu_ldq_be_mmu(CPUArchState *env, abi_ptr addr,
- MemOpIdx oi, uintptr_t ra)
+uint64_t cpu_ldq_mmu(CPUArchState *env, abi_ptr addr,
+ MemOpIdx oi, uintptr_t ra)
{
uint64_t ret;
- tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_BEUQ);
+ tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_64);
ret = do_ld8_mmu(env, addr, oi, ra, MMU_DATA_LOAD);
plugin_load_cb(env, addr, oi);
return ret;
}
-uint16_t cpu_ldw_le_mmu(CPUArchState *env, abi_ptr addr,
- MemOpIdx oi, uintptr_t ra)
-{
- uint16_t ret;
-
- tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_LEUW);
- ret = do_ld2_mmu(env, addr, oi, ra, MMU_DATA_LOAD);
- plugin_load_cb(env, addr, oi);
- return ret;
-}
-
-uint32_t cpu_ldl_le_mmu(CPUArchState *env, abi_ptr addr,
- MemOpIdx oi, uintptr_t ra)
-{
- uint32_t ret;
-
- tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_LEUL);
- ret = do_ld4_mmu(env, addr, oi, ra, MMU_DATA_LOAD);
- plugin_load_cb(env, addr, oi);
- return ret;
-}
-
-uint64_t cpu_ldq_le_mmu(CPUArchState *env, abi_ptr addr,
- MemOpIdx oi, uintptr_t ra)
-{
- uint64_t ret;
-
- tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_LEUQ);
- ret = do_ld8_mmu(env, addr, oi, ra, MMU_DATA_LOAD);
- plugin_load_cb(env, addr, oi);
- return ret;
-}
-
-Int128 cpu_ld16_be_mmu(CPUArchState *env, abi_ptr addr,
- MemOpIdx oi, uintptr_t ra)
+Int128 cpu_ld16_mmu(CPUArchState *env, abi_ptr addr,
+ MemOpIdx oi, uintptr_t ra)
{
Int128 ret;
- tcg_debug_assert((get_memop(oi) & (MO_BSWAP|MO_SIZE)) == (MO_BE|MO_128));
- ret = do_ld16_mmu(env, addr, oi, ra);
- plugin_load_cb(env, addr, oi);
- return ret;
-}
-
-Int128 cpu_ld16_le_mmu(CPUArchState *env, abi_ptr addr,
- MemOpIdx oi, uintptr_t ra)
-{
- Int128 ret;
-
- tcg_debug_assert((get_memop(oi) & (MO_BSWAP|MO_SIZE)) == (MO_LE|MO_128));
+ tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_128);
ret = do_ld16_mmu(env, addr, oi, ra);
plugin_load_cb(env, addr, oi);
return ret;
@@ -3045,66 +3001,34 @@ void cpu_stb_mmu(CPUArchState *env, target_ulong addr, uint8_t val,
plugin_store_cb(env, addr, oi);
}
-void cpu_stw_be_mmu(CPUArchState *env, target_ulong addr, uint16_t val,
- MemOpIdx oi, uintptr_t retaddr)
+void cpu_stw_mmu(CPUArchState *env, target_ulong addr, uint16_t val,
+ MemOpIdx oi, uintptr_t retaddr)
{
- tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_BEUW);
+ tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_16);
do_st2_mmu(env, addr, val, oi, retaddr);
plugin_store_cb(env, addr, oi);
}
-void cpu_stl_be_mmu(CPUArchState *env, target_ulong addr, uint32_t val,
+void cpu_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val,
MemOpIdx oi, uintptr_t retaddr)
{
- tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_BEUL);
+ tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_32);
do_st4_mmu(env, addr, val, oi, retaddr);
plugin_store_cb(env, addr, oi);
}
-void cpu_stq_be_mmu(CPUArchState *env, target_ulong addr, uint64_t val,
- MemOpIdx oi, uintptr_t retaddr)
+void cpu_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val,
+ MemOpIdx oi, uintptr_t retaddr)
{
- tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_BEUQ);
+ tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_64);
do_st8_mmu(env, addr, val, oi, retaddr);
plugin_store_cb(env, addr, oi);
}
-void cpu_stw_le_mmu(CPUArchState *env, target_ulong addr, uint16_t val,
- MemOpIdx oi, uintptr_t retaddr)
+void cpu_st16_mmu(CPUArchState *env, target_ulong addr, Int128 val,
+ MemOpIdx oi, uintptr_t retaddr)
{
- tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_LEUW);
- do_st2_mmu(env, addr, val, oi, retaddr);
- plugin_store_cb(env, addr, oi);
-}
-
-void cpu_stl_le_mmu(CPUArchState *env, target_ulong addr, uint32_t val,
- MemOpIdx oi, uintptr_t retaddr)
-{
- tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_LEUL);
- do_st4_mmu(env, addr, val, oi, retaddr);
- plugin_store_cb(env, addr, oi);
-}
-
-void cpu_stq_le_mmu(CPUArchState *env, target_ulong addr, uint64_t val,
- MemOpIdx oi, uintptr_t retaddr)
-{
- tcg_debug_assert((get_memop(oi) & (MO_BSWAP | MO_SIZE)) == MO_LEUQ);
- do_st8_mmu(env, addr, val, oi, retaddr);
- plugin_store_cb(env, addr, oi);
-}
-
-void cpu_st16_be_mmu(CPUArchState *env, target_ulong addr, Int128 val,
- MemOpIdx oi, uintptr_t retaddr)
-{
- tcg_debug_assert((get_memop(oi) & (MO_BSWAP|MO_SIZE)) == (MO_BE|MO_128));
- do_st16_mmu(env, addr, val, oi, retaddr);
- plugin_store_cb(env, addr, oi);
-}
-
-void cpu_st16_le_mmu(CPUArchState *env, target_ulong addr, Int128 val,
- MemOpIdx oi, uintptr_t retaddr)
-{
- tcg_debug_assert((get_memop(oi) & (MO_BSWAP|MO_SIZE)) == (MO_LE|MO_128));
+ tcg_debug_assert((get_memop(oi) & MO_SIZE) == MO_128);
do_st16_mmu(env, addr, val, oi, retaddr);
plugin_store_cb(env, addr, oi);
}
diff --git a/accel/tcg/user-exec.c b/accel/tcg/user-exec.c
index 36ad8284a5..19c2849c21 100644
--- a/accel/tcg/user-exec.c
+++ b/accel/tcg/user-exec.c
@@ -940,8 +940,8 @@ uint8_t cpu_ldb_mmu(CPUArchState *env, abi_ptr addr,
return ret;
}
-static uint16_t do_ld2_he_mmu(CPUArchState *env, abi_ptr addr,
- MemOp mop, uintptr_t ra)
+static uint16_t do_ld2_mmu(CPUArchState *env, abi_ptr addr,
+ MemOp mop, uintptr_t ra)
{
void *haddr;
uint16_t ret;
@@ -950,59 +950,35 @@ static uint16_t do_ld2_he_mmu(CPUArchState *env, abi_ptr addr,
haddr = cpu_mmu_lookup(env, addr, mop, ra, MMU_DATA_LOAD);
ret = load_atom_2(env, ra, haddr, mop);
clear_helper_retaddr();
+
+ if (mop & MO_BSWAP) {
+ ret = bswap16(ret);
+ }
return ret;
}
tcg_target_ulong helper_lduw_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t ra)
{
- MemOp mop = get_memop(oi);
- uint16_t ret = do_ld2_he_mmu(env, addr, mop, ra);
-
- if (mop & MO_BSWAP) {
- ret = bswap16(ret);
- }
- return ret;
+ return do_ld2_mmu(env, addr, get_memop(oi), ra);
}
tcg_target_ulong helper_ldsw_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t ra)
{
- MemOp mop = get_memop(oi);
- int16_t ret = do_ld2_he_mmu(env, addr, mop, ra);
+ return (int16_t)do_ld2_mmu(env, addr, get_memop(oi), ra);
+}
- if (mop & MO_BSWAP) {
- ret = bswap16(ret);
- }
+uint16_t cpu_ldw_mmu(CPUArchState *env, abi_ptr addr,
+ MemOpIdx oi, uintptr_t ra)
+{
+ uint16_t ret = do_ld2_mmu(env, addr, get_memop(oi), ra);
+ qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R);
return ret;
}
-uint16_t cpu_ldw_be_mmu(CPUArchState *env, abi_ptr addr,
- MemOpIdx oi, uintptr_t ra)
-{
- MemOp mop = get_memop(oi);
- uint16_t ret;
-
- tcg_debug_assert((mop & MO_BSWAP) == MO_BE);
- ret = do_ld2_he_mmu(env, addr, mop, ra);
- qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R);
- return cpu_to_be16(ret);
-}
-
-uint16_t cpu_ldw_le_mmu(CPUArchState *env, abi_ptr addr,
- MemOpIdx oi, uintptr_t ra)
-{
- MemOp mop = get_memop(oi);
- uint16_t ret;
-
- tcg_debug_assert((mop & MO_BSWAP) == MO_LE);
- ret = do_ld2_he_mmu(env, addr, mop, ra);
- qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R);
- return cpu_to_le16(ret);
-}
-
-static uint32_t do_ld4_he_mmu(CPUArchState *env, abi_ptr addr,
- MemOp mop, uintptr_t ra)
+static uint32_t do_ld4_mmu(CPUArchState *env, abi_ptr addr,
+ MemOp mop, uintptr_t ra)
{
void *haddr;
uint32_t ret;
@@ -1011,59 +987,35 @@ static uint32_t do_ld4_he_mmu(CPUArchState *env, abi_ptr addr,
haddr = cpu_mmu_lookup(env, addr, mop, ra, MMU_DATA_LOAD);
ret = load_atom_4(env, ra, haddr, mop);
clear_helper_retaddr();
+
+ if (mop & MO_BSWAP) {
+ ret = bswap32(ret);
+ }
return ret;
}
tcg_target_ulong helper_ldul_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t ra)
{
- MemOp mop = get_memop(oi);
- uint32_t ret = do_ld4_he_mmu(env, addr, mop, ra);
-
- if (mop & MO_BSWAP) {
- ret = bswap32(ret);
- }
- return ret;
+ return do_ld4_mmu(env, addr, get_memop(oi), ra);
}
tcg_target_ulong helper_ldsl_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t ra)
{
- MemOp mop = get_memop(oi);
- int32_t ret = do_ld4_he_mmu(env, addr, mop, ra);
+ return (int32_t)do_ld4_mmu(env, addr, get_memop(oi), ra);
+}
- if (mop & MO_BSWAP) {
- ret = bswap32(ret);
- }
+uint32_t cpu_ldl_mmu(CPUArchState *env, abi_ptr addr,
+ MemOpIdx oi, uintptr_t ra)
+{
+ uint32_t ret = do_ld4_mmu(env, addr, get_memop(oi), ra);
+ qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R);
return ret;
}
-uint32_t cpu_ldl_be_mmu(CPUArchState *env, abi_ptr addr,
- MemOpIdx oi, uintptr_t ra)
-{
- MemOp mop = get_memop(oi);
- uint32_t ret;
-
- tcg_debug_assert((mop & MO_BSWAP) == MO_BE);
- ret = do_ld4_he_mmu(env, addr, mop, ra);
- qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R);
- return cpu_to_be32(ret);
-}
-
-uint32_t cpu_ldl_le_mmu(CPUArchState *env, abi_ptr addr,
- MemOpIdx oi, uintptr_t ra)
-{
- MemOp mop = get_memop(oi);
- uint32_t ret;
-
- tcg_debug_assert((mop & MO_BSWAP) == MO_LE);
- ret = do_ld4_he_mmu(env, addr, mop, ra);
- qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R);
- return cpu_to_le32(ret);
-}
-
-static uint64_t do_ld8_he_mmu(CPUArchState *env, abi_ptr addr,
- MemOp mop, uintptr_t ra)
+static uint64_t do_ld8_mmu(CPUArchState *env, abi_ptr addr,
+ MemOp mop, uintptr_t ra)
{
void *haddr;
uint64_t ret;
@@ -1072,14 +1024,6 @@ static uint64_t do_ld8_he_mmu(CPUArchState *env, abi_ptr addr,
haddr = cpu_mmu_lookup(env, addr, mop, ra, MMU_DATA_LOAD);
ret = load_atom_8(env, ra, haddr, mop);
clear_helper_retaddr();
- return ret;
-}
-
-uint64_t helper_ldq_mmu(CPUArchState *env, uint64_t addr,
- MemOpIdx oi, uintptr_t ra)
-{
- MemOp mop = get_memop(oi);
- uint64_t ret = do_ld8_he_mmu(env, addr, mop, ra);
if (mop & MO_BSWAP) {
ret = bswap64(ret);
@@ -1087,32 +1031,22 @@ uint64_t helper_ldq_mmu(CPUArchState *env, uint64_t addr,
return ret;
}
-uint64_t cpu_ldq_be_mmu(CPUArchState *env, abi_ptr addr,
+uint64_t helper_ldq_mmu(CPUArchState *env, uint64_t addr,
MemOpIdx oi, uintptr_t ra)
{
- MemOp mop = get_memop(oi);
- uint64_t ret;
-
- tcg_debug_assert((mop & MO_BSWAP) == MO_BE);
- ret = do_ld8_he_mmu(env, addr, mop, ra);
- qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R);
- return cpu_to_be64(ret);
+ return do_ld8_mmu(env, addr, get_memop(oi), ra);
}
-uint64_t cpu_ldq_le_mmu(CPUArchState *env, abi_ptr addr,
- MemOpIdx oi, uintptr_t ra)
+uint64_t cpu_ldq_mmu(CPUArchState *env, abi_ptr addr,
+ MemOpIdx oi, uintptr_t ra)
{
- MemOp mop = get_memop(oi);
- uint64_t ret;
-
- tcg_debug_assert((mop & MO_BSWAP) == MO_LE);
- ret = do_ld8_he_mmu(env, addr, mop, ra);
+ uint64_t ret = do_ld8_mmu(env, addr, get_memop(oi), ra);
qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R);
- return cpu_to_le64(ret);
+ return ret;
}
-static Int128 do_ld16_he_mmu(CPUArchState *env, abi_ptr addr,
- MemOp mop, uintptr_t ra)
+static Int128 do_ld16_mmu(CPUArchState *env, abi_ptr addr,
+ MemOp mop, uintptr_t ra)
{
void *haddr;
Int128 ret;
@@ -1121,14 +1055,6 @@ static Int128 do_ld16_he_mmu(CPUArchState *env, abi_ptr addr,
haddr = cpu_mmu_lookup(env, addr, mop, ra, MMU_DATA_LOAD);
ret = load_atom_16(env, ra, haddr, mop);
clear_helper_retaddr();
- return ret;
-}
-
-Int128 helper_ld16_mmu(CPUArchState *env, uint64_t addr,
- MemOpIdx oi, uintptr_t ra)
-{
- MemOp mop = get_memop(oi);
- Int128 ret = do_ld16_he_mmu(env, addr, mop, ra);
if (mop & MO_BSWAP) {
ret = bswap128(ret);
@@ -1136,38 +1062,22 @@ Int128 helper_ld16_mmu(CPUArchState *env, uint64_t addr,
return ret;
}
+Int128 helper_ld16_mmu(CPUArchState *env, uint64_t addr,
+ MemOpIdx oi, uintptr_t ra)
+{
+ return do_ld16_mmu(env, addr, get_memop(oi), ra);
+}
+
Int128 helper_ld_i128(CPUArchState *env, uint64_t addr, MemOpIdx oi)
{
return helper_ld16_mmu(env, addr, oi, GETPC());
}
-Int128 cpu_ld16_be_mmu(CPUArchState *env, abi_ptr addr,
- MemOpIdx oi, uintptr_t ra)
+Int128 cpu_ld16_mmu(CPUArchState *env, abi_ptr addr,
+ MemOpIdx oi, uintptr_t ra)
{
- MemOp mop = get_memop(oi);
- Int128 ret;
-
- tcg_debug_assert((mop & MO_BSWAP) == MO_BE);
- ret = do_ld16_he_mmu(env, addr, mop, ra);
+ Int128 ret = do_ld16_mmu(env, addr, get_memop(oi), ra);
qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R);
- if (!HOST_BIG_ENDIAN) {
- ret = bswap128(ret);
- }
- return ret;
-}
-
-Int128 cpu_ld16_le_mmu(CPUArchState *env, abi_ptr addr,
- MemOpIdx oi, uintptr_t ra)
-{
- MemOp mop = get_memop(oi);
- Int128 ret;
-
- tcg_debug_assert((mop & MO_BSWAP) == MO_LE);
- ret = do_ld16_he_mmu(env, addr, mop, ra);
- qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R);
- if (HOST_BIG_ENDIAN) {
- ret = bswap128(ret);
- }
return ret;
}
@@ -1195,13 +1105,17 @@ void cpu_stb_mmu(CPUArchState *env, abi_ptr addr, uint8_t val,
qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W);
}
-static void do_st2_he_mmu(CPUArchState *env, abi_ptr addr, uint16_t val,
- MemOp mop, uintptr_t ra)
+static void do_st2_mmu(CPUArchState *env, abi_ptr addr, uint16_t val,
+ MemOp mop, uintptr_t ra)
{
void *haddr;
tcg_debug_assert((mop & MO_SIZE) == MO_16);
haddr = cpu_mmu_lookup(env, addr, mop, ra, MMU_DATA_STORE);
+
+ if (mop & MO_BSWAP) {
+ val = bswap16(val);
+ }
store_atom_2(env, ra, haddr, mop, val);
clear_helper_retaddr();
}
@@ -1209,41 +1123,27 @@ static void do_st2_he_mmu(CPUArchState *env, abi_ptr addr, uint16_t val,
void helper_stw_mmu(CPUArchState *env, uint64_t addr, uint32_t val,
MemOpIdx oi, uintptr_t ra)
{
- MemOp mop = get_memop(oi);
-
- if (mop & MO_BSWAP) {
- val = bswap16(val);
- }
- do_st2_he_mmu(env, addr, val, mop, ra);
+ do_st2_mmu(env, addr, val, get_memop(oi), ra);
}
-void cpu_stw_be_mmu(CPUArchState *env, abi_ptr addr, uint16_t val,
+void cpu_stw_mmu(CPUArchState *env, abi_ptr addr, uint16_t val,
MemOpIdx oi, uintptr_t ra)
{
- MemOp mop = get_memop(oi);
-
- tcg_debug_assert((mop & MO_BSWAP) == MO_BE);
- do_st2_he_mmu(env, addr, be16_to_cpu(val), mop, ra);
+ do_st2_mmu(env, addr, val, get_memop(oi), ra);
qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W);
}
-void cpu_stw_le_mmu(CPUArchState *env, abi_ptr addr, uint16_t val,
- MemOpIdx oi, uintptr_t ra)
-{
- MemOp mop = get_memop(oi);
-
- tcg_debug_assert((mop & MO_BSWAP) == MO_LE);
- do_st2_he_mmu(env, addr, le16_to_cpu(val), mop, ra);
- qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W);
-}
-
-static void do_st4_he_mmu(CPUArchState *env, abi_ptr addr, uint32_t val,
- MemOp mop, uintptr_t ra)
+static void do_st4_mmu(CPUArchState *env, abi_ptr addr, uint32_t val,
+ MemOp mop, uintptr_t ra)
{
void *haddr;
tcg_debug_assert((mop & MO_SIZE) == MO_32);
haddr = cpu_mmu_lookup(env, addr, mop, ra, MMU_DATA_STORE);
+
+ if (mop & MO_BSWAP) {
+ val = bswap32(val);
+ }
store_atom_4(env, ra, haddr, mop, val);
clear_helper_retaddr();
}
@@ -1251,41 +1151,27 @@ static void do_st4_he_mmu(CPUArchState *env, abi_ptr addr, uint32_t val,
void helper_stl_mmu(CPUArchState *env, uint64_t addr, uint32_t val,
MemOpIdx oi, uintptr_t ra)
{
- MemOp mop = get_memop(oi);
-
- if (mop & MO_BSWAP) {
- val = bswap32(val);
- }
- do_st4_he_mmu(env, addr, val, mop, ra);
+ do_st4_mmu(env, addr, val, get_memop(oi), ra);
}
-void cpu_stl_be_mmu(CPUArchState *env, abi_ptr addr, uint32_t val,
- MemOpIdx oi, uintptr_t ra)
+void cpu_stl_mmu(CPUArchState *env, abi_ptr addr, uint32_t val,
+ MemOpIdx oi, uintptr_t ra)
{
- MemOp mop = get_memop(oi);
-
- tcg_debug_assert((mop & MO_BSWAP) == MO_BE);
- do_st4_he_mmu(env, addr, be32_to_cpu(val), mop, ra);
+ do_st4_mmu(env, addr, val, get_memop(oi), ra);
qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W);
}
-void cpu_stl_le_mmu(CPUArchState *env, abi_ptr addr, uint32_t val,
- MemOpIdx oi, uintptr_t ra)
-{
- MemOp mop = get_memop(oi);
-
- tcg_debug_assert((mop & MO_BSWAP) == MO_LE);
- do_st4_he_mmu(env, addr, le32_to_cpu(val), mop, ra);
- qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W);
-}
-
-static void do_st8_he_mmu(CPUArchState *env, abi_ptr addr, uint64_t val,
- MemOp mop, uintptr_t ra)
+static void do_st8_mmu(CPUArchState *env, abi_ptr addr, uint64_t val,
+ MemOp mop, uintptr_t ra)
{
void *haddr;
tcg_debug_assert((mop & MO_SIZE) == MO_64);
haddr = cpu_mmu_lookup(env, addr, mop, ra, MMU_DATA_STORE);
+
+ if (mop & MO_BSWAP) {
+ val = bswap64(val);
+ }
store_atom_8(env, ra, haddr, mop, val);
clear_helper_retaddr();
}
@@ -1293,41 +1179,27 @@ static void do_st8_he_mmu(CPUArchState *env, abi_ptr addr, uint64_t val,
void helper_stq_mmu(CPUArchState *env, uint64_t addr, uint64_t val,
MemOpIdx oi, uintptr_t ra)
{
- MemOp mop = get_memop(oi);
-
- if (mop & MO_BSWAP) {
- val = bswap64(val);
- }
- do_st8_he_mmu(env, addr, val, mop, ra);
+ do_st8_mmu(env, addr, val, get_memop(oi), ra);
}
-void cpu_stq_be_mmu(CPUArchState *env, abi_ptr addr, uint64_t val,
+void cpu_stq_mmu(CPUArchState *env, abi_ptr addr, uint64_t val,
MemOpIdx oi, uintptr_t ra)
{
- MemOp mop = get_memop(oi);
-
- tcg_debug_assert((mop & MO_BSWAP) == MO_BE);
- do_st8_he_mmu(env, addr, cpu_to_be64(val), mop, ra);
+ do_st8_mmu(env, addr, val, get_memop(oi), ra);
qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W);
}
-void cpu_stq_le_mmu(CPUArchState *env, abi_ptr addr, uint64_t val,
- MemOpIdx oi, uintptr_t ra)
-{
- MemOp mop = get_memop(oi);
-
- tcg_debug_assert((mop & MO_BSWAP) == MO_LE);
- do_st8_he_mmu(env, addr, cpu_to_le64(val), mop, ra);
- qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W);
-}
-
-static void do_st16_he_mmu(CPUArchState *env, abi_ptr addr, Int128 val,
- MemOp mop, uintptr_t ra)
+static void do_st16_mmu(CPUArchState *env, abi_ptr addr, Int128 val,
+ MemOp mop, uintptr_t ra)
{
void *haddr;
tcg_debug_assert((mop & MO_SIZE) == MO_128);
haddr = cpu_mmu_lookup(env, addr, mop, ra, MMU_DATA_STORE);
+
+ if (mop & MO_BSWAP) {
+ val = bswap128(val);
+ }
store_atom_16(env, ra, haddr, mop, val);
clear_helper_retaddr();
}
@@ -1335,12 +1207,7 @@ static void do_st16_he_mmu(CPUArchState *env, abi_ptr addr, Int128 val,
void helper_st16_mmu(CPUArchState *env, uint64_t addr, Int128 val,
MemOpIdx oi, uintptr_t ra)
{
- MemOp mop = get_memop(oi);
-
- if (mop & MO_BSWAP) {
- val = bswap128(val);
- }
- do_st16_he_mmu(env, addr, val, mop, ra);
+ do_st16_mmu(env, addr, val, get_memop(oi), ra);
}
void helper_st_i128(CPUArchState *env, uint64_t addr, Int128 val, MemOpIdx oi)
@@ -1348,29 +1215,10 @@ void helper_st_i128(CPUArchState *env, uint64_t addr, Int128 val, MemOpIdx oi)
helper_st16_mmu(env, addr, val, oi, GETPC());
}
-void cpu_st16_be_mmu(CPUArchState *env, abi_ptr addr,
- Int128 val, MemOpIdx oi, uintptr_t ra)
+void cpu_st16_mmu(CPUArchState *env, abi_ptr addr,
+ Int128 val, MemOpIdx oi, uintptr_t ra)
{
- MemOp mop = get_memop(oi);
-
- tcg_debug_assert((mop & MO_BSWAP) == MO_BE);
- if (!HOST_BIG_ENDIAN) {
- val = bswap128(val);
- }
- do_st16_he_mmu(env, addr, val, mop, ra);
- qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W);
-}
-
-void cpu_st16_le_mmu(CPUArchState *env, abi_ptr addr,
- Int128 val, MemOpIdx oi, uintptr_t ra)
-{
- MemOp mop = get_memop(oi);
-
- tcg_debug_assert((mop & MO_BSWAP) == MO_LE);
- if (HOST_BIG_ENDIAN) {
- val = bswap128(val);
- }
- do_st16_he_mmu(env, addr, val, mop, ra);
+ do_st16_mmu(env, addr, val, get_memop(oi), ra);
qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W);
}
diff --git a/target/arm/tcg/m_helper.c b/target/arm/tcg/m_helper.c
index 9758f225d6..9cef70e5c9 100644
--- a/target/arm/tcg/m_helper.c
+++ b/target/arm/tcg/m_helper.c
@@ -1937,8 +1937,8 @@ static bool do_v7m_function_return(ARMCPU *cpu)
*/
mmu_idx = arm_v7m_mmu_idx_for_secstate(env, true);
oi = make_memop_idx(MO_LEUL, arm_to_core_mmu_idx(mmu_idx));
- newpc = cpu_ldl_le_mmu(env, frameptr, oi, 0);
- newpsr = cpu_ldl_le_mmu(env, frameptr + 4, oi, 0);
+ newpc = cpu_ldl_mmu(env, frameptr, oi, 0);
+ newpsr = cpu_ldl_mmu(env, frameptr + 4, oi, 0);
/* Consistency checks on new IPSR */
newpsr_exc = newpsr & XPSR_EXCP;
diff --git a/target/sparc/ldst_helper.c b/target/sparc/ldst_helper.c
index 7972d56a72..981a47d8bb 100644
--- a/target/sparc/ldst_helper.c
+++ b/target/sparc/ldst_helper.c
@@ -1334,25 +1334,13 @@ uint64_t helper_ld_asi(CPUSPARCState *env, target_ulong addr,
ret = cpu_ldb_mmu(env, addr, oi, GETPC());
break;
case 2:
- if (asi & 8) {
- ret = cpu_ldw_le_mmu(env, addr, oi, GETPC());
- } else {
- ret = cpu_ldw_be_mmu(env, addr, oi, GETPC());
- }
+ ret = cpu_ldw_mmu(env, addr, oi, GETPC());
break;
case 4:
- if (asi & 8) {
- ret = cpu_ldl_le_mmu(env, addr, oi, GETPC());
- } else {
- ret = cpu_ldl_be_mmu(env, addr, oi, GETPC());
- }
+ ret = cpu_ldl_mmu(env, addr, oi, GETPC());
break;
case 8:
- if (asi & 8) {
- ret = cpu_ldq_le_mmu(env, addr, oi, GETPC());
- } else {
- ret = cpu_ldq_be_mmu(env, addr, oi, GETPC());
- }
+ ret = cpu_ldq_mmu(env, addr, oi, GETPC());
break;
default:
g_assert_not_reached();
diff --git a/accel/tcg/ldst_common.c.inc b/accel/tcg/ldst_common.c.inc
index 6ac8d871a3..5f8144b33a 100644
--- a/accel/tcg/ldst_common.c.inc
+++ b/accel/tcg/ldst_common.c.inc
@@ -26,7 +26,7 @@ uint32_t cpu_lduw_be_mmuidx_ra(CPUArchState *env, abi_ptr addr,
int mmu_idx, uintptr_t ra)
{
MemOpIdx oi = make_memop_idx(MO_BEUW | MO_UNALN, mmu_idx);
- return cpu_ldw_be_mmu(env, addr, oi, ra);
+ return cpu_ldw_mmu(env, addr, oi, ra);
}
int cpu_ldsw_be_mmuidx_ra(CPUArchState *env, abi_ptr addr,
@@ -39,21 +39,21 @@ uint32_t cpu_ldl_be_mmuidx_ra(CPUArchState *env, abi_ptr addr,
int mmu_idx, uintptr_t ra)
{
MemOpIdx oi = make_memop_idx(MO_BEUL | MO_UNALN, mmu_idx);
- return cpu_ldl_be_mmu(env, addr, oi, ra);
+ return cpu_ldl_mmu(env, addr, oi, ra);
}
uint64_t cpu_ldq_be_mmuidx_ra(CPUArchState *env, abi_ptr addr,
int mmu_idx, uintptr_t ra)
{
MemOpIdx oi = make_memop_idx(MO_BEUQ | MO_UNALN, mmu_idx);
- return cpu_ldq_be_mmu(env, addr, oi, ra);
+ return cpu_ldq_mmu(env, addr, oi, ra);
}
uint32_t cpu_lduw_le_mmuidx_ra(CPUArchState *env, abi_ptr addr,
int mmu_idx, uintptr_t ra)
{
MemOpIdx oi = make_memop_idx(MO_LEUW | MO_UNALN, mmu_idx);
- return cpu_ldw_le_mmu(env, addr, oi, ra);
+ return cpu_ldw_mmu(env, addr, oi, ra);
}
int cpu_ldsw_le_mmuidx_ra(CPUArchState *env, abi_ptr addr,
@@ -66,14 +66,14 @@ uint32_t cpu_ldl_le_mmuidx_ra(CPUArchState *env, abi_ptr addr,
int mmu_idx, uintptr_t ra)
{
MemOpIdx oi = make_memop_idx(MO_LEUL | MO_UNALN, mmu_idx);
- return cpu_ldl_le_mmu(env, addr, oi, ra);
+ return cpu_ldl_mmu(env, addr, oi, ra);
}
uint64_t cpu_ldq_le_mmuidx_ra(CPUArchState *env, abi_ptr addr,
int mmu_idx, uintptr_t ra)
{
MemOpIdx oi = make_memop_idx(MO_LEUQ | MO_UNALN, mmu_idx);
- return cpu_ldq_le_mmu(env, addr, oi, ra);
+ return cpu_ldq_mmu(env, addr, oi, ra);
}
void cpu_stb_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint32_t val,
@@ -87,42 +87,42 @@ void cpu_stw_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint32_t val,
int mmu_idx, uintptr_t ra)
{
MemOpIdx oi = make_memop_idx(MO_BEUW | MO_UNALN, mmu_idx);
- cpu_stw_be_mmu(env, addr, val, oi, ra);
+ cpu_stw_mmu(env, addr, val, oi, ra);
}
void cpu_stl_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint32_t val,
int mmu_idx, uintptr_t ra)
{
MemOpIdx oi = make_memop_idx(MO_BEUL | MO_UNALN, mmu_idx);
- cpu_stl_be_mmu(env, addr, val, oi, ra);
+ cpu_stl_mmu(env, addr, val, oi, ra);
}
void cpu_stq_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint64_t val,
int mmu_idx, uintptr_t ra)
{
MemOpIdx oi = make_memop_idx(MO_BEUQ | MO_UNALN, mmu_idx);
- cpu_stq_be_mmu(env, addr, val, oi, ra);
+ cpu_stq_mmu(env, addr, val, oi, ra);
}
void cpu_stw_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint32_t val,
int mmu_idx, uintptr_t ra)
{
MemOpIdx oi = make_memop_idx(MO_LEUW | MO_UNALN, mmu_idx);
- cpu_stw_le_mmu(env, addr, val, oi, ra);
+ cpu_stw_mmu(env, addr, val, oi, ra);
}
void cpu_stl_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint32_t val,
int mmu_idx, uintptr_t ra)
{
MemOpIdx oi = make_memop_idx(MO_LEUL | MO_UNALN, mmu_idx);
- cpu_stl_le_mmu(env, addr, val, oi, ra);
+ cpu_stl_mmu(env, addr, val, oi, ra);
}
void cpu_stq_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint64_t val,
int mmu_idx, uintptr_t ra)
{
MemOpIdx oi = make_memop_idx(MO_LEUQ | MO_UNALN, mmu_idx);
- cpu_stq_le_mmu(env, addr, val, oi, ra);
+ cpu_stq_mmu(env, addr, val, oi, ra);
}
/*--------------------------*/
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 17/27] target/s390x: Use cpu_{ld,st}*_mmu in do_csst
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (15 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 16/27] accel/tcg: Unify cpu_{ld,st}*_{be,le}_mmu Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-21 11:21 ` Philippe Mathieu-Daudé
2023-05-22 8:43 ` David Hildenbrand
2023-05-20 16:26 ` [PATCH 18/27] target/s390x: Always use cpu_atomic_cmpxchgl_be_mmu " Richard Henderson
` (9 subsequent siblings)
26 siblings, 2 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm, qemu-s390x, David Hildenbrand, Ilya Leoshkevich
Use cpu_ld16_mmu and cpu_st16_mmu to eliminate the special case,
and change all of the *_data_ra functions to match.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
Cc: qemu-s390x@nongnu.org
Cc: David Hildenbrand <david@redhat.com>
Cc: Ilya Leoshkevich <iii@linux.ibm.com>
---
target/s390x/tcg/mem_helper.c | 65 ++++++++++++++---------------------
1 file changed, 26 insertions(+), 39 deletions(-)
diff --git a/target/s390x/tcg/mem_helper.c b/target/s390x/tcg/mem_helper.c
index 0e0d66b3b6..b6cf24403c 100644
--- a/target/s390x/tcg/mem_helper.c
+++ b/target/s390x/tcg/mem_helper.c
@@ -1737,6 +1737,9 @@ static uint32_t do_csst(CPUS390XState *env, uint32_t r3, uint64_t a1,
uint64_t a2, bool parallel)
{
uint32_t mem_idx = cpu_mmu_index(env, false);
+ MemOpIdx oi16 = make_memop_idx(MO_TE | MO_128, mem_idx);
+ MemOpIdx oi8 = make_memop_idx(MO_TE | MO_64, mem_idx);
+ MemOpIdx oi4 = make_memop_idx(MO_TE | MO_32, mem_idx);
uintptr_t ra = GETPC();
uint32_t fc = extract32(env->regs[0], 0, 8);
uint32_t sc = extract32(env->regs[0], 8, 8);
@@ -1780,15 +1783,17 @@ static uint32_t do_csst(CPUS390XState *env, uint32_t r3, uint64_t a1,
}
}
- /* All loads happen before all stores. For simplicity, load the entire
- store value area from the parameter list. */
- svh = cpu_ldq_data_ra(env, pl + 16, ra);
- svl = cpu_ldq_data_ra(env, pl + 24, ra);
+ /*
+ * All loads happen before all stores. For simplicity, load the entire
+ * store value area from the parameter list.
+ */
+ svh = cpu_ldq_mmu(env, pl + 16, oi8, ra);
+ svl = cpu_ldq_mmu(env, pl + 24, oi8, ra);
switch (fc) {
case 0:
{
- uint32_t nv = cpu_ldl_data_ra(env, pl, ra);
+ uint32_t nv = cpu_ldl_mmu(env, pl, oi4, ra);
uint32_t cv = env->regs[r3];
uint32_t ov;
@@ -1801,8 +1806,8 @@ static uint32_t do_csst(CPUS390XState *env, uint32_t r3, uint64_t a1,
ov = cpu_atomic_cmpxchgl_be_mmu(env, a1, cv, nv, oi, ra);
#endif
} else {
- ov = cpu_ldl_data_ra(env, a1, ra);
- cpu_stl_data_ra(env, a1, (ov == cv ? nv : ov), ra);
+ ov = cpu_ldl_mmu(env, a1, oi4, ra);
+ cpu_stl_mmu(env, a1, (ov == cv ? nv : ov), oi4, ra);
}
cc = (ov != cv);
env->regs[r3] = deposit64(env->regs[r3], 32, 32, ov);
@@ -1811,21 +1816,20 @@ static uint32_t do_csst(CPUS390XState *env, uint32_t r3, uint64_t a1,
case 1:
{
- uint64_t nv = cpu_ldq_data_ra(env, pl, ra);
+ uint64_t nv = cpu_ldq_mmu(env, pl, oi8, ra);
uint64_t cv = env->regs[r3];
uint64_t ov;
if (parallel) {
#ifdef CONFIG_ATOMIC64
- MemOpIdx oi = make_memop_idx(MO_TEUQ | MO_ALIGN, mem_idx);
- ov = cpu_atomic_cmpxchgq_be_mmu(env, a1, cv, nv, oi, ra);
+ ov = cpu_atomic_cmpxchgq_be_mmu(env, a1, cv, nv, oi8, ra);
#else
/* Note that we asserted !parallel above. */
g_assert_not_reached();
#endif
} else {
- ov = cpu_ldq_data_ra(env, a1, ra);
- cpu_stq_data_ra(env, a1, (ov == cv ? nv : ov), ra);
+ ov = cpu_ldq_mmu(env, a1, oi8, ra);
+ cpu_stq_mmu(env, a1, (ov == cv ? nv : ov), oi8, ra);
}
cc = (ov != cv);
env->regs[r3] = ov;
@@ -1834,27 +1838,19 @@ static uint32_t do_csst(CPUS390XState *env, uint32_t r3, uint64_t a1,
case 2:
{
- uint64_t nvh = cpu_ldq_data_ra(env, pl, ra);
- uint64_t nvl = cpu_ldq_data_ra(env, pl + 8, ra);
- Int128 nv = int128_make128(nvl, nvh);
+ Int128 nv = cpu_ld16_mmu(env, pl, oi16, ra);
Int128 cv = int128_make128(env->regs[r3 + 1], env->regs[r3]);
Int128 ov;
if (!parallel) {
- uint64_t oh = cpu_ldq_data_ra(env, a1 + 0, ra);
- uint64_t ol = cpu_ldq_data_ra(env, a1 + 8, ra);
-
- ov = int128_make128(ol, oh);
+ ov = cpu_ld16_mmu(env, a1, oi16, ra);
cc = !int128_eq(ov, cv);
if (cc) {
nv = ov;
}
-
- cpu_stq_data_ra(env, a1 + 0, int128_gethi(nv), ra);
- cpu_stq_data_ra(env, a1 + 8, int128_getlo(nv), ra);
+ cpu_st16_mmu(env, a1, nv, oi16, ra);
} else if (HAVE_CMPXCHG128) {
- MemOpIdx oi = make_memop_idx(MO_TE | MO_128 | MO_ALIGN, mem_idx);
- ov = cpu_atomic_cmpxchgo_be_mmu(env, a1, cv, nv, oi, ra);
+ ov = cpu_atomic_cmpxchgo_be_mmu(env, a1, cv, nv, oi16, ra);
cc = !int128_eq(ov, cv);
} else {
/* Note that we asserted !parallel above. */
@@ -1876,29 +1872,20 @@ static uint32_t do_csst(CPUS390XState *env, uint32_t r3, uint64_t a1,
if (cc == 0) {
switch (sc) {
case 0:
- cpu_stb_data_ra(env, a2, svh >> 56, ra);
+ cpu_stb_mmu(env, a2, svh >> 56, make_memop_idx(MO_8, mem_idx), ra);
break;
case 1:
- cpu_stw_data_ra(env, a2, svh >> 48, ra);
+ cpu_stw_mmu(env, a2, svh >> 48,
+ make_memop_idx(MO_TE | MO_16, mem_idx), ra);
break;
case 2:
- cpu_stl_data_ra(env, a2, svh >> 32, ra);
+ cpu_stl_mmu(env, a2, svh >> 32, oi4, ra);
break;
case 3:
- cpu_stq_data_ra(env, a2, svh, ra);
+ cpu_stq_mmu(env, a2, svh, oi8, ra);
break;
case 4:
- if (!parallel) {
- cpu_stq_data_ra(env, a2 + 0, svh, ra);
- cpu_stq_data_ra(env, a2 + 8, svl, ra);
- } else if (HAVE_ATOMIC128) {
- MemOpIdx oi = make_memop_idx(MO_TEUQ | MO_ALIGN_16, mem_idx);
- Int128 sv = int128_make128(svl, svh);
- cpu_atomic_sto_be_mmu(env, a2, sv, oi, ra);
- } else {
- /* Note that we asserted !parallel above. */
- g_assert_not_reached();
- }
+ cpu_st16_mmu(env, a2, int128_make128(svl, svh), oi16, ra);
break;
default:
g_assert_not_reached();
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 18/27] target/s390x: Always use cpu_atomic_cmpxchgl_be_mmu in do_csst
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (16 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 17/27] target/s390x: Use cpu_{ld,st}*_mmu in do_csst Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-22 8:44 ` David Hildenbrand
2023-05-20 16:26 ` [PATCH 19/27] accel/tcg: Remove cpu_atomic_{ld,st}o_*_mmu Richard Henderson
` (8 subsequent siblings)
26 siblings, 1 reply; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm, qemu-s390x, David Hildenbrand, Ilya Leoshkevich
Eliminate the CONFIG_USER_ONLY specialization.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
Cc: qemu-s390x@nongnu.org
Cc: David Hildenbrand <david@redhat.com>
Cc: Ilya Leoshkevich <iii@linux.ibm.com>
---
target/s390x/tcg/mem_helper.c | 8 +-------
1 file changed, 1 insertion(+), 7 deletions(-)
diff --git a/target/s390x/tcg/mem_helper.c b/target/s390x/tcg/mem_helper.c
index b6cf24403c..bad789a742 100644
--- a/target/s390x/tcg/mem_helper.c
+++ b/target/s390x/tcg/mem_helper.c
@@ -1798,13 +1798,7 @@ static uint32_t do_csst(CPUS390XState *env, uint32_t r3, uint64_t a1,
uint32_t ov;
if (parallel) {
-#ifdef CONFIG_USER_ONLY
- uint32_t *haddr = g2h(env_cpu(env), a1);
- ov = qatomic_cmpxchg__nocheck(haddr, cv, nv);
-#else
- MemOpIdx oi = make_memop_idx(MO_TEUL | MO_ALIGN, mem_idx);
- ov = cpu_atomic_cmpxchgl_be_mmu(env, a1, cv, nv, oi, ra);
-#endif
+ ov = cpu_atomic_cmpxchgl_be_mmu(env, a1, cv, nv, oi4, ra);
} else {
ov = cpu_ldl_mmu(env, a1, oi4, ra);
cpu_stl_mmu(env, a1, (ov == cv ? nv : ov), oi4, ra);
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 19/27] accel/tcg: Remove cpu_atomic_{ld,st}o_*_mmu
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (17 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 18/27] target/s390x: Always use cpu_atomic_cmpxchgl_be_mmu " Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-20 16:26 ` [PATCH 20/27] accel/tcg: Remove prot argument to atomic_mmu_lookup Richard Henderson
` (7 subsequent siblings)
26 siblings, 0 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm
Atomic load/store of 128-byte quantities is now handled
by cpu_{ld,st}16_mmu.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
accel/tcg/atomic_template.h | 61 +++--------------------------------
include/exec/cpu_ldst.h | 9 ------
accel/tcg/atomic_common.c.inc | 14 --------
3 files changed, 4 insertions(+), 80 deletions(-)
diff --git a/accel/tcg/atomic_template.h b/accel/tcg/atomic_template.h
index 404a530f7c..30eee9d066 100644
--- a/accel/tcg/atomic_template.h
+++ b/accel/tcg/atomic_template.h
@@ -87,33 +87,7 @@ ABI_TYPE ATOMIC_NAME(cmpxchg)(CPUArchState *env, target_ulong addr,
return ret;
}
-#if DATA_SIZE >= 16
-#if HAVE_ATOMIC128
-ABI_TYPE ATOMIC_NAME(ld)(CPUArchState *env, target_ulong addr,
- MemOpIdx oi, uintptr_t retaddr)
-{
- DATA_TYPE *haddr = atomic_mmu_lookup(env, addr, oi, DATA_SIZE,
- PAGE_READ, retaddr);
- DATA_TYPE val;
-
- val = atomic16_read(haddr);
- ATOMIC_MMU_CLEANUP;
- atomic_trace_ld_post(env, addr, oi);
- return val;
-}
-
-void ATOMIC_NAME(st)(CPUArchState *env, target_ulong addr, ABI_TYPE val,
- MemOpIdx oi, uintptr_t retaddr)
-{
- DATA_TYPE *haddr = atomic_mmu_lookup(env, addr, oi, DATA_SIZE,
- PAGE_WRITE, retaddr);
-
- atomic16_set(haddr, val);
- ATOMIC_MMU_CLEANUP;
- atomic_trace_st_post(env, addr, oi);
-}
-#endif
-#else
+#if DATA_SIZE < 16
ABI_TYPE ATOMIC_NAME(xchg)(CPUArchState *env, target_ulong addr, ABI_TYPE val,
MemOpIdx oi, uintptr_t retaddr)
{
@@ -188,7 +162,7 @@ GEN_ATOMIC_HELPER_FN(smax_fetch, MAX, SDATA_TYPE, new)
GEN_ATOMIC_HELPER_FN(umax_fetch, MAX, DATA_TYPE, new)
#undef GEN_ATOMIC_HELPER_FN
-#endif /* DATA SIZE >= 16 */
+#endif /* DATA SIZE < 16 */
#undef END
@@ -220,34 +194,7 @@ ABI_TYPE ATOMIC_NAME(cmpxchg)(CPUArchState *env, target_ulong addr,
return BSWAP(ret);
}
-#if DATA_SIZE >= 16
-#if HAVE_ATOMIC128
-ABI_TYPE ATOMIC_NAME(ld)(CPUArchState *env, target_ulong addr,
- MemOpIdx oi, uintptr_t retaddr)
-{
- DATA_TYPE *haddr = atomic_mmu_lookup(env, addr, oi, DATA_SIZE,
- PAGE_READ, retaddr);
- DATA_TYPE val;
-
- val = atomic16_read(haddr);
- ATOMIC_MMU_CLEANUP;
- atomic_trace_ld_post(env, addr, oi);
- return BSWAP(val);
-}
-
-void ATOMIC_NAME(st)(CPUArchState *env, target_ulong addr, ABI_TYPE val,
- MemOpIdx oi, uintptr_t retaddr)
-{
- DATA_TYPE *haddr = atomic_mmu_lookup(env, addr, oi, DATA_SIZE,
- PAGE_WRITE, retaddr);
-
- val = BSWAP(val);
- atomic16_set(haddr, val);
- ATOMIC_MMU_CLEANUP;
- atomic_trace_st_post(env, addr, oi);
-}
-#endif
-#else
+#if DATA_SIZE < 16
ABI_TYPE ATOMIC_NAME(xchg)(CPUArchState *env, target_ulong addr, ABI_TYPE val,
MemOpIdx oi, uintptr_t retaddr)
{
@@ -326,7 +273,7 @@ GEN_ATOMIC_HELPER_FN(add_fetch, ADD, DATA_TYPE, new)
#undef ADD
#undef GEN_ATOMIC_HELPER_FN
-#endif /* DATA_SIZE >= 16 */
+#endif /* DATA_SIZE < 16 */
#undef END
#endif /* DATA_SIZE > 1 */
diff --git a/include/exec/cpu_ldst.h b/include/exec/cpu_ldst.h
index fc1d3d9301..5939688f69 100644
--- a/include/exec/cpu_ldst.h
+++ b/include/exec/cpu_ldst.h
@@ -300,15 +300,6 @@ Int128 cpu_atomic_cmpxchgo_be_mmu(CPUArchState *env, target_ulong addr,
Int128 cmpv, Int128 newv,
MemOpIdx oi, uintptr_t retaddr);
-Int128 cpu_atomic_ldo_le_mmu(CPUArchState *env, target_ulong addr,
- MemOpIdx oi, uintptr_t retaddr);
-Int128 cpu_atomic_ldo_be_mmu(CPUArchState *env, target_ulong addr,
- MemOpIdx oi, uintptr_t retaddr);
-void cpu_atomic_sto_le_mmu(CPUArchState *env, target_ulong addr, Int128 val,
- MemOpIdx oi, uintptr_t retaddr);
-void cpu_atomic_sto_be_mmu(CPUArchState *env, target_ulong addr, Int128 val,
- MemOpIdx oi, uintptr_t retaddr);
-
#if defined(CONFIG_USER_ONLY)
extern __thread uintptr_t helper_retaddr;
diff --git a/accel/tcg/atomic_common.c.inc b/accel/tcg/atomic_common.c.inc
index fe0eea018f..f255c9e215 100644
--- a/accel/tcg/atomic_common.c.inc
+++ b/accel/tcg/atomic_common.c.inc
@@ -19,20 +19,6 @@ static void atomic_trace_rmw_post(CPUArchState *env, uint64_t addr,
qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_RW);
}
-#if HAVE_ATOMIC128
-static void atomic_trace_ld_post(CPUArchState *env, uint64_t addr,
- MemOpIdx oi)
-{
- qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_R);
-}
-
-static void atomic_trace_st_post(CPUArchState *env, uint64_t addr,
- MemOpIdx oi)
-{
- qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, oi, QEMU_PLUGIN_MEM_W);
-}
-#endif
-
/*
* Atomic helpers callable from TCG.
* These have a common interface and all defer to cpu_atomic_*
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 20/27] accel/tcg: Remove prot argument to atomic_mmu_lookup
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (18 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 19/27] accel/tcg: Remove cpu_atomic_{ld,st}o_*_mmu Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-20 16:26 ` [PATCH 21/27] accel/tcg: Eliminate #if on HAVE_ATOMIC128 and HAVE_CMPXCHG128 Richard Henderson
` (6 subsequent siblings)
26 siblings, 0 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm
Now that load/store are gone, we're always passing
PAGE_READ | PAGE_WRITE for RMW atomic operations.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
accel/tcg/atomic_template.h | 32 ++++++--------
accel/tcg/cputlb.c | 85 ++++++++++++++-----------------------
accel/tcg/user-exec.c | 8 +---
3 files changed, 45 insertions(+), 80 deletions(-)
diff --git a/accel/tcg/atomic_template.h b/accel/tcg/atomic_template.h
index 30eee9d066..e312acd16d 100644
--- a/accel/tcg/atomic_template.h
+++ b/accel/tcg/atomic_template.h
@@ -73,8 +73,7 @@ ABI_TYPE ATOMIC_NAME(cmpxchg)(CPUArchState *env, target_ulong addr,
ABI_TYPE cmpv, ABI_TYPE newv,
MemOpIdx oi, uintptr_t retaddr)
{
- DATA_TYPE *haddr = atomic_mmu_lookup(env, addr, oi, DATA_SIZE,
- PAGE_READ | PAGE_WRITE, retaddr);
+ DATA_TYPE *haddr = atomic_mmu_lookup(env, addr, oi, DATA_SIZE, retaddr);
DATA_TYPE ret;
#if DATA_SIZE == 16
@@ -91,8 +90,7 @@ ABI_TYPE ATOMIC_NAME(cmpxchg)(CPUArchState *env, target_ulong addr,
ABI_TYPE ATOMIC_NAME(xchg)(CPUArchState *env, target_ulong addr, ABI_TYPE val,
MemOpIdx oi, uintptr_t retaddr)
{
- DATA_TYPE *haddr = atomic_mmu_lookup(env, addr, oi, DATA_SIZE,
- PAGE_READ | PAGE_WRITE, retaddr);
+ DATA_TYPE *haddr = atomic_mmu_lookup(env, addr, oi, DATA_SIZE, retaddr);
DATA_TYPE ret;
ret = qatomic_xchg__nocheck(haddr, val);
@@ -105,9 +103,8 @@ ABI_TYPE ATOMIC_NAME(xchg)(CPUArchState *env, target_ulong addr, ABI_TYPE val,
ABI_TYPE ATOMIC_NAME(X)(CPUArchState *env, target_ulong addr, \
ABI_TYPE val, MemOpIdx oi, uintptr_t retaddr) \
{ \
- DATA_TYPE *haddr = atomic_mmu_lookup(env, addr, oi, DATA_SIZE, \
- PAGE_READ | PAGE_WRITE, retaddr); \
- DATA_TYPE ret; \
+ DATA_TYPE *haddr, ret; \
+ haddr = atomic_mmu_lookup(env, addr, oi, DATA_SIZE, retaddr); \
ret = qatomic_##X(haddr, val); \
ATOMIC_MMU_CLEANUP; \
atomic_trace_rmw_post(env, addr, oi); \
@@ -137,9 +134,8 @@ GEN_ATOMIC_HELPER(xor_fetch)
ABI_TYPE ATOMIC_NAME(X)(CPUArchState *env, target_ulong addr, \
ABI_TYPE xval, MemOpIdx oi, uintptr_t retaddr) \
{ \
- XDATA_TYPE *haddr = atomic_mmu_lookup(env, addr, oi, DATA_SIZE, \
- PAGE_READ | PAGE_WRITE, retaddr); \
- XDATA_TYPE cmp, old, new, val = xval; \
+ XDATA_TYPE *haddr, cmp, old, new, val = xval; \
+ haddr = atomic_mmu_lookup(env, addr, oi, DATA_SIZE, retaddr); \
smp_mb(); \
cmp = qatomic_read__nocheck(haddr); \
do { \
@@ -180,8 +176,7 @@ ABI_TYPE ATOMIC_NAME(cmpxchg)(CPUArchState *env, target_ulong addr,
ABI_TYPE cmpv, ABI_TYPE newv,
MemOpIdx oi, uintptr_t retaddr)
{
- DATA_TYPE *haddr = atomic_mmu_lookup(env, addr, oi, DATA_SIZE,
- PAGE_READ | PAGE_WRITE, retaddr);
+ DATA_TYPE *haddr = atomic_mmu_lookup(env, addr, oi, DATA_SIZE, retaddr);
DATA_TYPE ret;
#if DATA_SIZE == 16
@@ -198,8 +193,7 @@ ABI_TYPE ATOMIC_NAME(cmpxchg)(CPUArchState *env, target_ulong addr,
ABI_TYPE ATOMIC_NAME(xchg)(CPUArchState *env, target_ulong addr, ABI_TYPE val,
MemOpIdx oi, uintptr_t retaddr)
{
- DATA_TYPE *haddr = atomic_mmu_lookup(env, addr, oi, DATA_SIZE,
- PAGE_READ | PAGE_WRITE, retaddr);
+ DATA_TYPE *haddr = atomic_mmu_lookup(env, addr, oi, DATA_SIZE, retaddr);
ABI_TYPE ret;
ret = qatomic_xchg__nocheck(haddr, BSWAP(val));
@@ -212,9 +206,8 @@ ABI_TYPE ATOMIC_NAME(xchg)(CPUArchState *env, target_ulong addr, ABI_TYPE val,
ABI_TYPE ATOMIC_NAME(X)(CPUArchState *env, target_ulong addr, \
ABI_TYPE val, MemOpIdx oi, uintptr_t retaddr) \
{ \
- DATA_TYPE *haddr = atomic_mmu_lookup(env, addr, oi, DATA_SIZE, \
- PAGE_READ | PAGE_WRITE, retaddr); \
- DATA_TYPE ret; \
+ DATA_TYPE *haddr, ret; \
+ haddr = atomic_mmu_lookup(env, addr, oi, DATA_SIZE, retaddr); \
ret = qatomic_##X(haddr, BSWAP(val)); \
ATOMIC_MMU_CLEANUP; \
atomic_trace_rmw_post(env, addr, oi); \
@@ -241,9 +234,8 @@ GEN_ATOMIC_HELPER(xor_fetch)
ABI_TYPE ATOMIC_NAME(X)(CPUArchState *env, target_ulong addr, \
ABI_TYPE xval, MemOpIdx oi, uintptr_t retaddr) \
{ \
- XDATA_TYPE *haddr = atomic_mmu_lookup(env, addr, oi, DATA_SIZE, \
- PAGE_READ | PAGE_WRITE, retaddr); \
- XDATA_TYPE ldo, ldn, old, new, val = xval; \
+ XDATA_TYPE *haddr, ldo, ldn, old, new, val = xval; \
+ haddr = atomic_mmu_lookup(env, addr, oi, DATA_SIZE, retaddr); \
smp_mb(); \
ldn = qatomic_read__nocheck(haddr); \
do { \
diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index b1e13d165c..9cb0b697d1 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -1896,12 +1896,9 @@ static bool mmu_lookup(CPUArchState *env, target_ulong addr, MemOpIdx oi,
/*
* Probe for an atomic operation. Do not allow unaligned operations,
* or io operations to proceed. Return the host address.
- *
- * @prot may be PAGE_READ, PAGE_WRITE, or PAGE_READ|PAGE_WRITE.
*/
static void *atomic_mmu_lookup(CPUArchState *env, target_ulong addr,
- MemOpIdx oi, int size, int prot,
- uintptr_t retaddr)
+ MemOpIdx oi, int size, uintptr_t retaddr)
{
uintptr_t mmu_idx = get_mmuidx(oi);
MemOp mop = get_memop(oi);
@@ -1937,54 +1934,37 @@ static void *atomic_mmu_lookup(CPUArchState *env, target_ulong addr,
tlbe = tlb_entry(env, mmu_idx, addr);
/* Check TLB entry and enforce page permissions. */
- if (prot & PAGE_WRITE) {
- tlb_addr = tlb_addr_write(tlbe);
- if (!tlb_hit(tlb_addr, addr)) {
- if (!victim_tlb_hit(env, mmu_idx, index, MMU_DATA_STORE,
- addr & TARGET_PAGE_MASK)) {
- tlb_fill(env_cpu(env), addr, size,
- MMU_DATA_STORE, mmu_idx, retaddr);
- index = tlb_index(env, mmu_idx, addr);
- tlbe = tlb_entry(env, mmu_idx, addr);
- }
- tlb_addr = tlb_addr_write(tlbe) & ~TLB_INVALID_MASK;
- }
-
- if (prot & PAGE_READ) {
- /*
- * Let the guest notice RMW on a write-only page.
- * We have just verified that the page is writable.
- * Subpage lookups may have left TLB_INVALID_MASK set,
- * but addr_read will only be -1 if PAGE_READ was unset.
- */
- if (unlikely(tlbe->addr_read == -1)) {
- tlb_fill(env_cpu(env), addr, size,
- MMU_DATA_LOAD, mmu_idx, retaddr);
- /*
- * Since we don't support reads and writes to different
- * addresses, and we do have the proper page loaded for
- * write, this shouldn't ever return. But just in case,
- * handle via stop-the-world.
- */
- goto stop_the_world;
- }
- /* Collect TLB_WATCHPOINT for read. */
- tlb_addr |= tlbe->addr_read;
- }
- } else /* if (prot & PAGE_READ) */ {
- tlb_addr = tlbe->addr_read;
- if (!tlb_hit(tlb_addr, addr)) {
- if (!victim_tlb_hit(env, mmu_idx, index, MMU_DATA_LOAD,
- addr & TARGET_PAGE_MASK)) {
- tlb_fill(env_cpu(env), addr, size,
- MMU_DATA_LOAD, mmu_idx, retaddr);
- index = tlb_index(env, mmu_idx, addr);
- tlbe = tlb_entry(env, mmu_idx, addr);
- }
- tlb_addr = tlbe->addr_read & ~TLB_INVALID_MASK;
+ tlb_addr = tlb_addr_write(tlbe);
+ if (!tlb_hit(tlb_addr, addr)) {
+ if (!victim_tlb_hit(env, mmu_idx, index, MMU_DATA_STORE,
+ addr & TARGET_PAGE_MASK)) {
+ tlb_fill(env_cpu(env), addr, size,
+ MMU_DATA_STORE, mmu_idx, retaddr);
+ index = tlb_index(env, mmu_idx, addr);
+ tlbe = tlb_entry(env, mmu_idx, addr);
}
+ tlb_addr = tlb_addr_write(tlbe) & ~TLB_INVALID_MASK;
}
+ /*
+ * Let the guest notice RMW on a write-only page.
+ * We have just verified that the page is writable.
+ * Subpage lookups may have left TLB_INVALID_MASK set,
+ * but addr_read will only be -1 if PAGE_READ was unset.
+ */
+ if (unlikely(tlbe->addr_read == -1)) {
+ tlb_fill(env_cpu(env), addr, size, MMU_DATA_LOAD, mmu_idx, retaddr);
+ /*
+ * Since we don't support reads and writes to different
+ * addresses, and we do have the proper page loaded for
+ * write, this shouldn't ever return. But just in case,
+ * handle via stop-the-world.
+ */
+ goto stop_the_world;
+ }
+ /* Collect TLB_WATCHPOINT for read. */
+ tlb_addr |= tlbe->addr_read;
+
/* Notice an IO access or a needs-MMU-lookup access */
if (unlikely(tlb_addr & (TLB_MMIO | TLB_DISCARD_WRITE))) {
/* There's really nothing that can be done to
@@ -2000,11 +1980,8 @@ static void *atomic_mmu_lookup(CPUArchState *env, target_ulong addr,
}
if (unlikely(tlb_addr & TLB_WATCHPOINT)) {
- QEMU_BUILD_BUG_ON(PAGE_READ != BP_MEM_READ);
- QEMU_BUILD_BUG_ON(PAGE_WRITE != BP_MEM_WRITE);
- /* therefore prot == watchpoint bits */
- cpu_check_watchpoint(env_cpu(env), addr, size,
- full->attrs, prot, retaddr);
+ cpu_check_watchpoint(env_cpu(env), addr, size, full->attrs,
+ BP_MEM_READ | BP_MEM_WRITE, retaddr);
}
return hostaddr;
diff --git a/accel/tcg/user-exec.c b/accel/tcg/user-exec.c
index 19c2849c21..1e085b1210 100644
--- a/accel/tcg/user-exec.c
+++ b/accel/tcg/user-exec.c
@@ -1323,12 +1323,9 @@ uint64_t cpu_ldq_code_mmu(CPUArchState *env, abi_ptr addr,
/*
* Do not allow unaligned operations to proceed. Return the host address.
- *
- * @prot may be PAGE_READ, PAGE_WRITE, or PAGE_READ|PAGE_WRITE.
*/
static void *atomic_mmu_lookup(CPUArchState *env, target_ulong addr,
- MemOpIdx oi, int size, int prot,
- uintptr_t retaddr)
+ MemOpIdx oi, int size, uintptr_t retaddr)
{
MemOp mop = get_memop(oi);
int a_bits = get_alignment_bits(mop);
@@ -1336,8 +1333,7 @@ static void *atomic_mmu_lookup(CPUArchState *env, target_ulong addr,
/* Enforce guest required alignment. */
if (unlikely(addr & ((1 << a_bits) - 1))) {
- MMUAccessType t = prot == PAGE_READ ? MMU_DATA_LOAD : MMU_DATA_STORE;
- cpu_loop_exit_sigbus(env_cpu(env), addr, t, retaddr);
+ cpu_loop_exit_sigbus(env_cpu(env), addr, MMU_DATA_STORE, retaddr);
}
/* Enforce qemu required alignment. */
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 21/27] accel/tcg: Eliminate #if on HAVE_ATOMIC128 and HAVE_CMPXCHG128
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (19 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 20/27] accel/tcg: Remove prot argument to atomic_mmu_lookup Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-20 16:26 ` [PATCH 22/27] qemu/atomic128: Split atomic16_read Richard Henderson
` (5 subsequent siblings)
26 siblings, 0 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm
These symbols will shortly become dynamic runtime tests and
therefore not appropriate for the preprocessor. Use the
matching CONFIG_* symbols for that purpose.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
host/include/aarch64/host/atomic128-cas.h | 2 ++
host/include/generic/host/atomic128-ldst.h | 2 +-
accel/tcg/cputlb.c | 2 +-
accel/tcg/user-exec.c | 2 +-
4 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/host/include/aarch64/host/atomic128-cas.h b/host/include/aarch64/host/atomic128-cas.h
index 33f365ce67..ff0451d1aa 100644
--- a/host/include/aarch64/host/atomic128-cas.h
+++ b/host/include/aarch64/host/atomic128-cas.h
@@ -37,6 +37,8 @@ static inline Int128 atomic16_cmpxchg(Int128 *ptr, Int128 cmp, Int128 new)
return int128_make128(oldl, oldh);
}
+
+# define CONFIG_CMPXCHG128 1
# define HAVE_CMPXCHG128 1
#endif
diff --git a/host/include/generic/host/atomic128-ldst.h b/host/include/generic/host/atomic128-ldst.h
index 46911dfb61..06a62e9dd0 100644
--- a/host/include/generic/host/atomic128-ldst.h
+++ b/host/include/generic/host/atomic128-ldst.h
@@ -33,7 +33,7 @@ atomic16_set(Int128 *ptr, Int128 val)
}
# define HAVE_ATOMIC128 1
-#elif !defined(CONFIG_USER_ONLY) && HAVE_CMPXCHG128
+#elif defined(CONFIG_CMPXCHG128) && !defined(CONFIG_USER_ONLY)
static inline Int128 ATTRIBUTE_ATOMIC128_OPT
atomic16_read(Int128 *ptr)
{
diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 9cb0b697d1..0bd06bf894 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -3038,7 +3038,7 @@ void cpu_st16_mmu(CPUArchState *env, target_ulong addr, Int128 val,
#include "atomic_template.h"
#endif
-#if HAVE_CMPXCHG128 || HAVE_ATOMIC128
+#if defined(CONFIG_ATOMIC128) || defined(CONFIG_CMPXCHG128)
#define DATA_SIZE 16
#include "atomic_template.h"
#endif
diff --git a/accel/tcg/user-exec.c b/accel/tcg/user-exec.c
index 1e085b1210..dc8d6b5d40 100644
--- a/accel/tcg/user-exec.c
+++ b/accel/tcg/user-exec.c
@@ -1371,7 +1371,7 @@ static void *atomic_mmu_lookup(CPUArchState *env, target_ulong addr,
#include "atomic_template.h"
#endif
-#if HAVE_ATOMIC128 || HAVE_CMPXCHG128
+#if defined(CONFIG_ATOMIC128) || defined(CONFIG_CMPXCHG128)
#define DATA_SIZE 16
#include "atomic_template.h"
#endif
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 22/27] qemu/atomic128: Split atomic16_read
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (20 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 21/27] accel/tcg: Eliminate #if on HAVE_ATOMIC128 and HAVE_CMPXCHG128 Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-20 16:26 ` [PATCH 23/27] accel/tcg: Correctly use atomic128.h in ldst_atomicity.c.inc Richard Henderson
` (4 subsequent siblings)
26 siblings, 0 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm
Create both atomic16_read_ro and atomic16_read_rw.
Previously we pretended that we had atomic16_read in system mode,
because we "know" that all ram is always writable to the host.
Now, expose read-only and read-write versions all of the time.
For aarch64, do not fall back to __atomic_read_16 even if
supported by the compiler, to work around a clang bug.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
host/include/aarch64/host/atomic128-ldst.h | 21 ++++++++-------
host/include/generic/host/atomic128-ldst.h | 31 ++++++++++++++++------
target/s390x/tcg/mem_helper.c | 2 +-
3 files changed, 36 insertions(+), 18 deletions(-)
diff --git a/host/include/aarch64/host/atomic128-ldst.h b/host/include/aarch64/host/atomic128-ldst.h
index c2e7b44bc5..6959b2bd8e 100644
--- a/host/include/aarch64/host/atomic128-ldst.h
+++ b/host/include/aarch64/host/atomic128-ldst.h
@@ -11,10 +11,18 @@
#ifndef AARCH64_ATOMIC128_LDST_H
#define AARCH64_ATOMIC128_LDST_H
-/* Through gcc 10, aarch64 has no support for 128-bit atomics. */
-#if !defined(CONFIG_ATOMIC128) && !defined(CONFIG_USER_ONLY)
-/* We can do better than cmpxchg for AArch64. */
-static inline Int128 atomic16_read(Int128 *ptr)
+/*
+ * Through gcc 10, aarch64 has no support for 128-bit atomics.
+ * Through clang 16, without -march=armv8.4-a, __atomic_load_16
+ * is incorrectly expanded to a read-write operation.
+ */
+
+#define HAVE_ATOMIC128_RO 0
+#define HAVE_ATOMIC128_RW 1
+
+Int128 QEMU_ERROR("unsupported atomic") atomic16_read_ro(const Int128 *ptr);
+
+static inline Int128 atomic16_read_rw(Int128 *ptr)
{
uint64_t l, h;
uint32_t tmp;
@@ -41,9 +49,4 @@ static inline void atomic16_set(Int128 *ptr, Int128 val)
: [l] "r"(l), [h] "r"(h));
}
-# define HAVE_ATOMIC128 1
-#else
-#include "host/include/generic/host/atomic128-ldst.h"
-#endif
-
#endif /* AARCH64_ATOMIC128_LDST_H */
diff --git a/host/include/generic/host/atomic128-ldst.h b/host/include/generic/host/atomic128-ldst.h
index 06a62e9dd0..79d208b7a4 100644
--- a/host/include/generic/host/atomic128-ldst.h
+++ b/host/include/generic/host/atomic128-ldst.h
@@ -12,16 +12,25 @@
#define HOST_ATOMIC128_LDST_H
#if defined(CONFIG_ATOMIC128)
+# define HAVE_ATOMIC128_RO 1
+# define HAVE_ATOMIC128_RW 1
+
static inline Int128 ATTRIBUTE_ATOMIC128_OPT
-atomic16_read(Int128 *ptr)
+atomic16_read_ro(const Int128 *ptr)
{
- __int128_t *ptr_align = __builtin_assume_aligned(ptr, 16);
+ const __int128_t *ptr_align = __builtin_assume_aligned(ptr, 16);
Int128Alias r;
r.i = qatomic_read__nocheck(ptr_align);
return r.s;
}
+static inline Int128 ATTRIBUTE_ATOMIC128_OPT
+atomic16_read_rw(Int128 *ptr)
+{
+ return atomic16_read_ro(ptr);
+}
+
static inline void ATTRIBUTE_ATOMIC128_OPT
atomic16_set(Int128 *ptr, Int128 val)
{
@@ -32,10 +41,14 @@ atomic16_set(Int128 *ptr, Int128 val)
qatomic_set__nocheck(ptr_align, v.i);
}
-# define HAVE_ATOMIC128 1
-#elif defined(CONFIG_CMPXCHG128) && !defined(CONFIG_USER_ONLY)
+#elif defined(CONFIG_CMPXCHG128)
+# define HAVE_ATOMIC128_RO 0
+# define HAVE_ATOMIC128_RW 1
+
+Int128 QEMU_ERROR("unsupported atomic") atomic16_read_ro(const Int128 *ptr);
+
static inline Int128 ATTRIBUTE_ATOMIC128_OPT
-atomic16_read(Int128 *ptr)
+atomic16_read_rw(Int128 *ptr)
{
/* Maybe replace 0 with 0, returning the old value. */
Int128 z = int128_make64(0);
@@ -52,12 +65,14 @@ atomic16_set(Int128 *ptr, Int128 val)
} while (int128_ne(old, cmp));
}
-# define HAVE_ATOMIC128 1
#else
+# define HAVE_ATOMIC128_RO 0
+# define HAVE_ATOMIC128_RW 0
+
/* Fallback definitions that must be optimized away, or error. */
-Int128 QEMU_ERROR("unsupported atomic") atomic16_read(Int128 *ptr);
+Int128 QEMU_ERROR("unsupported atomic") atomic16_read_ro(const Int128 *ptr);
+Int128 QEMU_ERROR("unsupported atomic") atomic16_read_rw(Int128 *ptr);
void QEMU_ERROR("unsupported atomic") atomic16_set(Int128 *ptr, Int128 val);
-# define HAVE_ATOMIC128 0
#endif
#endif /* HOST_ATOMIC128_LDST_H */
diff --git a/target/s390x/tcg/mem_helper.c b/target/s390x/tcg/mem_helper.c
index bad789a742..db22995171 100644
--- a/target/s390x/tcg/mem_helper.c
+++ b/target/s390x/tcg/mem_helper.c
@@ -1778,7 +1778,7 @@ static uint32_t do_csst(CPUS390XState *env, uint32_t r3, uint64_t a1,
max = 3;
#endif
if ((HAVE_CMPXCHG128 ? 0 : fc + 2 > max) ||
- (HAVE_ATOMIC128 ? 0 : sc > max)) {
+ (HAVE_ATOMIC128_RW ? 0 : sc > max)) {
cpu_loop_exit_atomic(env_cpu(env), ra);
}
}
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 23/27] accel/tcg: Correctly use atomic128.h in ldst_atomicity.c.inc
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (21 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 22/27] qemu/atomic128: Split atomic16_read Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-20 16:26 ` [PATCH 24/27] tcg: Split out tcg/debug-assert.h Richard Henderson
` (3 subsequent siblings)
26 siblings, 0 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm
Remove the locally defined load_atomic16 and store_atomic16,
along with HAVE_al16 and HAVE_al16_fast in favor of the
routines defined in atomic128.h.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
accel/tcg/cputlb.c | 2 +-
accel/tcg/ldst_atomicity.c.inc | 118 +++++++--------------------------
2 files changed, 24 insertions(+), 96 deletions(-)
diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 0bd06bf894..90c72c9940 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -2712,7 +2712,7 @@ static uint64_t do_st16_leN(CPUArchState *env, MMULookupPageData *p,
case MO_ATOM_WITHIN16_PAIR:
/* Since size > 8, this is the half that must be atomic. */
- if (!HAVE_al16) {
+ if (!HAVE_ATOMIC128_RW) {
cpu_loop_exit_atomic(env_cpu(env), ra);
}
return store_whole_le16(p->haddr, p->size, val_le);
diff --git a/accel/tcg/ldst_atomicity.c.inc b/accel/tcg/ldst_atomicity.c.inc
index b89631bbef..0f6b3f8ab6 100644
--- a/accel/tcg/ldst_atomicity.c.inc
+++ b/accel/tcg/ldst_atomicity.c.inc
@@ -16,18 +16,6 @@
#endif
#define HAVE_al8_fast (ATOMIC_REG_SIZE >= 8)
-#if defined(CONFIG_ATOMIC128)
-# define HAVE_al16_fast true
-#else
-# define HAVE_al16_fast false
-#endif
-#if defined(CONFIG_ATOMIC128) || defined(CONFIG_CMPXCHG128)
-# define HAVE_al16 true
-#else
-# define HAVE_al16 false
-#endif
-
-
/**
* required_atomicity:
*
@@ -146,26 +134,6 @@ static inline uint64_t load_atomic8(void *pv)
return qatomic_read__nocheck(p);
}
-/**
- * load_atomic16:
- * @pv: host address
- *
- * Atomically load 16 aligned bytes from @pv.
- */
-static inline Int128 ATTRIBUTE_ATOMIC128_OPT
-load_atomic16(void *pv)
-{
-#ifdef CONFIG_ATOMIC128
- __uint128_t *p = __builtin_assume_aligned(pv, 16);
- Int128Alias r;
-
- r.u = qatomic_read__nocheck(p);
- return r.s;
-#else
- qemu_build_not_reached();
-#endif
-}
-
/**
* load_atomic8_or_exit:
* @env: cpu context
@@ -211,8 +179,8 @@ static Int128 load_atomic16_or_exit(CPUArchState *env, uintptr_t ra, void *pv)
{
Int128 *p = __builtin_assume_aligned(pv, 16);
- if (HAVE_al16_fast) {
- return load_atomic16(p);
+ if (HAVE_ATOMIC128_RO) {
+ return atomic16_read_ro(p);
}
#ifdef CONFIG_USER_ONLY
@@ -232,14 +200,9 @@ static Int128 load_atomic16_or_exit(CPUArchState *env, uintptr_t ra, void *pv)
* In system mode all guest pages are writable, and for user-only
* we have just checked writability. Try cmpxchg.
*/
-#if defined(CONFIG_CMPXCHG128)
- /* Swap 0 with 0, with the side-effect of returning the old value. */
- {
- Int128Alias r;
- r.u = __sync_val_compare_and_swap_16((__uint128_t *)p, 0, 0);
- return r.s;
+ if (HAVE_ATOMIC128_RW) {
+ return atomic16_read_rw(p);
}
-#endif
/* Ultimate fallback: re-execute in serial context. */
cpu_loop_exit_atomic(env_cpu(env), ra);
@@ -360,11 +323,10 @@ static uint64_t load_atom_extract_al16_or_exit(CPUArchState *env, uintptr_t ra,
static inline uint64_t ATTRIBUTE_ATOMIC128_OPT
load_atom_extract_al16_or_al8(void *pv, int s)
{
-#if defined(CONFIG_ATOMIC128)
uintptr_t pi = (uintptr_t)pv;
int o = pi & 7;
int shr = (HOST_BIG_ENDIAN ? 16 - s - o : o) * 8;
- __uint128_t r;
+ Int128 r;
pv = (void *)(pi & ~7);
if (pi & 8) {
@@ -373,18 +335,14 @@ load_atom_extract_al16_or_al8(void *pv, int s)
uint64_t b = qatomic_read__nocheck(p8 + 1);
if (HOST_BIG_ENDIAN) {
- r = ((__uint128_t)a << 64) | b;
+ r = int128_make128(b, a);
} else {
- r = ((__uint128_t)b << 64) | a;
+ r = int128_make128(a, b);
}
} else {
- __uint128_t *p16 = __builtin_assume_aligned(pv, 16, 0);
- r = qatomic_read__nocheck(p16);
+ r = atomic16_read_ro(pv);
}
- return r >> shr;
-#else
- qemu_build_not_reached();
-#endif
+ return int128_getlo(int128_urshift(r, shr));
}
/**
@@ -472,7 +430,7 @@ static uint16_t load_atom_2(CPUArchState *env, uintptr_t ra,
if (likely((pi & 1) == 0)) {
return load_atomic2(pv);
}
- if (HAVE_al16_fast) {
+ if (HAVE_ATOMIC128_RO) {
return load_atom_extract_al16_or_al8(pv, 2);
}
@@ -511,7 +469,7 @@ static uint32_t load_atom_4(CPUArchState *env, uintptr_t ra,
if (likely((pi & 3) == 0)) {
return load_atomic4(pv);
}
- if (HAVE_al16_fast) {
+ if (HAVE_ATOMIC128_RO) {
return load_atom_extract_al16_or_al8(pv, 4);
}
@@ -557,7 +515,7 @@ static uint64_t load_atom_8(CPUArchState *env, uintptr_t ra,
if (HAVE_al8 && likely((pi & 7) == 0)) {
return load_atomic8(pv);
}
- if (HAVE_al16_fast) {
+ if (HAVE_ATOMIC128_RO) {
return load_atom_extract_al16_or_al8(pv, 8);
}
@@ -607,8 +565,8 @@ static Int128 load_atom_16(CPUArchState *env, uintptr_t ra,
* If the host does not support 16-byte atomics, wait until we have
* examined the atomicity parameters below.
*/
- if (HAVE_al16_fast && likely((pi & 15) == 0)) {
- return load_atomic16(pv);
+ if (HAVE_ATOMIC128_RO && likely((pi & 15) == 0)) {
+ return atomic16_read_ro(pv);
}
atmax = required_atomicity(env, pi, memop);
@@ -687,36 +645,6 @@ static inline void store_atomic8(void *pv, uint64_t val)
qatomic_set__nocheck(p, val);
}
-/**
- * store_atomic16:
- * @pv: host address
- * @val: value to store
- *
- * Atomically store 16 aligned bytes to @pv.
- */
-static inline void ATTRIBUTE_ATOMIC128_OPT
-store_atomic16(void *pv, Int128Alias val)
-{
-#if defined(CONFIG_ATOMIC128)
- __uint128_t *pu = __builtin_assume_aligned(pv, 16);
- qatomic_set__nocheck(pu, val.u);
-#elif defined(CONFIG_CMPXCHG128)
- __uint128_t *pu = __builtin_assume_aligned(pv, 16);
- __uint128_t o;
-
- /*
- * Without CONFIG_ATOMIC128, __atomic_compare_exchange_n will always
- * defer to libatomic, so we must use __sync_*_compare_and_swap_16
- * and accept the sequential consistency that comes with it.
- */
- do {
- o = *pu;
- } while (!__sync_bool_compare_and_swap_16(pu, o, val.u));
-#else
- qemu_build_not_reached();
-#endif
-}
-
/**
* store_atom_4x2
*/
@@ -957,7 +885,7 @@ static uint64_t store_whole_le16(void *pv, int size, Int128 val_le)
int sh = o * 8;
Int128 m, v;
- qemu_build_assert(HAVE_al16);
+ qemu_build_assert(HAVE_ATOMIC128_RW);
/* Like MAKE_64BIT_MASK(0, sz), but larger. */
if (sz <= 64) {
@@ -1017,7 +945,7 @@ static void store_atom_2(CPUArchState *env, uintptr_t ra,
return;
}
} else if ((pi & 15) == 7) {
- if (HAVE_al16) {
+ if (HAVE_ATOMIC128_RW) {
Int128 v = int128_lshift(int128_make64(val), 56);
Int128 m = int128_lshift(int128_make64(0xffff), 56);
store_atom_insert_al16(pv - 7, v, m);
@@ -1086,7 +1014,7 @@ static void store_atom_4(CPUArchState *env, uintptr_t ra,
return;
}
} else {
- if (HAVE_al16) {
+ if (HAVE_ATOMIC128_RW) {
store_whole_le16(pv, 4, int128_make64(cpu_to_le32(val)));
return;
}
@@ -1151,7 +1079,7 @@ static void store_atom_8(CPUArchState *env, uintptr_t ra,
}
break;
case MO_64:
- if (HAVE_al16) {
+ if (HAVE_ATOMIC128_RW) {
store_whole_le16(pv, 8, int128_make64(cpu_to_le64(val)));
return;
}
@@ -1177,8 +1105,8 @@ static void store_atom_16(CPUArchState *env, uintptr_t ra,
uint64_t a, b;
int atmax;
- if (HAVE_al16_fast && likely((pi & 15) == 0)) {
- store_atomic16(pv, val);
+ if (HAVE_ATOMIC128_RW && likely((pi & 15) == 0)) {
+ atomic16_set(pv, val);
return;
}
@@ -1206,7 +1134,7 @@ static void store_atom_16(CPUArchState *env, uintptr_t ra,
}
break;
case -MO_64:
- if (HAVE_al16) {
+ if (HAVE_ATOMIC128_RW) {
uint64_t val_le;
int s2 = pi & 15;
int s1 = 16 - s2;
@@ -1233,8 +1161,8 @@ static void store_atom_16(CPUArchState *env, uintptr_t ra,
}
break;
case MO_128:
- if (HAVE_al16) {
- store_atomic16(pv, val);
+ if (HAVE_ATOMIC128_RW) {
+ atomic16_set(pv, val);
return;
}
break;
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 24/27] tcg: Split out tcg/debug-assert.h
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (22 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 23/27] accel/tcg: Correctly use atomic128.h in ldst_atomicity.c.inc Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-21 11:25 ` Philippe Mathieu-Daudé
2023-05-20 16:26 ` [PATCH 25/27] qemu/atomic128: Improve cmpxchg fallback for atomic16_set Richard Henderson
` (2 subsequent siblings)
26 siblings, 1 reply; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
include/tcg/debug-assert.h | 17 +++++++++++++++++
include/tcg/tcg.h | 9 +--------
2 files changed, 18 insertions(+), 8 deletions(-)
create mode 100644 include/tcg/debug-assert.h
diff --git a/include/tcg/debug-assert.h b/include/tcg/debug-assert.h
new file mode 100644
index 0000000000..596765a3d2
--- /dev/null
+++ b/include/tcg/debug-assert.h
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Define tcg_debug_assert
+ * Copyright (c) 2008 Fabrice Bellard
+ */
+
+#ifndef TCG_DEBUG_ASSERT_H
+#define TCG_DEBUG_ASSERT_H
+
+#if defined CONFIG_DEBUG_TCG || defined QEMU_STATIC_ANALYSIS
+# define tcg_debug_assert(X) do { assert(X); } while (0)
+#else
+# define tcg_debug_assert(X) \
+ do { if (!(X)) { __builtin_unreachable(); } } while (0)
+#endif
+
+#endif
diff --git a/include/tcg/tcg.h b/include/tcg/tcg.h
index cd6327b175..072c35f7f5 100644
--- a/include/tcg/tcg.h
+++ b/include/tcg/tcg.h
@@ -34,6 +34,7 @@
#include "tcg/tcg-mo.h"
#include "tcg-target.h"
#include "tcg/tcg-cond.h"
+#include "tcg/debug-assert.h"
/* XXX: make safe guess about sizes */
#define MAX_OP_PER_INSTR 266
@@ -222,14 +223,6 @@ typedef uint64_t tcg_insn_unit;
/* The port better have done this. */
#endif
-
-#if defined CONFIG_DEBUG_TCG || defined QEMU_STATIC_ANALYSIS
-# define tcg_debug_assert(X) do { assert(X); } while (0)
-#else
-# define tcg_debug_assert(X) \
- do { if (!(X)) { __builtin_unreachable(); } } while (0)
-#endif
-
typedef struct TCGRelocation TCGRelocation;
struct TCGRelocation {
QSIMPLEQ_ENTRY(TCGRelocation) next;
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 25/27] qemu/atomic128: Improve cmpxchg fallback for atomic16_set
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (23 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 24/27] tcg: Split out tcg/debug-assert.h Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-20 16:26 ` [PATCH 26/27] qemu/atomic128: Add runtime test for FEAT_LSE2 Richard Henderson
2023-05-20 16:26 ` [PATCH 27/27] qemu/atomic128: Add x86_64 atomic128-ldst.h Richard Henderson
26 siblings, 0 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm
Use __sync_bool_compare_and_swap_16 to control the loop,
rather than a separate comparison.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
host/include/generic/host/atomic128-ldst.h | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/host/include/generic/host/atomic128-ldst.h b/host/include/generic/host/atomic128-ldst.h
index 79d208b7a4..80fff0643a 100644
--- a/host/include/generic/host/atomic128-ldst.h
+++ b/host/include/generic/host/atomic128-ldst.h
@@ -58,11 +58,14 @@ atomic16_read_rw(Int128 *ptr)
static inline void ATTRIBUTE_ATOMIC128_OPT
atomic16_set(Int128 *ptr, Int128 val)
{
- Int128 old = *ptr, cmp;
+ __int128_t *ptr_align = __builtin_assume_aligned(ptr, 16);
+ __int128_t old;
+ Int128Alias new;
+
+ new.s = val;
do {
- cmp = old;
- old = atomic16_cmpxchg(ptr, cmp, val);
- } while (int128_ne(old, cmp));
+ old = *ptr_align;
+ } while (!__sync_bool_compare_and_swap_16(ptr_align, old, new.i));
}
#else
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 26/27] qemu/atomic128: Add runtime test for FEAT_LSE2
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (24 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 25/27] qemu/atomic128: Improve cmpxchg fallback for atomic16_set Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
2023-05-20 16:26 ` [PATCH 27/27] qemu/atomic128: Add x86_64 atomic128-ldst.h Richard Henderson
26 siblings, 0 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm
With FEAT_LSE2, load and store of int128 is directly supported.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
host/include/aarch64/host/atomic128-ldst.h | 53 ++++++++++++++++------
1 file changed, 40 insertions(+), 13 deletions(-)
diff --git a/host/include/aarch64/host/atomic128-ldst.h b/host/include/aarch64/host/atomic128-ldst.h
index 6959b2bd8e..57455c9b06 100644
--- a/host/include/aarch64/host/atomic128-ldst.h
+++ b/host/include/aarch64/host/atomic128-ldst.h
@@ -11,27 +11,48 @@
#ifndef AARCH64_ATOMIC128_LDST_H
#define AARCH64_ATOMIC128_LDST_H
+#include "host/cpuinfo.h"
+#include "tcg/debug-assert.h"
+
/*
* Through gcc 10, aarch64 has no support for 128-bit atomics.
* Through clang 16, without -march=armv8.4-a, __atomic_load_16
* is incorrectly expanded to a read-write operation.
+ *
+ * Anyway, this method allows runtime detection of FEAT_LSE2.
*/
-#define HAVE_ATOMIC128_RO 0
+#define HAVE_ATOMIC128_RO (cpuinfo & CPUINFO_LSE2)
#define HAVE_ATOMIC128_RW 1
-Int128 QEMU_ERROR("unsupported atomic") atomic16_read_ro(const Int128 *ptr);
+static inline Int128 atomic16_read_ro(const Int128 *ptr)
+{
+ uint64_t l, h;
+
+ tcg_debug_assert(HAVE_ATOMIC128_RO);
+ /* With FEAT_LSE2, 16-byte aligned LDP is atomic. */
+ asm("ldp %[l], %[h], %[mem]"
+ : [l] "=r"(l), [h] "=r"(h) : [mem] "m"(*ptr));
+
+ return int128_make128(l, h);
+}
static inline Int128 atomic16_read_rw(Int128 *ptr)
{
uint64_t l, h;
uint32_t tmp;
- /* The load must be paired with the store to guarantee not tearing. */
- asm("0: ldxp %[l], %[h], %[mem]\n\t"
- "stxp %w[tmp], %[l], %[h], %[mem]\n\t"
- "cbnz %w[tmp], 0b"
- : [mem] "+m"(*ptr), [tmp] "=r"(tmp), [l] "=r"(l), [h] "=r"(h));
+ if (cpuinfo & CPUINFO_LSE2) {
+ /* With FEAT_LSE2, 16-byte aligned LDP is atomic. */
+ asm("ldp %[l], %[h], %[mem]"
+ : [l] "=r"(l), [h] "=r"(h) : [mem] "m"(*ptr));
+ } else {
+ /* The load must be paired with the store to guarantee not tearing. */
+ asm("0: ldxp %[l], %[h], %[mem]\n\t"
+ "stxp %w[tmp], %[l], %[h], %[mem]\n\t"
+ "cbnz %w[tmp], 0b"
+ : [mem] "+m"(*ptr), [tmp] "=r"(tmp), [l] "=r"(l), [h] "=r"(h));
+ }
return int128_make128(l, h);
}
@@ -41,12 +62,18 @@ static inline void atomic16_set(Int128 *ptr, Int128 val)
uint64_t l = int128_getlo(val), h = int128_gethi(val);
uint64_t t1, t2;
- /* Load into temporaries to acquire the exclusive access lock. */
- asm("0: ldxp %[t1], %[t2], %[mem]\n\t"
- "stxp %w[t1], %[l], %[h], %[mem]\n\t"
- "cbnz %w[t1], 0b"
- : [mem] "+m"(*ptr), [t1] "=&r"(t1), [t2] "=&r"(t2)
- : [l] "r"(l), [h] "r"(h));
+ if (cpuinfo & CPUINFO_LSE2) {
+ /* With FEAT_LSE2, 16-byte aligned STP is atomic. */
+ asm("stp %[l], %[h], %[mem]"
+ : [mem] "=m"(*ptr) : [l] "r"(l), [h] "r"(h));
+ } else {
+ /* Load into temporaries to acquire the exclusive access lock. */
+ asm("0: ldxp %[t1], %[t2], %[mem]\n\t"
+ "stxp %w[t1], %[l], %[h], %[mem]\n\t"
+ "cbnz %w[t1], 0b"
+ : [mem] "+m"(*ptr), [t1] "=&r"(t1), [t2] "=&r"(t2)
+ : [l] "r"(l), [h] "r"(h));
+ }
}
#endif /* AARCH64_ATOMIC128_LDST_H */
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [PATCH 27/27] qemu/atomic128: Add x86_64 atomic128-ldst.h
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
` (25 preceding siblings ...)
2023-05-20 16:26 ` [PATCH 26/27] qemu/atomic128: Add runtime test for FEAT_LSE2 Richard Henderson
@ 2023-05-20 16:26 ` Richard Henderson
26 siblings, 0 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-20 16:26 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-arm
With CPUINFO_ATOMIC_VMOVDQA, we can perform proper atomic
load/store without cmpxchg16b.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
host/include/x86_64/host/atomic128-ldst.h | 54 +++++++++++++++++++++++
1 file changed, 54 insertions(+)
create mode 100644 host/include/x86_64/host/atomic128-ldst.h
diff --git a/host/include/x86_64/host/atomic128-ldst.h b/host/include/x86_64/host/atomic128-ldst.h
new file mode 100644
index 0000000000..4be9071d3f
--- /dev/null
+++ b/host/include/x86_64/host/atomic128-ldst.h
@@ -0,0 +1,54 @@
+/*
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ * Load/store for 128-bit atomic operations, x86_64 version.
+ *
+ * Copyright (C) 2023 Linaro, Ltd.
+ *
+ * See docs/devel/atomics.rst for discussion about the guarantees each
+ * atomic primitive is meant to provide.
+ */
+
+#ifndef AARCH64_ATOMIC128_LDST_H
+#define AARCH64_ATOMIC128_LDST_H
+
+#include "host/cpuinfo.h"
+#include "tcg/debug-assert.h"
+
+#define HAVE_ATOMIC128_RO likely(cpuinfo & CPUINFO_ATOMIC_VMOVDQA)
+#define HAVE_ATOMIC128_RW 1
+
+static inline Int128 atomic16_read_ro(const Int128 *ptr)
+{
+ Int128Alias r;
+
+ tcg_debug_assert(HAVE_ATOMIC128_RO);
+ asm("vmovdqa %1, %0" : "=x" (r.i) : "m" (*ptr));
+
+ return r.s;
+}
+
+static inline Int128 atomic16_read_rw(Int128 *ptr)
+{
+ Int128Alias r;
+
+ if (HAVE_ATOMIC128_RO) {
+ asm("vmovdqa %1, %0" : "=x" (r.i) : "m" (*ptr));
+ } else {
+ r.i = __sync_val_compare_and_swap_16(ptr, 0, 0);
+ }
+ return r.s;
+}
+
+static inline void atomic16_set(Int128 *ptr, Int128Alias val)
+{
+ if (HAVE_ATOMIC128_RO) {
+ asm("vmovdqa %1, %0" : "=m"(*ptr) : "x" (val.i));
+ } else {
+ Int128Alias old;
+ do {
+ old.s = *ptr;
+ } while (!__sync_bool_compare_and_swap_16(ptr, old.i, val.i));
+ }
+}
+
+#endif /* AARCH64_ATOMIC128_LDST_H */
--
2.34.1
^ permalink raw reply related [flat|nested] 46+ messages in thread
* Re: [PATCH 10/27] include/host: Split out atomic128-cas.h
2023-05-20 16:26 ` [PATCH 10/27] include/host: Split out atomic128-cas.h Richard Henderson
@ 2023-05-21 10:44 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2023-05-21 10:44 UTC (permalink / raw)
To: Richard Henderson, qemu-devel; +Cc: qemu-arm
On 20/5/23 18:26, Richard Henderson wrote:
> Separates the aarch64-specific portion into its own file.
>
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
> host/include/aarch64/host/atomic128-cas.h | 43 ++++++++++++++++++
> host/include/generic/host/atomic128-cas.h | 43 ++++++++++++++++++
> include/qemu/atomic128.h | 55 +----------------------
> 3 files changed, 87 insertions(+), 54 deletions(-)
> create mode 100644 host/include/aarch64/host/atomic128-cas.h
> create mode 100644 host/include/generic/host/atomic128-cas.h
>
> diff --git a/host/include/aarch64/host/atomic128-cas.h b/host/include/aarch64/host/atomic128-cas.h
> new file mode 100644
> index 0000000000..1247995419
> --- /dev/null
> +++ b/host/include/aarch64/host/atomic128-cas.h
> @@ -0,0 +1,43 @@
> +/*
> + * SPDX-License-Identifier: GPL-2.0-or-later
> + * Compare-and-swap for 128-bit atomic operations, generic version.
"Aarch64 specific"
> + *
> + * Copyright (C) 2018, 2023 Linaro, Ltd.
> + *
> + * See docs/devel/atomics.rst for discussion about the guarantees each
> + * atomic primitive is meant to provide.
> + */
> +
> +#ifndef AARCH64_ATOMIC128_CAS_H
> +#define AARCH64_ATOMIC128_CAS_H
> +
> +/* Through gcc 10, aarch64 has no support for 128-bit atomics. */
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH 01/27] util: Introduce host-specific cpuinfo.h
2023-05-20 16:26 ` [PATCH 01/27] util: Introduce host-specific cpuinfo.h Richard Henderson
@ 2023-05-21 10:47 ` Philippe Mathieu-Daudé
2023-05-23 15:56 ` Alex Bennée
1 sibling, 0 replies; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2023-05-21 10:47 UTC (permalink / raw)
To: Richard Henderson, qemu-devel; +Cc: qemu-arm, Juan Quintela
On 20/5/23 18:26, Richard Henderson wrote:
> The entire contents of the header is host-specific, but the
> existence of such a header is not, which could prevent some
> host specific ifdefs at the top of the file for the include.
>
> Add host/include/{arch,generic} to the project arguments.
>
> Reviewed-by: Juan Quintela <quintela@redhat.com>
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
> host/include/generic/host/cpuinfo.h | 4 ++++
> meson.build | 8 ++++++++
> 2 files changed, 12 insertions(+)
> create mode 100644 host/include/generic/host/cpuinfo.h
>
> diff --git a/host/include/generic/host/cpuinfo.h b/host/include/generic/host/cpuinfo.h
> new file mode 100644
> index 0000000000..eca672064a
> --- /dev/null
> +++ b/host/include/generic/host/cpuinfo.h
> @@ -0,0 +1,4 @@
> +/*
> + * No host specific cpu indentification.
> + * SPDX-License-Identifier: GPL-2.0-or-later
> + */
> diff --git a/meson.build b/meson.build
> index 0a5cdefd4d..4ffc0d3e59 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -512,6 +512,14 @@ add_project_arguments('-iquote', '.',
> '-iquote', meson.current_source_dir() / 'include',
> language: all_languages)
>
> +host_include = meson.current_source_dir() / 'host/include/'
> +if fs.is_dir(host_include / host_arch)
> + add_project_arguments('-iquote', host_include / host_arch,
> + language: all_languages)
> +endif
Maybe add a comment "generic include path must come last, after
host specific include path".
> +add_project_arguments('-iquote', host_include / 'generic',
> + language: all_languages)
> +
> sparse = find_program('cgcc', required: get_option('sparse'))
> if sparse.found()
> run_target('sparse',
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH 12/27] meson: Fix detect atomic128 support with optimization
2023-05-20 16:26 ` [PATCH 12/27] meson: Fix detect atomic128 support with optimization Richard Henderson
@ 2023-05-21 10:54 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2023-05-21 10:54 UTC (permalink / raw)
To: Richard Henderson, qemu-devel; +Cc: qemu-arm
On 20/5/23 18:26, Richard Henderson wrote:
> Silly typo: sizeof(16) != 16.
>
> Fixes: e61f1efeb730 ("meson: Detect atomic128 support with optimization")
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
> meson.build | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meson.build b/meson.build
> index 4ffc0d3e59..5e7fc6345f 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -2555,7 +2555,7 @@ if has_int128
> # __alignof(unsigned __int128) for the host.
> atomic_test_128 = '''
> int main(int ac, char **av) {
> - unsigned __int128 *p = __builtin_assume_aligned(av[ac - 1], sizeof(16));
> + unsigned __int128 *p = __builtin_assume_aligned(av[ac - 1], 16);
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH 16/27] accel/tcg: Unify cpu_{ld,st}*_{be,le}_mmu
2023-05-20 16:26 ` [PATCH 16/27] accel/tcg: Unify cpu_{ld,st}*_{be,le}_mmu Richard Henderson
@ 2023-05-21 11:15 ` Philippe Mathieu-Daudé
2023-05-21 15:00 ` Richard Henderson
0 siblings, 1 reply; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2023-05-21 11:15 UTC (permalink / raw)
To: Richard Henderson, qemu-devel
Cc: qemu-arm, Mark Cave-Ayland, Artyom Tarasenko
Hi Richard,
On 20/5/23 18:26, Richard Henderson wrote:
> With the current structure of cputlb.c, there is no difference
> between the little-endian and big-endian entry points, aside
> from the assert. Unify the pairs of functions.
>
> The only use of the functions with explicit endianness was in
> target/sparc64, and that was only to satisfy the assert.
I'm having hard time to follow all the handling of the various
ASI definitions from target/sparc/asi.h. ...
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
> include/exec/cpu_ldst.h | 58 ++-----
> accel/tcg/cputlb.c | 122 +++-----------
> accel/tcg/user-exec.c | 322 ++++++++++--------------------------
> target/arm/tcg/m_helper.c | 4 +-
> target/sparc/ldst_helper.c | 18 +-
> accel/tcg/ldst_common.c.inc | 24 +--
> 6 files changed, 137 insertions(+), 411 deletions(-)
> diff --git a/target/sparc/ldst_helper.c b/target/sparc/ldst_helper.c
> index 7972d56a72..981a47d8bb 100644
> --- a/target/sparc/ldst_helper.c
> +++ b/target/sparc/ldst_helper.c
> @@ -1334,25 +1334,13 @@ uint64_t helper_ld_asi(CPUSPARCState *env, target_ulong addr,
Shouldn't we propagate the ASI endianness?
...
+ memop |= (asi & 8) ? MO_LE : MO_BE;
oi = make_memop_idx(memop, idx);
switch (size) {
case 1:
> ret = cpu_ldb_mmu(env, addr, oi, GETPC());
> break;
> case 2:
> - if (asi & 8) {
> - ret = cpu_ldw_le_mmu(env, addr, oi, GETPC());
> - } else {
> - ret = cpu_ldw_be_mmu(env, addr, oi, GETPC());
> - }
> + ret = cpu_ldw_mmu(env, addr, oi, GETPC());
> break;
> case 4:
> - if (asi & 8) {
> - ret = cpu_ldl_le_mmu(env, addr, oi, GETPC());
> - } else {
> - ret = cpu_ldl_be_mmu(env, addr, oi, GETPC());
> - }
> + ret = cpu_ldl_mmu(env, addr, oi, GETPC());
> break;
> case 8:
> - if (asi & 8) {
> - ret = cpu_ldq_le_mmu(env, addr, oi, GETPC());
> - } else {
> - ret = cpu_ldq_be_mmu(env, addr, oi, GETPC());
> - }
> + ret = cpu_ldq_mmu(env, addr, oi, GETPC());
> break;
> default:
> g_assert_not_reached();
Otherwise great simplification!
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH 17/27] target/s390x: Use cpu_{ld,st}*_mmu in do_csst
2023-05-20 16:26 ` [PATCH 17/27] target/s390x: Use cpu_{ld,st}*_mmu in do_csst Richard Henderson
@ 2023-05-21 11:21 ` Philippe Mathieu-Daudé
2023-05-21 15:01 ` Richard Henderson
2023-05-22 8:43 ` David Hildenbrand
1 sibling, 1 reply; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2023-05-21 11:21 UTC (permalink / raw)
To: Richard Henderson, qemu-devel
Cc: qemu-arm, qemu-s390x, David Hildenbrand, Ilya Leoshkevich
Hi Richard,
On 20/5/23 18:26, Richard Henderson wrote:
> Use cpu_ld16_mmu and cpu_st16_mmu to eliminate the special case,
> and change all of the *_data_ra functions to match.
>
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
> Cc: qemu-s390x@nongnu.org
> Cc: David Hildenbrand <david@redhat.com>
> Cc: Ilya Leoshkevich <iii@linux.ibm.com>
> ---
> target/s390x/tcg/mem_helper.c | 65 ++++++++++++++---------------------
> 1 file changed, 26 insertions(+), 39 deletions(-)
>
> diff --git a/target/s390x/tcg/mem_helper.c b/target/s390x/tcg/mem_helper.c
> index 0e0d66b3b6..b6cf24403c 100644
> --- a/target/s390x/tcg/mem_helper.c
> +++ b/target/s390x/tcg/mem_helper.c
> @@ -1737,6 +1737,9 @@ static uint32_t do_csst(CPUS390XState *env, uint32_t r3, uint64_t a1,
> uint64_t a2, bool parallel)
> {
> uint32_t mem_idx = cpu_mmu_index(env, false);
> + MemOpIdx oi16 = make_memop_idx(MO_TE | MO_128, mem_idx);
> + MemOpIdx oi8 = make_memop_idx(MO_TE | MO_64, mem_idx);
> if (parallel) {
> #ifdef CONFIG_ATOMIC64
> - MemOpIdx oi = make_memop_idx(MO_TEUQ | MO_ALIGN, mem_idx);
> - ov = cpu_atomic_cmpxchgq_be_mmu(env, a1, cv, nv, oi, ra);
> + ov = cpu_atomic_cmpxchgq_be_mmu(env, a1, cv, nv, oi8, ra);
Why is it safe to remove MO_ALIGN here?
> #else
> /* Note that we asserted !parallel above. */
> g_assert_not_reached();
> #endif
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH 24/27] tcg: Split out tcg/debug-assert.h
2023-05-20 16:26 ` [PATCH 24/27] tcg: Split out tcg/debug-assert.h Richard Henderson
@ 2023-05-21 11:25 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2023-05-21 11:25 UTC (permalink / raw)
To: Richard Henderson, qemu-devel; +Cc: qemu-arm
On 20/5/23 18:26, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
> include/tcg/debug-assert.h | 17 +++++++++++++++++
> include/tcg/tcg.h | 9 +--------
> 2 files changed, 18 insertions(+), 8 deletions(-)
> create mode 100644 include/tcg/debug-assert.h
While here:
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -156,6 +156,7 @@ F: include/exec/target_long.h
F: include/exec/helper*.h
F: include/sysemu/cpus.h
F: include/sysemu/tcg.h
+F: include/tcg/
F: include/hw/core/tcg-cpu-ops.h
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH 02/27] util: Add cpuinfo-i386.c
2023-05-20 16:26 ` [PATCH 02/27] util: Add cpuinfo-i386.c Richard Henderson
@ 2023-05-21 11:28 ` Philippe Mathieu-Daudé
2023-05-21 15:05 ` Richard Henderson
0 siblings, 1 reply; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2023-05-21 11:28 UTC (permalink / raw)
To: Richard Henderson, qemu-devel; +Cc: qemu-arm, Juan Quintela, Paolo Bonzini
On 20/5/23 18:26, Richard Henderson wrote:
> Add cpuinfo.h for i386 and x86_64, and the initialization
> for that in util/. Populate that with a slightly altered
> copy of the tcg host probing code. Other uses of cpuid.h
> will be adjusted one patch at a time.
>
> Reviewed-by: Juan Quintela <quintela@redhat.com>
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
> host/include/i386/host/cpuinfo.h | 38 ++++++++++++
> host/include/x86_64/host/cpuinfo.h | 1 +
> util/cpuinfo-i386.c | 97 ++++++++++++++++++++++++++++++
> util/meson.build | 4 ++
> 4 files changed, 140 insertions(+)
> create mode 100644 host/include/i386/host/cpuinfo.h
> create mode 100644 host/include/x86_64/host/cpuinfo.h
> create mode 100644 util/cpuinfo-i386.c
Missing F: entry in MAINTAINERS file. We probably need new sections.
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH 16/27] accel/tcg: Unify cpu_{ld,st}*_{be,le}_mmu
2023-05-21 11:15 ` Philippe Mathieu-Daudé
@ 2023-05-21 15:00 ` Richard Henderson
2023-05-22 6:39 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 46+ messages in thread
From: Richard Henderson @ 2023-05-21 15:00 UTC (permalink / raw)
To: Philippe Mathieu-Daudé, qemu-devel
Cc: qemu-arm, Mark Cave-Ayland, Artyom Tarasenko
On 5/21/23 04:15, Philippe Mathieu-Daudé wrote:
> Hi Richard,
>
> On 20/5/23 18:26, Richard Henderson wrote:
>> With the current structure of cputlb.c, there is no difference
>> between the little-endian and big-endian entry points, aside
>> from the assert. Unify the pairs of functions.
>>
>> The only use of the functions with explicit endianness was in
>> target/sparc64, and that was only to satisfy the assert.
>
> I'm having hard time to follow all the handling of the various
> ASI definitions from target/sparc/asi.h. ...
>
>> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
>> ---
>> include/exec/cpu_ldst.h | 58 ++-----
>> accel/tcg/cputlb.c | 122 +++-----------
>> accel/tcg/user-exec.c | 322 ++++++++++--------------------------
>> target/arm/tcg/m_helper.c | 4 +-
>> target/sparc/ldst_helper.c | 18 +-
>> accel/tcg/ldst_common.c.inc | 24 +--
>> 6 files changed, 137 insertions(+), 411 deletions(-)
>
>
>> diff --git a/target/sparc/ldst_helper.c b/target/sparc/ldst_helper.c
>> index 7972d56a72..981a47d8bb 100644
>> --- a/target/sparc/ldst_helper.c
>> +++ b/target/sparc/ldst_helper.c
>> @@ -1334,25 +1334,13 @@ uint64_t helper_ld_asi(CPUSPARCState *env, target_ulong addr,
>
>
> Shouldn't we propagate the ASI endianness?
Already done in translate, get_asi():
/* The little-endian asis all have bit 3 set. */
if (asi & 8) {
memop ^= MO_BSWAP;
}
r~
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH 17/27] target/s390x: Use cpu_{ld,st}*_mmu in do_csst
2023-05-21 11:21 ` Philippe Mathieu-Daudé
@ 2023-05-21 15:01 ` Richard Henderson
0 siblings, 0 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-21 15:01 UTC (permalink / raw)
To: Philippe Mathieu-Daudé, qemu-devel
Cc: qemu-arm, qemu-s390x, David Hildenbrand, Ilya Leoshkevich
On 5/21/23 04:21, Philippe Mathieu-Daudé wrote:
> Hi Richard,
>
> On 20/5/23 18:26, Richard Henderson wrote:
>> Use cpu_ld16_mmu and cpu_st16_mmu to eliminate the special case,
>> and change all of the *_data_ra functions to match.
>>
>> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
>> ---
>> Cc: qemu-s390x@nongnu.org
>> Cc: David Hildenbrand <david@redhat.com>
>> Cc: Ilya Leoshkevich <iii@linux.ibm.com>
>> ---
>> target/s390x/tcg/mem_helper.c | 65 ++++++++++++++---------------------
>> 1 file changed, 26 insertions(+), 39 deletions(-)
>>
>> diff --git a/target/s390x/tcg/mem_helper.c b/target/s390x/tcg/mem_helper.c
>> index 0e0d66b3b6..b6cf24403c 100644
>> --- a/target/s390x/tcg/mem_helper.c
>> +++ b/target/s390x/tcg/mem_helper.c
>> @@ -1737,6 +1737,9 @@ static uint32_t do_csst(CPUS390XState *env, uint32_t r3, uint64_t a1,
>> uint64_t a2, bool parallel)
>> {
>> uint32_t mem_idx = cpu_mmu_index(env, false);
>> + MemOpIdx oi16 = make_memop_idx(MO_TE | MO_128, mem_idx);
>> + MemOpIdx oi8 = make_memop_idx(MO_TE | MO_64, mem_idx);
>
>
>> if (parallel) {
>> #ifdef CONFIG_ATOMIC64
>> - MemOpIdx oi = make_memop_idx(MO_TEUQ | MO_ALIGN, mem_idx);
>> - ov = cpu_atomic_cmpxchgq_be_mmu(env, a1, cv, nv, oi, ra);
>> + ov = cpu_atomic_cmpxchgq_be_mmu(env, a1, cv, nv, oi8, ra);
>
> Why is it safe to remove MO_ALIGN here?
Alignment check already done at the start of the function:
/* Sanity check the alignments. */
if (extract32(a1, 0, fc + 2) || extract32(a2, 0, sc)) {
goto spec_exception;
}
r~
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH 02/27] util: Add cpuinfo-i386.c
2023-05-21 11:28 ` Philippe Mathieu-Daudé
@ 2023-05-21 15:05 ` Richard Henderson
2023-05-23 16:01 ` Alex Bennée
0 siblings, 1 reply; 46+ messages in thread
From: Richard Henderson @ 2023-05-21 15:05 UTC (permalink / raw)
To: Philippe Mathieu-Daudé, qemu-devel
Cc: qemu-arm, Juan Quintela, Paolo Bonzini
On 5/21/23 04:28, Philippe Mathieu-Daudé wrote:
> On 20/5/23 18:26, Richard Henderson wrote:
>> Add cpuinfo.h for i386 and x86_64, and the initialization
>> for that in util/. Populate that with a slightly altered
>> copy of the tcg host probing code. Other uses of cpuid.h
>> will be adjusted one patch at a time.
>>
>> Reviewed-by: Juan Quintela <quintela@redhat.com>
>> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
>> ---
>> host/include/i386/host/cpuinfo.h | 38 ++++++++++++
>> host/include/x86_64/host/cpuinfo.h | 1 +
>> util/cpuinfo-i386.c | 97 ++++++++++++++++++++++++++++++
>> util/meson.build | 4 ++
>> 4 files changed, 140 insertions(+)
>> create mode 100644 host/include/i386/host/cpuinfo.h
>> create mode 100644 host/include/x86_64/host/cpuinfo.h
>> create mode 100644 util/cpuinfo-i386.c
>
> Missing F: entry in MAINTAINERS file. We probably need new sections.
What would you put there?
r~
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH 16/27] accel/tcg: Unify cpu_{ld,st}*_{be,le}_mmu
2023-05-21 15:00 ` Richard Henderson
@ 2023-05-22 6:39 ` Philippe Mathieu-Daudé
2023-05-22 16:24 ` Richard Henderson
0 siblings, 1 reply; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2023-05-22 6:39 UTC (permalink / raw)
To: Richard Henderson, qemu-devel
Cc: qemu-arm, Mark Cave-Ayland, Artyom Tarasenko
On 21/5/23 17:00, Richard Henderson wrote:
> On 5/21/23 04:15, Philippe Mathieu-Daudé wrote:
>> Hi Richard,
>>
>> On 20/5/23 18:26, Richard Henderson wrote:
>>> With the current structure of cputlb.c, there is no difference
>>> between the little-endian and big-endian entry points, aside
>>> from the assert. Unify the pairs of functions.
>>>
>>> The only use of the functions with explicit endianness was in
>>> target/sparc64, and that was only to satisfy the assert.
>>
>> I'm having hard time to follow all the handling of the various
>> ASI definitions from target/sparc/asi.h. ...
>>
>>> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
>>> ---
>>> include/exec/cpu_ldst.h | 58 ++-----
>>> accel/tcg/cputlb.c | 122 +++-----------
>>> accel/tcg/user-exec.c | 322 ++++++++++--------------------------
>>> target/arm/tcg/m_helper.c | 4 +-
>>> target/sparc/ldst_helper.c | 18 +-
>>> accel/tcg/ldst_common.c.inc | 24 +--
>>> 6 files changed, 137 insertions(+), 411 deletions(-)
>>
>>
>>> diff --git a/target/sparc/ldst_helper.c b/target/sparc/ldst_helper.c
>>> index 7972d56a72..981a47d8bb 100644
>>> --- a/target/sparc/ldst_helper.c
>>> +++ b/target/sparc/ldst_helper.c
>>> @@ -1334,25 +1334,13 @@ uint64_t helper_ld_asi(CPUSPARCState *env,
>>> target_ulong addr,
>>
>>
>> Shouldn't we propagate the ASI endianness?
>
> Already done in translate, get_asi():
>
> /* The little-endian asis all have bit 3 set. */
> if (asi & 8) {
> memop ^= MO_BSWAP;
> }
Just in front of my eyes 🤦♂️ So:
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Maybe amend the commit description "The ASI endianness is
already taken care of in get_asi() ..."?
While looking at get_asi(), ASI_FL16_* cases overwrite
'memop', possibly discarding MO_ALIGN bit. Maybe this can't
happen.
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH 15/27] target/s390x: Use tcg_gen_qemu_{ld,st}_i128 for LPQ, STPQ
2023-05-20 16:26 ` [PATCH 15/27] target/s390x: Use tcg_gen_qemu_{ld, st}_i128 for LPQ, STPQ Richard Henderson
@ 2023-05-22 8:35 ` David Hildenbrand
2023-05-22 14:15 ` Richard Henderson
0 siblings, 1 reply; 46+ messages in thread
From: David Hildenbrand @ 2023-05-22 8:35 UTC (permalink / raw)
To: Richard Henderson, qemu-devel; +Cc: qemu-arm, qemu-s390x, Ilya Leoshkevich
On 20.05.23 18:26, Richard Henderson wrote:
> No need to roll our own, as this is now provided by tcg.
> This was the last use of retxl, so remove that too.
That's nice!
>
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
> Cc: qemu-s390x@nongnu.org
> Cc: David Hildenbrand <david@redhat.com>
> Cc: Ilya Leoshkevich <iii@linux.ibm.com>
> ---
[...]
> /* psw.addr */
> @@ -3130,15 +3125,9 @@ static DisasJumpType op_lpd(DisasContext *s, DisasOps *o)
>
> static DisasJumpType op_lpq(DisasContext *s, DisasOps *o)
> {
> - if (!(tb_cflags(s->base.tb) & CF_PARALLEL)) {
> - gen_helper_lpq(o->out, cpu_env, o->in2);
> - } else if (HAVE_ATOMIC128) {
> - gen_helper_lpq_parallel(o->out, cpu_env, o->in2);
> - } else {
> - gen_helper_exit_atomic(cpu_env);
> - return DISAS_NORETURN;
> - }
> - return_low128(o->out2);
> + o->out_128 = tcg_temp_new_i128();
> + tcg_gen_qemu_ld_i128(o->out_128, o->in2, get_mem_index(s),
> + MO_TE | MO_128 | MO_ALIGN);
> return DISAS_NEXT;
> }
>
> @@ -4533,14 +4522,11 @@ static DisasJumpType op_stmh(DisasContext *s, DisasOps *o)
>
> static DisasJumpType op_stpq(DisasContext *s, DisasOps *o)
> {
> - if (!(tb_cflags(s->base.tb) & CF_PARALLEL)) {
> - gen_helper_stpq(cpu_env, o->in2, o->out2, o->out);
> - } else if (HAVE_ATOMIC128) {
> - gen_helper_stpq_parallel(cpu_env, o->in2, o->out2, o->out);
> - } else {
> - gen_helper_exit_atomic(cpu_env);
> - return DISAS_NORETURN;
> - }
> + TCGv_i128 t16 = tcg_temp_new_i128();
> +
> + tcg_gen_concat_i64_i128(t16, o->out2, o->out);
> + tcg_gen_qemu_st_i128(t16, o->in2, get_mem_index(s),
> + MO_TE | MO_128 | MO_ALIGN);
I briefly glimpsed at tcg_gen_qemu_ld_i128_int (and
use_two_i64_for_i128()), does this really provide the atomic guarantees
we need in all cases?
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH 17/27] target/s390x: Use cpu_{ld,st}*_mmu in do_csst
2023-05-20 16:26 ` [PATCH 17/27] target/s390x: Use cpu_{ld,st}*_mmu in do_csst Richard Henderson
2023-05-21 11:21 ` Philippe Mathieu-Daudé
@ 2023-05-22 8:43 ` David Hildenbrand
1 sibling, 0 replies; 46+ messages in thread
From: David Hildenbrand @ 2023-05-22 8:43 UTC (permalink / raw)
To: Richard Henderson, qemu-devel; +Cc: qemu-arm, qemu-s390x, Ilya Leoshkevich
On 20.05.23 18:26, Richard Henderson wrote:
> Use cpu_ld16_mmu and cpu_st16_mmu to eliminate the special case,
> and change all of the *_data_ra functions to match.
>
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
> Cc: qemu-s390x@nongnu.org
> Cc: David Hildenbrand <david@redhat.com>
> Cc: Ilya Leoshkevich <iii@linux.ibm.com>
> ---
[...]
> /* Note that we asserted !parallel above. */
> @@ -1876,29 +1872,20 @@ static uint32_t do_csst(CPUS390XState *env, uint32_t r3, uint64_t a1,
> if (cc == 0) {
> switch (sc) {
> case 0:
> - cpu_stb_data_ra(env, a2, svh >> 56, ra);
> + cpu_stb_mmu(env, a2, svh >> 56, make_memop_idx(MO_8, mem_idx), ra);
> break;
> case 1:
> - cpu_stw_data_ra(env, a2, svh >> 48, ra);
> + cpu_stw_mmu(env, a2, svh >> 48,
> + make_memop_idx(MO_TE | MO_16, mem_idx), ra);
To make these two cases look less special, maybe just define oi1 and oi2
as well at the top?
LGTM
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH 18/27] target/s390x: Always use cpu_atomic_cmpxchgl_be_mmu in do_csst
2023-05-20 16:26 ` [PATCH 18/27] target/s390x: Always use cpu_atomic_cmpxchgl_be_mmu " Richard Henderson
@ 2023-05-22 8:44 ` David Hildenbrand
0 siblings, 0 replies; 46+ messages in thread
From: David Hildenbrand @ 2023-05-22 8:44 UTC (permalink / raw)
To: Richard Henderson, qemu-devel; +Cc: qemu-arm, qemu-s390x, Ilya Leoshkevich
On 20.05.23 18:26, Richard Henderson wrote:
> Eliminate the CONFIG_USER_ONLY specialization.
>
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
> Cc: qemu-s390x@nongnu.org
> Cc: David Hildenbrand <david@redhat.com>
> Cc: Ilya Leoshkevich <iii@linux.ibm.com>
> ---
Reviewed-by: David Hildenbrand <david@redhat.com>
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH 15/27] target/s390x: Use tcg_gen_qemu_{ld,st}_i128 for LPQ, STPQ
2023-05-22 8:35 ` [PATCH 15/27] target/s390x: Use tcg_gen_qemu_{ld,st}_i128 " David Hildenbrand
@ 2023-05-22 14:15 ` Richard Henderson
0 siblings, 0 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-22 14:15 UTC (permalink / raw)
To: David Hildenbrand, qemu-devel; +Cc: qemu-arm, qemu-s390x, Ilya Leoshkevich
On 5/22/23 01:35, David Hildenbrand wrote:
>> @@ -4533,14 +4522,11 @@ static DisasJumpType op_stmh(DisasContext *s, DisasOps *o)
>> static DisasJumpType op_stpq(DisasContext *s, DisasOps *o)
>> {
>> - if (!(tb_cflags(s->base.tb) & CF_PARALLEL)) {
>> - gen_helper_stpq(cpu_env, o->in2, o->out2, o->out);
>> - } else if (HAVE_ATOMIC128) {
>> - gen_helper_stpq_parallel(cpu_env, o->in2, o->out2, o->out);
>> - } else {
>> - gen_helper_exit_atomic(cpu_env);
>> - return DISAS_NORETURN;
>> - }
>> + TCGv_i128 t16 = tcg_temp_new_i128();
>> +
>> + tcg_gen_concat_i64_i128(t16, o->out2, o->out);
>> + tcg_gen_qemu_st_i128(t16, o->in2, get_mem_index(s),
>> + MO_TE | MO_128 | MO_ALIGN);
>
> I briefly glimpsed at tcg_gen_qemu_ld_i128_int (and use_two_i64_for_i128()), does this
> really provide the atomic guarantees we need in all cases?
Yes. The CF_PARALLEL check is the same as the one removed above.
r~
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH 16/27] accel/tcg: Unify cpu_{ld,st}*_{be,le}_mmu
2023-05-22 6:39 ` Philippe Mathieu-Daudé
@ 2023-05-22 16:24 ` Richard Henderson
0 siblings, 0 replies; 46+ messages in thread
From: Richard Henderson @ 2023-05-22 16:24 UTC (permalink / raw)
To: Philippe Mathieu-Daudé, qemu-devel
Cc: qemu-arm, Mark Cave-Ayland, Artyom Tarasenko
On 5/21/23 23:39, Philippe Mathieu-Daudé wrote:
> On 21/5/23 17:00, Richard Henderson wrote:
>> On 5/21/23 04:15, Philippe Mathieu-Daudé wrote:
>>> Hi Richard,
>>>
>>> On 20/5/23 18:26, Richard Henderson wrote:
>>>> With the current structure of cputlb.c, there is no difference
>>>> between the little-endian and big-endian entry points, aside
>>>> from the assert. Unify the pairs of functions.
>>>>
>>>> The only use of the functions with explicit endianness was in
>>>> target/sparc64, and that was only to satisfy the assert.
>>>
>>> I'm having hard time to follow all the handling of the various
>>> ASI definitions from target/sparc/asi.h. ...
>>>
>>>> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
>>>> ---
>>>> include/exec/cpu_ldst.h | 58 ++-----
>>>> accel/tcg/cputlb.c | 122 +++-----------
>>>> accel/tcg/user-exec.c | 322 ++++++++++--------------------------
>>>> target/arm/tcg/m_helper.c | 4 +-
>>>> target/sparc/ldst_helper.c | 18 +-
>>>> accel/tcg/ldst_common.c.inc | 24 +--
>>>> 6 files changed, 137 insertions(+), 411 deletions(-)
>>>
>>>
>>>> diff --git a/target/sparc/ldst_helper.c b/target/sparc/ldst_helper.c
>>>> index 7972d56a72..981a47d8bb 100644
>>>> --- a/target/sparc/ldst_helper.c
>>>> +++ b/target/sparc/ldst_helper.c
>>>> @@ -1334,25 +1334,13 @@ uint64_t helper_ld_asi(CPUSPARCState *env, target_ulong addr,
>>>
>>>
>>> Shouldn't we propagate the ASI endianness?
>>
>> Already done in translate, get_asi():
>>
>> /* The little-endian asis all have bit 3 set. */
>> if (asi & 8) {
>> memop ^= MO_BSWAP;
>> }
>
> Just in front of my eyes 🤦♂️ So:
>
> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
>
> Maybe amend the commit description "The ASI endianness is
> already taken care of in get_asi() ..."?
That's what I was trying to say with "only there to satisfy the assert". I have expanded
on that a bit.
> While looking at get_asi(), ASI_FL16_* cases overwrite
> 'memop', possibly discarding MO_ALIGN bit. Maybe this can't
> happen.
Ah, that does look like a bug in one of my recent conversions.
r~
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH 01/27] util: Introduce host-specific cpuinfo.h
2023-05-20 16:26 ` [PATCH 01/27] util: Introduce host-specific cpuinfo.h Richard Henderson
2023-05-21 10:47 ` Philippe Mathieu-Daudé
@ 2023-05-23 15:56 ` Alex Bennée
1 sibling, 0 replies; 46+ messages in thread
From: Alex Bennée @ 2023-05-23 15:56 UTC (permalink / raw)
To: Richard Henderson; +Cc: qemu-devel, Juan Quintela, qemu-arm
Richard Henderson <richard.henderson@linaro.org> writes:
> The entire contents of the header is host-specific, but the
> existence of such a header is not, which could prevent some
> host specific ifdefs at the top of the file for the include.
>
> Add host/include/{arch,generic} to the project arguments.
>
> Reviewed-by: Juan Quintela <quintela@redhat.com>
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [PATCH 02/27] util: Add cpuinfo-i386.c
2023-05-21 15:05 ` Richard Henderson
@ 2023-05-23 16:01 ` Alex Bennée
0 siblings, 0 replies; 46+ messages in thread
From: Alex Bennée @ 2023-05-23 16:01 UTC (permalink / raw)
To: Richard Henderson
Cc: Philippe Mathieu-Daudé, qemu-devel, Juan Quintela,
Paolo Bonzini, qemu-arm
Richard Henderson <richard.henderson@linaro.org> writes:
> On 5/21/23 04:28, Philippe Mathieu-Daudé wrote:
>> On 20/5/23 18:26, Richard Henderson wrote:
>>> Add cpuinfo.h for i386 and x86_64, and the initialization
>>> for that in util/. Populate that with a slightly altered
>>> copy of the tcg host probing code. Other uses of cpuid.h
>>> will be adjusted one patch at a time.
>>>
>>> Reviewed-by: Juan Quintela <quintela@redhat.com>
>>> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
>>> ---
>>> host/include/i386/host/cpuinfo.h | 38 ++++++++++++
>>> host/include/x86_64/host/cpuinfo.h | 1 +
>>> util/cpuinfo-i386.c | 97 ++++++++++++++++++++++++++++++
>>> util/meson.build | 4 ++
>>> 4 files changed, 140 insertions(+)
>>> create mode 100644 host/include/i386/host/cpuinfo.h
>>> create mode 100644 host/include/x86_64/host/cpuinfo.h
>>> create mode 100644 util/cpuinfo-i386.c
>> Missing F: entry in MAINTAINERS file. We probably need new sections.
>
> What would you put there?
Part of Guest CPU cores (TCG) I guess.
Anyway:
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
^ permalink raw reply [flat|nested] 46+ messages in thread
end of thread, other threads:[~2023-05-23 16:06 UTC | newest]
Thread overview: 46+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-05-20 16:26 [PATCH 00/27] accel/tcg: Improvements to atomic128.h Richard Henderson
2023-05-20 16:26 ` [PATCH 01/27] util: Introduce host-specific cpuinfo.h Richard Henderson
2023-05-21 10:47 ` Philippe Mathieu-Daudé
2023-05-23 15:56 ` Alex Bennée
2023-05-20 16:26 ` [PATCH 02/27] util: Add cpuinfo-i386.c Richard Henderson
2023-05-21 11:28 ` Philippe Mathieu-Daudé
2023-05-21 15:05 ` Richard Henderson
2023-05-23 16:01 ` Alex Bennée
2023-05-20 16:26 ` [PATCH 03/27] util: Add i386 CPUINFO_ATOMIC_VMOVDQU Richard Henderson
2023-05-20 16:26 ` [PATCH 04/27] tcg/i386: Use host/cpuinfo.h Richard Henderson
2023-05-20 16:26 ` [PATCH 05/27] util/bufferiszero: Use i386 host/cpuinfo.h Richard Henderson
2023-05-20 16:26 ` [PATCH 06/27] migration/xbzrle: Shuffle function order Richard Henderson
2023-05-20 16:26 ` [PATCH 07/27] migration/xbzrle: Use i386 host/cpuinfo.h Richard Henderson
2023-05-20 16:26 ` [PATCH 08/27] migration: Build migration_files once Richard Henderson
2023-05-20 16:26 ` [PATCH 09/27] util: Add cpuinfo-aarch64.c Richard Henderson
2023-05-20 16:26 ` [PATCH 10/27] include/host: Split out atomic128-cas.h Richard Henderson
2023-05-21 10:44 ` Philippe Mathieu-Daudé
2023-05-20 16:26 ` [PATCH 11/27] include/host: Split out atomic128-ldst.h Richard Henderson
2023-05-20 16:26 ` [PATCH 12/27] meson: Fix detect atomic128 support with optimization Richard Henderson
2023-05-21 10:54 ` Philippe Mathieu-Daudé
2023-05-20 16:26 ` [PATCH 13/27] include/qemu: Move CONFIG_ATOMIC128_OPT handling to atomic128.h Richard Henderson
2023-05-20 16:26 ` [PATCH 14/27] target/ppc: Use tcg_gen_qemu_{ld, st}_i128 for LQARX, LQ, STQ Richard Henderson
2023-05-20 16:26 ` [PATCH 15/27] target/s390x: Use tcg_gen_qemu_{ld, st}_i128 for LPQ, STPQ Richard Henderson
2023-05-22 8:35 ` [PATCH 15/27] target/s390x: Use tcg_gen_qemu_{ld,st}_i128 " David Hildenbrand
2023-05-22 14:15 ` Richard Henderson
2023-05-20 16:26 ` [PATCH 16/27] accel/tcg: Unify cpu_{ld,st}*_{be,le}_mmu Richard Henderson
2023-05-21 11:15 ` Philippe Mathieu-Daudé
2023-05-21 15:00 ` Richard Henderson
2023-05-22 6:39 ` Philippe Mathieu-Daudé
2023-05-22 16:24 ` Richard Henderson
2023-05-20 16:26 ` [PATCH 17/27] target/s390x: Use cpu_{ld,st}*_mmu in do_csst Richard Henderson
2023-05-21 11:21 ` Philippe Mathieu-Daudé
2023-05-21 15:01 ` Richard Henderson
2023-05-22 8:43 ` David Hildenbrand
2023-05-20 16:26 ` [PATCH 18/27] target/s390x: Always use cpu_atomic_cmpxchgl_be_mmu " Richard Henderson
2023-05-22 8:44 ` David Hildenbrand
2023-05-20 16:26 ` [PATCH 19/27] accel/tcg: Remove cpu_atomic_{ld,st}o_*_mmu Richard Henderson
2023-05-20 16:26 ` [PATCH 20/27] accel/tcg: Remove prot argument to atomic_mmu_lookup Richard Henderson
2023-05-20 16:26 ` [PATCH 21/27] accel/tcg: Eliminate #if on HAVE_ATOMIC128 and HAVE_CMPXCHG128 Richard Henderson
2023-05-20 16:26 ` [PATCH 22/27] qemu/atomic128: Split atomic16_read Richard Henderson
2023-05-20 16:26 ` [PATCH 23/27] accel/tcg: Correctly use atomic128.h in ldst_atomicity.c.inc Richard Henderson
2023-05-20 16:26 ` [PATCH 24/27] tcg: Split out tcg/debug-assert.h Richard Henderson
2023-05-21 11:25 ` Philippe Mathieu-Daudé
2023-05-20 16:26 ` [PATCH 25/27] qemu/atomic128: Improve cmpxchg fallback for atomic16_set Richard Henderson
2023-05-20 16:26 ` [PATCH 26/27] qemu/atomic128: Add runtime test for FEAT_LSE2 Richard Henderson
2023-05-20 16:26 ` [PATCH 27/27] qemu/atomic128: Add x86_64 atomic128-ldst.h Richard Henderson
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).