[PATCH v2 00/50] Introduce helper-to-tcg

[PATCH v2 00/50] Introduce helper-to-tcg

[Anton Johansson via qemu development]

Hi all, this patchset introduces helper-to-tcg, a LLVM based build-time
C to TCG translator, as a QEMU subproject.  The purpose of this tool is
to simplify implementation of instructions in TCG by automatically
translating helper functions for a given target to TCG.  It may also be
used as a standalone tool for getting a base TCG implementation for
complicated instructions.

See KVM forum 2023 and 2025 presentations which give an overview of its
application to Hexagon and RISCV: 

  - https://www.youtube.com/watch?v=Gwz0kp7IZPE
  - https://www.youtube.com/watch?v=7qIwraM3-4I

helper-to-tcg is also applied to the Hexagon frontend, managing to
translate 1280 instructions, 162 of which are HVX instructions.  In the current
patchset we translate 1240 instructions due to limiting emission of the heavier
vectorized instructions which requires larger TBs and more vector temporaries.
For the time being, idef-parser remains translating 281 instructions consisting
mostly of complicated load instructions.  This count will be reduced
over time until idef-parser can be deprecated.

As an example, consider the following helper function implementation of
a Hexagon instruction for performing a 2-element scalar product, using
signed saturated arithmetic

  void HELPER(V6_vdmpyhvsat)(CPUHexagonState *env,
                             void * restrict VdV_void,
                             void * restrict VuV_void,
                             void * restrict VvV_void)
  {
      fVFOREACH(32, i) {
          size8s_t accum = fMPY16SS(fGETHALF(0,VuV.w[i]),fGETHALF(0, VvV.w[i]));
          accum += fMPY16SS(fGETHALF(1,VuV.w[i]),fGETHALF(1, VvV.w[i]));
          VdV.w[i] = fVSATW(accum);
      }
  }

which at the end of the helper-to-tcg pipeline will have been converted
to the following LLVM IR

  define void @helper_V6_vdmpyhvsat(%struct.CPUArchState* %0,
                                    i8* %1, i8* %2, i8* %3) {
      %4 = bitcast i8* %2 to <32 x i32>*
      %wide.load = load <32 x i32>, <32 x i32>* %4
      %5 = call <32 x i32> @VecShlScalar(<32 x i32> %wide.load, i32 16)
      %6 = call <32 x i32> @VecAShrScalar(<32 x i32> %5, i32 16)
      %7 = bitcast i8* %3 to <32 x i32>*
      %wide.load23 = load <32 x i32>, <32 x i32>* %7
      %8 = call <32 x i32> @VecShlScalar(<32 x i32> %wide.load23, i32 16)
      %9 = call <32 x i32> @VecAShrScalar(<32 x i32> %8, i32 16)
      %10 = mul nsw <32 x i32> %9, %6
      %11 = call <32 x i32> @VecAShrScalar(<32 x i32> %wide.load, i32 16)
      %12 = call <32 x i32> @VecAShrScalar(<32 x i32> %wide.load23, i32 16)
      %13 = mul nsw <32 x i32> %12, %11
      %14 = bitcast i8* %1 to <32 x i32>*
      ret void
  }

which, in TCG, gets emitted as

  void emit_V6_vdmpyhvsat(TCGv_env env, intptr_t vec3,
                          intptr_t vec7, intptr_t vec6) {
      VectorMem mem = {0};
      intptr_t vec0 = temp_new_gvec(&mem, 128);
      tcg_gen_gvec_shli(MO_32, vec0, vec7, 16, 128, 128);
      intptr_t vec5 = temp_new_gvec(&mem, 128);
      tcg_gen_gvec_sari(MO_32, vec5, vec0, 16, 128, 128);
      intptr_t vec1 = temp_new_gvec(&mem, 128);
      tcg_gen_gvec_shli(MO_32, vec1, vec6, 16, 128, 128);
      tcg_gen_gvec_sari(MO_32, vec1, vec1, 16, 128, 128);
      tcg_gen_gvec_mul(MO_32, vec1, vec1, vec5, 128, 128);
      intptr_t vec2 = temp_new_gvec(&mem, 128);
      tcg_gen_gvec_sari(MO_32, vec2, vec7, 16, 128, 128);
      tcg_gen_gvec_sari(MO_32, vec0, vec6, 16, 128, 128);
      tcg_gen_gvec_mul(MO_32, vec2, vec0, vec2, 128, 128);
      tcg_gen_gvec_ssadd(MO_32, vec3, vec1, vec2, 128, 128);
  }

consisting of a few vectorized shifts, multiplications, and a signed
saturated add.

For a more in-depth usage guide see `subprojects/helper-to-tcg/README.md`.

Limitations:
  - Does not handle functions with multiple return values. On Hexagon,
    a large set of instructions still translated by idef-parser fall
    into this category.

  - Can emit complicated vector operations that are not the most
    host-friendly, identifying and emitting custom GVec patterns would
    be interesting.

Patchset overview:
  1. helper-to-tcg (patches 6-40) - Introduces the actual translator as
     a QEMU subproject.

  2. Fills gaps in TCG instructions (patches 1,2,5) - Since the tool is
     LLVM based it allows for translation of vector instructions to gvec
     instructions in Tinycode. This requires the introduction of a few
     new tcg_gen_gvec_*() functions for dealing with sign- and
     zero-extension, along with a function for initializing a vector to
     a constant, and functions for bitreversal and funnel shift.

  3. Mapping of cpu state (patch 3) - helper-to-tcg needs information
     about the offsets of fields in the cpu state that correspond to TCG
     globals, so these can be emitted in the output code.  For this
     purpose, a target may define an array of `struct cpu_tcg_mapping`
     to map fields in the cpu state to TCG globals in a declarative way.
     This global array can be parsed by helper-to-tcg, and replaces
     manually calling tcg_global_mem_new*() in frontend.

  4. Increases max size of generated TB code (patch 4) - Due to the
     power of the LLVM auto-vectorizer, helper-to-tcg can emit quite
     complicated vectorized gvec code.  Particularly for Hexagon where a
     single instruction packet can consist of multiple vector
     instructions.  A single instruction packet can in rare cases exceed
     the TB buffer size of 128 longs.

  5. Applies helper-to-tcg to Hexagon (patches 41-50) - helper-to-tcg is
     used on the Hexagon frontend to translate a majority of helper
     functions in place of idef-parser.  For the time being idef-parser
     will remain in use to translate instructions with multiple return
     values that are not representable as helper functions and therefore
     translatable with helper-to-tcg.

Changes in v2:
  - Use QEMU_ANNOTATE over introducing LLVM_ANNOTATE (Philippe);
  - Move .h -> .hpp to avoid checkpatch (Philppe);
  - Meson changes (Paolo);
  - Gvec size changing operations (zext/sext/trunc) along with constant
    vector creation has been moved from general gvec operations to being
    emitted by helper-to-tcg on demand. A simple interface for specifying
    the target vector layout is available through commandline
    options (Richard);
  - Constant vectors initialized from arrays now vectorized;
  - Introduced a model of target vector layouts;
  - Shifted LLVM version support from 10-14 to 15-21;
  - Improved end-to-end testing, now verifies output across LLVM versions,
    currently these tests along with their expected output serve as usage
    examples;
  - Added docker tests for supported LLVM versions;
  - Dropped the tcg_gen_callN dispatcher in favour of emitting gen_helper_*()
    definitions directly;
  - Optionally allow calls to declarations, custom jump instructions, and propagation of
    DisasContext to better slot into existing frontends;
  - Limit vector usage for hexagon to 8 temporaries and 16 gvec
    instructions, also use existing `tmp_VRegs[]` memory.
  - Many cleanups and bugfixes.

Anton Johansson (50):
  accel/tcg: Add bitreverse and funnel-shift runtime helper functions
  accel/tcg: Add getpc helper
  tcg: Introduce tcg-global-mappings
  tcg: Increase maximum TB size
  tcg: Expose tcg_gen_ussub_sat()
  Add helper-to-tcg subproject
  helper-to-tcg: Introduce get-llvm-ir.py
  helper-to-tcg: Handle LLVM version compatibility
  helper-to-tcg: Introduce custom LLVM pipeline
  helper-to-tcg: Add pipeline --debug and --debug-only
  helper-to-tcg: Add simple error creation helper
  helper-to-tcg: Introduce PrepareForOptPass
  helper-to-tcg: PrepareForOptPass, demangle function names
  helper-to-tcg: PrepareForOptPass, map annotations
  helper-to-tcg: PrepareForOptPass, cull unused functions
  helper-to-tcg: PrepareForOptPass, undef llvm.returnaddress
  helper-to-tcg: PrepareForOptPass, fixup inline attributes
  helper-to-tcg: PrepareForOptPass, collect debuginfo
  helper-to-tcg: Pipeline, run optimization pass
  helper-to-tcg: Introduce pseudo instructions
  helper-to-tcg: Add guest vector layout description
  helper-to-tcg: Introduce PrepareForTcgPass
  helper-to-tcg: PrepareForTcgPass, remove functions with cycles
  helper-to-tcg: PrepareForTcgPass, demote PHI nodes
  helper-to-tcg: PrepareForTcgPass, map TCG globals
  helper-to-tcg: PrepareForTcgPass, transform GEPs
  helper-to-tcg: PrepareForTcgPass, canonicalize IR
  helper-to-tcg: PrepareForTcgPass, identity map trivial expressions
  helper-to-tcg: Introduce TcgV structure
  helper-to-tcg: Introduce TcgGenPass
  helper-to-tcg: TcgGenPass, linearize basic blocks
  helper-to-tcg: TcgGenPass, introduce Value <-> TcgV map
  helper-to-tcg: TcgGenPass, add structs for string emission
  helper-to-tcg: TcgGenPass, map arguments to TCG
  helper-to-tcg: TcgGenPass, propagate constant expresssions
  helper-to-tcg: TcgGenPass, allocate TCG registers
  helper-to-tcg: TcgGenPass, emit TCG strings
  helper-to-tcg: Add README
  helper-to-tcg: Add end-to-end tests
  test: helper-to-tcg docker tests
  target/hexagon: Add get_tb_mmu_index()
  target/hexagon: Increase VECTOR_TEMPS_MAX
  target/hexagon: Provide env to tcg global mapping
  target/hexagon: Keep gen_slotval/check_noshuf for helper-to-tcg
  target/hexagon: Emit annotations for helpers
  target/hexagon: Split probe_and_commit helper
  target/hexagon: Use helper-to-tcg helper calls
  target/hexagon: Manually call generated HVX instructions
  target/hexagon: Use idef-parser as a fallback
  target/hexagon: Use helper-to-tcg

 accel/tcg/tcg-runtime.c                       |   33 +
 accel/tcg/tcg-runtime.h                       |    8 +
 configs/targets/hexagon-linux-user.mak        |    1 +
 configs/targets/hexagon-softmmu.mak           |    1 +
 include/tcg/tcg-global-mappings.h             |  118 ++
 include/tcg/tcg-op-common.h                   |    2 +
 include/tcg/tcg.h                             |    2 +-
 meson.build                                   |   10 +
 meson_options.txt                             |    2 +
 scripts/meson-buildoptions.sh                 |    5 +
 subprojects/helper-to-tcg/README.md           |  297 ++++
 subprojects/helper-to-tcg/get-llvm-ir.py      |  145 ++
 .../helper-to-tcg/include/CmdLineOptions.hpp  |   46 +
 .../helper-to-tcg/include/DebugInfo.hpp       |   46 +
 subprojects/helper-to-tcg/include/Error.hpp   |   38 +
 .../include/FunctionAnnotation.hpp            |  104 ++
 .../helper-to-tcg/include/LlvmCompat.hpp      |  124 ++
 .../include/PrepareForOptPass.hpp             |   42 +
 .../include/PrepareForTcgPass.hpp             |   37 +
 .../helper-to-tcg/include/PseudoInst.hpp      |   75 +
 .../helper-to-tcg/include/PseudoInst.inc      |   78 ++
 .../helper-to-tcg/include/TcgGenPass.hpp      |   63 +
 .../helper-to-tcg/include/TcgGlobalMap.hpp    |   40 +
 subprojects/helper-to-tcg/include/TcgType.hpp |  276 ++++
 .../helper-to-tcg/include/VectorLayout.hpp    |  116 ++
 subprojects/helper-to-tcg/meson.build         |   86 ++
 subprojects/helper-to-tcg/meson_options.txt   |    2 +
 subprojects/helper-to-tcg/src/LlvmCompat.cpp  |   89 ++
 subprojects/helper-to-tcg/src/Pipeline.cpp    |  399 ++++++
 .../PrepareForOptPass/PrepareForOptPass.cpp   |  363 +++++
 .../src/PrepareForTcgPass/CanonicalizeIR.cpp  | 1193 ++++++++++++++++
 .../src/PrepareForTcgPass/CanonicalizeIR.hpp  |   24 +
 .../src/PrepareForTcgPass/IdentityMap.cpp     |   91 ++
 .../src/PrepareForTcgPass/IdentityMap.hpp     |   39 +
 .../PrepareForTcgPass/PrepareForTcgPass.cpp   |  150 ++
 .../src/PrepareForTcgPass/TransformGEPs.cpp   |  355 +++++
 .../src/PrepareForTcgPass/TransformGEPs.hpp   |   45 +
 subprojects/helper-to-tcg/src/PseudoInst.cpp  |  183 +++
 .../src/TcgGenPass/LinearizeBlocks.cpp        |  140 ++
 .../src/TcgGenPass/LinearizeBlocks.hpp        |   32 +
 .../src/TcgGenPass/MapArguments.cpp           |  122 ++
 .../src/TcgGenPass/MapConstantExpressions.cpp |  328 +++++
 .../src/TcgGenPass/MapTcgOperations.cpp       |  988 ++++++++++++++
 .../src/TcgGenPass/MapTemporaries.cpp         |  619 +++++++++
 .../helper-to-tcg/src/TcgGenPass/TcgEmit.cpp  | 1203 +++++++++++++++++
 .../helper-to-tcg/src/TcgGenPass/TcgEmit.hpp  |  315 +++++
 .../src/TcgGenPass/TcgGenPass.cpp             |  385 ++++++
 .../src/TcgGenPass/ValueMapping.hpp           |  110 ++
 .../helper-to-tcg/tests/call-to-declaration.c |   33 +
 .../helper-to-tcg/tests/call-translation.c    |   35 +
 subprojects/helper-to-tcg/tests/cpustate.c    |   44 +
 .../helper-to-tcg/tests/forward-context.c     |   21 +
 subprojects/helper-to-tcg/tests/ldst.c        |   17 +
 subprojects/helper-to-tcg/tests/meson.build   |  121 ++
 .../tests/ref/15/call-to-declaration.c        |   26 +
 .../tests/ref/15/call-translation.c           |   28 +
 .../helper-to-tcg/tests/ref/15/cpustate.c     |   36 +
 .../tests/ref/15/forward-context.c            |   30 +
 subprojects/helper-to-tcg/tests/ref/15/ldst.c |   21 +
 .../helper-to-tcg/tests/ref/15/scalar.c       |   21 +
 .../tests/ref/15/user-pcrel-jump.c            |   33 +
 .../tests/ref/15/vector-layout-f16L.c         |  527 ++++++++
 .../tests/ref/15/vector-layout-f16M.c         |  527 ++++++++
 .../tests/ref/15/vector-layout-f32L.c         |  527 ++++++++
 .../tests/ref/15/vector-layout-f32M.c         |  527 ++++++++
 .../tests/ref/15/vector-layout-f64L.c         |  527 ++++++++
 .../tests/ref/15/vector-layout-f64M.c         |  527 ++++++++
 .../tests/ref/15/vector-layout-f8L.c          |  527 ++++++++
 .../tests/ref/15/vector-layout-f8M.c          |  527 ++++++++
 .../tests/ref/15/vector-layout-t16L.c         |  527 ++++++++
 .../tests/ref/15/vector-layout-t16M.c         |  527 ++++++++
 .../tests/ref/15/vector-layout-t32L.c         |  527 ++++++++
 .../tests/ref/15/vector-layout-t32M.c         |  527 ++++++++
 .../tests/ref/15/vector-layout-t64L.c         |  527 ++++++++
 .../tests/ref/15/vector-layout-t64M.c         |  527 ++++++++
 .../tests/ref/15/vector-layout-t8L.c          |  527 ++++++++
 .../tests/ref/15/vector-layout-t8M.c          |  527 ++++++++
 .../helper-to-tcg/tests/ref/15/vector.c       |   31 +
 .../tests/ref/21/call-to-declaration.c        |   26 +
 .../tests/ref/21/call-translation.c           |   28 +
 .../helper-to-tcg/tests/ref/21/cpustate.c     |   36 +
 .../tests/ref/21/forward-context.c            |   30 +
 subprojects/helper-to-tcg/tests/ref/21/ldst.c |   21 +
 .../helper-to-tcg/tests/ref/21/scalar.c       |   21 +
 .../tests/ref/21/user-pcrel-jump.c            |   33 +
 .../tests/ref/21/vector-layout-f16L.c         |  527 ++++++++
 .../tests/ref/21/vector-layout-f16M.c         |  527 ++++++++
 .../tests/ref/21/vector-layout-f32L.c         |  527 ++++++++
 .../tests/ref/21/vector-layout-f32M.c         |  527 ++++++++
 .../tests/ref/21/vector-layout-f64L.c         |  527 ++++++++
 .../tests/ref/21/vector-layout-f64M.c         |  527 ++++++++
 .../tests/ref/21/vector-layout-f8L.c          |  527 ++++++++
 .../tests/ref/21/vector-layout-f8M.c          |  527 ++++++++
 .../tests/ref/21/vector-layout-t16L.c         |  527 ++++++++
 .../tests/ref/21/vector-layout-t16M.c         |  527 ++++++++
 .../tests/ref/21/vector-layout-t32L.c         |  527 ++++++++
 .../tests/ref/21/vector-layout-t32M.c         |  527 ++++++++
 .../tests/ref/21/vector-layout-t64L.c         |  527 ++++++++
 .../tests/ref/21/vector-layout-t64M.c         |  527 ++++++++
 .../tests/ref/21/vector-layout-t8L.c          |  527 ++++++++
 .../tests/ref/21/vector-layout-t8M.c          |  527 ++++++++
 .../helper-to-tcg/tests/ref/21/vector.c       |   31 +
 subprojects/helper-to-tcg/tests/scalar.c      |   15 +
 .../helper-to-tcg/tests/tcg-global-mappings.h |  118 ++
 .../helper-to-tcg/tests/user-pcrel-jump.c     |   28 +
 .../helper-to-tcg/tests/vector-layout.c       |   51 +
 subprojects/helper-to-tcg/tests/vector.c      |   34 +
 target/hexagon/cpu.h                          |    6 +-
 target/hexagon/gen_helper_funcs.py            |   17 +-
 target/hexagon/gen_helper_protos.py           |    2 +-
 target/hexagon/gen_idef_parser_funcs.py       |    9 +
 target/hexagon/gen_tcg_funcs.py               |   16 +-
 target/hexagon/genptr.c                       |    6 +-
 target/hexagon/helper.h                       |    9 +
 target/hexagon/hex_common.py                  |  103 +-
 target/hexagon/meson.build                    |  148 +-
 target/hexagon/op_helper.c                    |   17 +-
 target/hexagon/translate.c                    |  139 +-
 target/hexagon/translate.h                    |    4 +
 tcg/meson.build                               |    1 +
 tcg/tcg-global-mappings.c                     |   62 +
 tcg/tcg-op-gvec.c                             |    4 +-
 tests/docker/dockerfiles/debian-llvm.docker   |   38 +
 tests/docker/test-helper-to-tcg               |   41 +
 124 files changed, 27303 insertions(+), 122 deletions(-)
 create mode 100644 include/tcg/tcg-global-mappings.h
 create mode 100644 subprojects/helper-to-tcg/README.md
 create mode 100755 subprojects/helper-to-tcg/get-llvm-ir.py
 create mode 100644 subprojects/helper-to-tcg/include/CmdLineOptions.hpp
 create mode 100644 subprojects/helper-to-tcg/include/DebugInfo.hpp
 create mode 100644 subprojects/helper-to-tcg/include/Error.hpp
 create mode 100644 subprojects/helper-to-tcg/include/FunctionAnnotation.hpp
 create mode 100644 subprojects/helper-to-tcg/include/LlvmCompat.hpp
 create mode 100644 subprojects/helper-to-tcg/include/PrepareForOptPass.hpp
 create mode 100644 subprojects/helper-to-tcg/include/PrepareForTcgPass.hpp
 create mode 100644 subprojects/helper-to-tcg/include/PseudoInst.hpp
 create mode 100644 subprojects/helper-to-tcg/include/PseudoInst.inc
 create mode 100644 subprojects/helper-to-tcg/include/TcgGenPass.hpp
 create mode 100644 subprojects/helper-to-tcg/include/TcgGlobalMap.hpp
 create mode 100644 subprojects/helper-to-tcg/include/TcgType.hpp
 create mode 100644 subprojects/helper-to-tcg/include/VectorLayout.hpp
 create mode 100644 subprojects/helper-to-tcg/meson.build
 create mode 100644 subprojects/helper-to-tcg/meson_options.txt
 create mode 100644 subprojects/helper-to-tcg/src/LlvmCompat.cpp
 create mode 100644 subprojects/helper-to-tcg/src/Pipeline.cpp
 create mode 100644 subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp
 create mode 100644 subprojects/helper-to-tcg/src/PrepareForTcgPass/CanonicalizeIR.cpp
 create mode 100644 subprojects/helper-to-tcg/src/PrepareForTcgPass/CanonicalizeIR.hpp
 create mode 100644 subprojects/helper-to-tcg/src/PrepareForTcgPass/IdentityMap.cpp
 create mode 100644 subprojects/helper-to-tcg/src/PrepareForTcgPass/IdentityMap.hpp
 create mode 100644 subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp
 create mode 100644 subprojects/helper-to-tcg/src/PrepareForTcgPass/TransformGEPs.cpp
 create mode 100644 subprojects/helper-to-tcg/src/PrepareForTcgPass/TransformGEPs.hpp
 create mode 100644 subprojects/helper-to-tcg/src/PseudoInst.cpp
 create mode 100644 subprojects/helper-to-tcg/src/TcgGenPass/LinearizeBlocks.cpp
 create mode 100644 subprojects/helper-to-tcg/src/TcgGenPass/LinearizeBlocks.hpp
 create mode 100644 subprojects/helper-to-tcg/src/TcgGenPass/MapArguments.cpp
 create mode 100644 subprojects/helper-to-tcg/src/TcgGenPass/MapConstantExpressions.cpp
 create mode 100644 subprojects/helper-to-tcg/src/TcgGenPass/MapTcgOperations.cpp
 create mode 100644 subprojects/helper-to-tcg/src/TcgGenPass/MapTemporaries.cpp
 create mode 100644 subprojects/helper-to-tcg/src/TcgGenPass/TcgEmit.cpp
 create mode 100644 subprojects/helper-to-tcg/src/TcgGenPass/TcgEmit.hpp
 create mode 100644 subprojects/helper-to-tcg/src/TcgGenPass/TcgGenPass.cpp
 create mode 100644 subprojects/helper-to-tcg/src/TcgGenPass/ValueMapping.hpp
 create mode 100644 subprojects/helper-to-tcg/tests/call-to-declaration.c
 create mode 100644 subprojects/helper-to-tcg/tests/call-translation.c
 create mode 100644 subprojects/helper-to-tcg/tests/cpustate.c
 create mode 100644 subprojects/helper-to-tcg/tests/forward-context.c
 create mode 100644 subprojects/helper-to-tcg/tests/ldst.c
 create mode 100644 subprojects/helper-to-tcg/tests/meson.build
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/call-to-declaration.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/call-translation.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/cpustate.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/forward-context.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/ldst.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/scalar.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/user-pcrel-jump.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/vector-layout-f16L.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/vector-layout-f16M.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/vector-layout-f32L.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/vector-layout-f32M.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/vector-layout-f64L.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/vector-layout-f64M.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/vector-layout-f8L.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/vector-layout-f8M.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/vector-layout-t16L.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/vector-layout-t16M.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/vector-layout-t32L.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/vector-layout-t32M.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/vector-layout-t64L.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/vector-layout-t64M.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/vector-layout-t8L.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/vector-layout-t8M.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/15/vector.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/call-to-declaration.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/call-translation.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/cpustate.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/forward-context.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/ldst.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/scalar.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/user-pcrel-jump.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/vector-layout-f16L.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/vector-layout-f16M.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/vector-layout-f32L.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/vector-layout-f32M.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/vector-layout-f64L.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/vector-layout-f64M.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/vector-layout-f8L.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/vector-layout-f8M.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/vector-layout-t16L.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/vector-layout-t16M.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/vector-layout-t32L.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/vector-layout-t32M.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/vector-layout-t64L.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/vector-layout-t64M.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/vector-layout-t8L.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/vector-layout-t8M.c
 create mode 100644 subprojects/helper-to-tcg/tests/ref/21/vector.c
 create mode 100644 subprojects/helper-to-tcg/tests/scalar.c
 create mode 100644 subprojects/helper-to-tcg/tests/tcg-global-mappings.h
 create mode 100644 subprojects/helper-to-tcg/tests/user-pcrel-jump.c
 create mode 100644 subprojects/helper-to-tcg/tests/vector-layout.c
 create mode 100644 subprojects/helper-to-tcg/tests/vector.c
 create mode 100644 tcg/tcg-global-mappings.c
 create mode 100644 tests/docker/dockerfiles/debian-llvm.docker
 create mode 100755 tests/docker/test-helper-to-tcg

-- 
2.52.0

[PATCH v2 01/50] accel/tcg: Add bitreverse and funnel-shift runtime helper functions

[Anton Johansson via qemu development]

Adds necessary helper functions for mapping LLVM IR onto TCG.
Specifically, helpers corresponding to the bitreverse and funnel-shift
intrinsics in LLVM.

Note: these may be converted to more efficient implementations in the
future, but for the time being it allows helper-to-tcg to support a
wider subset of LLVM IR.

Signed-off-by: Anton Johansson <anjo@rev.ng>
--
TODO: `tcg_gen_*()` variants will be added in the next version, it
slipped through (Richard).
---
 accel/tcg/tcg-runtime.c | 28 ++++++++++++++++++++++++++++
 accel/tcg/tcg-runtime.h |  6 ++++++
 2 files changed, 34 insertions(+)

diff --git a/accel/tcg/tcg-runtime.c b/accel/tcg/tcg-runtime.c
index 7c0aab98a8..d42c00736b 100644
--- a/accel/tcg/tcg-runtime.c
+++ b/accel/tcg/tcg-runtime.c
@@ -24,6 +24,7 @@
 #include "qemu/osdep.h"
 #include "qemu/host-utils.h"
 #include "exec/cpu-common.h"
+#include "qemu/int128.h"
 #include "exec/helper-proto-common.h"
 #include "accel/tcg/cpu-loop.h"
 #include "accel/tcg/getpc.h"
@@ -54,8 +55,30 @@ uint32_t HELPER(remu_i32)(uint32_t arg1, uint32_t arg2)
     return arg1 % arg2;
 }
 
+uint32_t HELPER(bitreverse8_i32)(uint32_t x)
+{
+  return revbit8((uint8_t) x);
+}
+
+uint32_t HELPER(bitreverse16_i32)(uint32_t x)
+{
+  return revbit16((uint16_t) x);
+}
+
+uint32_t HELPER(bitreverse32_i32)(uint32_t x)
+{
+  return revbit32(x);
+}
+
 /* 64-bit helpers */
 
+uint64_t HELPER(fshl_i64)(uint64_t a, uint64_t b, uint64_t c)
+{
+    Int128 d = int128_make128(b, a);
+    Int128 shift = int128_lshift(d, c);
+    return int128_gethi(shift);
+}
+
 int64_t HELPER(div_i64)(int64_t arg1, int64_t arg2)
 {
     return arg1 / arg2;
@@ -76,6 +99,11 @@ uint64_t HELPER(remu_i64)(uint64_t arg1, uint64_t arg2)
     return arg1 % arg2;
 }
 
+uint64_t HELPER(bitreverse64_i64)(uint64_t x)
+{
+    return revbit64(x);
+}
+
 uint64_t HELPER(muluh_i64)(uint64_t arg1, uint64_t arg2)
 {
     uint64_t l, h;
diff --git a/accel/tcg/tcg-runtime.h b/accel/tcg/tcg-runtime.h
index 0b832176b3..197d658b40 100644
--- a/accel/tcg/tcg-runtime.h
+++ b/accel/tcg/tcg-runtime.h
@@ -2,11 +2,17 @@ DEF_HELPER_FLAGS_2(div_i32, TCG_CALL_NO_RWG_SE, s32, s32, s32)
 DEF_HELPER_FLAGS_2(rem_i32, TCG_CALL_NO_RWG_SE, s32, s32, s32)
 DEF_HELPER_FLAGS_2(divu_i32, TCG_CALL_NO_RWG_SE, i32, i32, i32)
 DEF_HELPER_FLAGS_2(remu_i32, TCG_CALL_NO_RWG_SE, i32, i32, i32)
+DEF_HELPER_FLAGS_1(bitreverse8_i32,  TCG_CALL_NO_RWG_SE, i32, i32)
+DEF_HELPER_FLAGS_1(bitreverse16_i32, TCG_CALL_NO_RWG_SE, i32, i32)
+DEF_HELPER_FLAGS_1(bitreverse32_i32, TCG_CALL_NO_RWG_SE, i32, i32)
 
 DEF_HELPER_FLAGS_2(div_i64, TCG_CALL_NO_RWG_SE, s64, s64, s64)
 DEF_HELPER_FLAGS_2(rem_i64, TCG_CALL_NO_RWG_SE, s64, s64, s64)
 DEF_HELPER_FLAGS_2(divu_i64, TCG_CALL_NO_RWG_SE, i64, i64, i64)
 DEF_HELPER_FLAGS_2(remu_i64, TCG_CALL_NO_RWG_SE, i64, i64, i64)
+DEF_HELPER_FLAGS_1(bitreverse64_i64, TCG_CALL_NO_RWG_SE, i64, i64)
+
+DEF_HELPER_FLAGS_3(fshl_i64, TCG_CALL_NO_RWG_SE, i64, i64, i64, i64)
 
 DEF_HELPER_FLAGS_2(mulsh_i64, TCG_CALL_NO_RWG_SE, s64, s64, s64)
 DEF_HELPER_FLAGS_2(muluh_i64, TCG_CALL_NO_RWG_SE, i64, i64, i64)
-- 
2.52.0

[PATCH v2 02/50] accel/tcg: Add getpc helper

[Anton Johansson via qemu development]

Introduces a helper function to return the current pc (returnaddress of
the helper), this allows helper-to-tcg to correctly translate nested
helper functions that checks for faulting memory operations.

This is useful as an optimization where a helper functions has a
commonly taken fast path that doesn't fault e.g.

  void HELPER(outer)(...)
  {
      ... // non faulting operations
      if (some_uncommon_condition) {
          helper_inner(..., GETPC())
      }
  }

the outer helper along with the condition can then be emitted as TCG,
and helper_inner() will be emitted as a gen_helper_inner(, ra), where ra
is the result of gen_helper_getpc().

NOTE: This is not ideal since we're introducing extra helper calls,
      and the solution doesn't deal with the "inner" function not being
      a helper.  A better solution and what we'll probably do in the
      next version of the patchset is to instead emit a helper
      definition for the "inner" function that uses GETPC().

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 accel/tcg/tcg-runtime.c | 5 +++++
 accel/tcg/tcg-runtime.h | 2 ++
 2 files changed, 7 insertions(+)

diff --git a/accel/tcg/tcg-runtime.c b/accel/tcg/tcg-runtime.c
index d42c00736b..87b9530a50 100644
--- a/accel/tcg/tcg-runtime.c
+++ b/accel/tcg/tcg-runtime.c
@@ -162,3 +162,8 @@ void HELPER(exit_atomic)(CPUArchState *env)
 {
     cpu_loop_exit_atomic(env_cpu(env), GETPC());
 }
+
+uint64_t HELPER(getpc)(void)
+{
+    return GETPC();
+}
diff --git a/accel/tcg/tcg-runtime.h b/accel/tcg/tcg-runtime.h
index 197d658b40..e25712d1fc 100644
--- a/accel/tcg/tcg-runtime.h
+++ b/accel/tcg/tcg-runtime.h
@@ -30,6 +30,8 @@ DEF_HELPER_FLAGS_1(lookup_tb_ptr, TCG_CALL_NO_WG_SE, cptr, env)
 
 DEF_HELPER_FLAGS_1(exit_atomic, TCG_CALL_NO_WG, noreturn, env)
 
+DEF_HELPER_FLAGS_0(getpc, TCG_CALL_NO_RWG_SE, i64)
+
 #ifndef IN_HELPER_PROTO
 /*
  * Pass calls to memset directly to libc, without a thunk in qemu.
-- 
2.52.0

[PATCH v2 03/50] tcg: Introduce tcg-global-mappings

[Anton Johansson via qemu development]

Adds a cpu_mapping struct to describe, in a declarative fashion, the
mapping between fields in a struct, and a corresponding TCG global.  As
such, tcg_global_mem_new() can be automatically called given an array of
cpu_mappings.

This change is not limited to helper-to-tcg, but will be required in
future commits to map between offsets into CPUArchState and TCGv
globals in a target-agnostic way.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 include/tcg/tcg-global-mappings.h | 118 ++++++++++++++++++++++++++++++
 tcg/meson.build                   |   1 +
 tcg/tcg-global-mappings.c         |  62 ++++++++++++++++
 3 files changed, 181 insertions(+)
 create mode 100644 include/tcg/tcg-global-mappings.h
 create mode 100644 tcg/tcg-global-mappings.c

diff --git a/include/tcg/tcg-global-mappings.h b/include/tcg/tcg-global-mappings.h
new file mode 100644
index 0000000000..341dcd20b3
--- /dev/null
+++ b/include/tcg/tcg-global-mappings.h
@@ -0,0 +1,118 @@
+/*
+ *  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef TCG_GLOBAL_MAP_H
+#define TCG_GLOBAL_MAP_H
+
+/**
+ * cpu_tcg_mapping: Declarative mapping of offsets into a struct to global
+ *                  TCGvs.  Parseable by LLVM-based tools.
+ * @tcg_var_name: String name of the TCGv to use as destination of the mapping.
+ * @tcg_var_base_address: Address of the above TCGv.
+ * @cpu_type_name: String name of the base type being mapped, e.g. "CPUArchState".
+ * @cpu_var_names: Array of printable names of TCGvs, used when calling
+ *                 tcg_global_mem_new from init_cpu_tcg_mappings.  Contains
+ *                 @number_of_elements strings.
+ * @cpu_var_base_offset: Base offset of field in the source struct.
+ * @cpu_var_size: Size of field in the source struct, if the field is an array,
+ *                this holds the size of the element type.
+ * @cpu_var_stride: Stride between array elements in the source struct.  This
+ *                  can be greater than the element size when mapping a field
+ *                  in an array of structs.
+ * @number_of_elements: Number of elements of array in the source struct.
+ */
+typedef struct cpu_tcg_mapping {
+    const char *tcg_var_name;
+    void *tcg_var_base_address;
+
+    const char *cpu_type_name;
+    const char *const *cpu_var_names;
+    size_t cpu_var_base_offset;
+    size_t cpu_var_size;
+    size_t cpu_var_stride;
+
+    size_t number_of_elements;
+} cpu_tcg_mapping;
+
+#define STRUCT_SIZEOF_FIELD(S, member) sizeof(((S *)0)->member)
+
+#define STRUCT_ARRAY_SIZE(S, array)                                            \
+    (STRUCT_SIZEOF_FIELD(S, array) / STRUCT_SIZEOF_FIELD(S, array[0]))
+
+/*
+ * Following are a few macros that aid in constructing
+ * `cpu_tcg_mapping`s for a few common cases.
+ */
+
+/* Map between single CPU register and to TCG global */
+#define CPU_TCG_MAP(struct_type, tcg_var, cpu_var)                             \
+    (cpu_tcg_mapping)                                                          \
+    {                                                                          \
+        .tcg_var_name = stringify(tcg_var),                                    \
+        .tcg_var_base_address = &tcg_var,                                      \
+        .cpu_type_name = stringify(struct_type),                               \
+        .cpu_var_names = (const char *[]){stringify(cpu_var)},                 \
+        .cpu_var_base_offset = offsetof(struct_type, cpu_var),                 \
+        .cpu_var_size = STRUCT_SIZEOF_FIELD(struct_type, cpu_var),             \
+        .cpu_var_stride = 0, .number_of_elements = 1,                          \
+    }
+
+/* Map between array of CPU registers and array of TCG globals. */
+#define CPU_TCG_MAP_ARRAY(struct_type, tcg_var, cpu_var, names)                \
+    (cpu_tcg_mapping)                                                          \
+    {                                                                          \
+        .tcg_var_name = stringify(tcg_var),                                    \
+        .tcg_var_base_address = tcg_var,                                       \
+        .cpu_type_name = stringify(struct_type),                               \
+        .cpu_var_names = names,                                                \
+        .cpu_var_base_offset = offsetof(struct_type, cpu_var),                 \
+        .cpu_var_size = STRUCT_SIZEOF_FIELD(struct_type, cpu_var[0]),          \
+        .cpu_var_stride = STRUCT_SIZEOF_FIELD(struct_type, cpu_var[0]),        \
+        .number_of_elements = STRUCT_ARRAY_SIZE(struct_type, cpu_var),         \
+    }
+
+/*
+ * Map between single member in an array of structs to an array
+ * of TCG globals, e.g. maps
+ *
+ *     cpu_state.array_of_structs[i].member
+ *
+ * to
+ *
+ *     tcg_global_member[i]
+ */
+#define CPU_TCG_MAP_ARRAY_OF_STRUCTS(struct_type, tcg_var, cpu_struct,         \
+                                     cpu_var, names)                           \
+    (cpu_tcg_mapping)                                                          \
+    {                                                                          \
+        .tcg_var_name = stringify(tcg_var),                                    \
+        .tcg_var_base_address = tcg_var,                                       \
+        .cpu_type_name = stringify(struct_type),                               \
+        .cpu_var_names = names,                                                \
+        .cpu_var_base_offset = offsetof(struct_type, cpu_struct[0].cpu_var),   \
+        .cpu_var_size =                                                        \
+            STRUCT_SIZEOF_FIELD(struct_type, cpu_struct[0].cpu_var),           \
+        .cpu_var_stride = STRUCT_SIZEOF_FIELD(struct_type, cpu_struct[0]),     \
+        .number_of_elements = STRUCT_ARRAY_SIZE(struct_type, cpu_struct),      \
+    }
+
+extern cpu_tcg_mapping tcg_global_mappings[];
+extern size_t tcg_global_mapping_count;
+
+void init_cpu_tcg_mappings(cpu_tcg_mapping *mappings, size_t size);
+
+#endif /* TCG_GLOBAL_MAP_H */
diff --git a/tcg/meson.build b/tcg/meson.build
index 706a6eb260..c7c16eeb05 100644
--- a/tcg/meson.build
+++ b/tcg/meson.build
@@ -13,6 +13,7 @@ tcg_ss.add(files(
   'tcg-op-ldst.c',
   'tcg-op-gvec.c',
   'tcg-op-vec.c',
+  'tcg-global-mappings.c',
 ))
 
 if get_option('tcg_interpreter')
diff --git a/tcg/tcg-global-mappings.c b/tcg/tcg-global-mappings.c
new file mode 100644
index 0000000000..a75fad0c21
--- /dev/null
+++ b/tcg/tcg-global-mappings.c
@@ -0,0 +1,62 @@
+/*
+ *  Copyright(c) 2024 rev.ng Labs Srl. All Rights Reserved.
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "qemu/osdep.h"
+#include "tcg/tcg-global-mappings.h"
+#include "tcg/tcg-op-common.h"
+#include "tcg/tcg.h"
+
+void init_cpu_tcg_mappings(cpu_tcg_mapping *mappings, size_t size)
+{
+    uintptr_t tcg_addr;
+    size_t cpu_offset;
+    const char *name;
+    cpu_tcg_mapping m;
+
+    /*
+     * Paranoid assertion, this should always hold since
+     * they're typedef'd to pointers. But you never know!
+     */
+    g_assert(sizeof(TCGv_i32) == sizeof(TCGv_i64));
+
+    /*
+     * Loop over entries in tcg_global_mappings and
+     * create the `mapped to` TCGv's.
+     */
+    for (int i = 0; i < size; ++i) {
+        m = mappings[i];
+
+        for (int j = 0; j < m.number_of_elements; ++j) {
+            /*
+             * Here we are using the fact that
+             * sizeof(TCGv_i32) == sizeof(TCGv_i64) == sizeof(TCGv)
+             */
+            assert(sizeof(TCGv_i32) == sizeof(TCGv_i64));
+            tcg_addr = (uintptr_t)m.tcg_var_base_address + j * sizeof(TCGv_i32);
+            cpu_offset = m.cpu_var_base_offset + j * m.cpu_var_stride;
+            name = m.cpu_var_names[j];
+
+            if (m.cpu_var_size < 8) {
+                *(TCGv_i32 *)tcg_addr =
+                    tcg_global_mem_new_i32(tcg_env, cpu_offset, name);
+            } else {
+                *(TCGv_i64 *)tcg_addr =
+                    tcg_global_mem_new_i64(tcg_env, cpu_offset, name);
+            }
+        }
+    }
+}
-- 
2.52.0

[PATCH v2 04/50] tcg: Increase maximum TB size

[Anton Johansson via qemu development]

Doubles amount of space allocated for translation blocks.  This is
needed, particularly for Hexagon, where a single instruction packet may
consist of up to four vector instructions.  If each vector instruction
then gets expanded into gvec operations that utilize a small host vector
size the TB blows up quite quickly.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 include/tcg/tcg.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/tcg/tcg.h b/include/tcg/tcg.h
index 7669dc1c2d..9f0a35f1be 100644
--- a/include/tcg/tcg.h
+++ b/include/tcg/tcg.h
@@ -40,7 +40,7 @@
 /* XXX: make safe guess about sizes */
 #define MAX_OP_PER_INSTR 266
 
-#define CPU_TEMP_BUF_NLONGS 128
+#define CPU_TEMP_BUF_NLONGS 256
 #define TCG_STATIC_FRAME_SIZE  (CPU_TEMP_BUF_NLONGS * sizeof(long))
 
 typedef int64_t tcg_target_long;
-- 
2.52.0

[PATCH v2 05/50] tcg: Expose tcg_gen_ussub_sat()

[Anton Johansson via qemu development]

Maps nicely to LLVMs usub.sat instrinsic.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 include/tcg/tcg-op-common.h | 2 ++
 tcg/tcg-op-gvec.c           | 4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/include/tcg/tcg-op-common.h b/include/tcg/tcg-op-common.h
index 1fe342db0d..9df8a72c08 100644
--- a/include/tcg/tcg-op-common.h
+++ b/include/tcg/tcg-op-common.h
@@ -192,6 +192,7 @@ void tcg_gen_sar_i32(TCGv_i32 ret, TCGv_i32 arg1, TCGv_i32 arg2);
 void tcg_gen_mul_i32(TCGv_i32 ret, TCGv_i32 arg1, TCGv_i32 arg2);
 void tcg_gen_neg_i32(TCGv_i32 ret, TCGv_i32 arg);
 void tcg_gen_not_i32(TCGv_i32 ret, TCGv_i32 arg);
+void tcg_gen_ussub_i32(TCGv_i32 d, TCGv_i32 a, TCGv_i32 b);
 
 /* 64 bit ops */
 
@@ -258,6 +259,7 @@ void tcg_gen_mulu2_i64(TCGv_i64 rl, TCGv_i64 rh, TCGv_i64 arg1, TCGv_i64 arg2);
 void tcg_gen_muls2_i64(TCGv_i64 rl, TCGv_i64 rh, TCGv_i64 arg1, TCGv_i64 arg2);
 void tcg_gen_mulsu2_i64(TCGv_i64 rl, TCGv_i64 rh, TCGv_i64 arg1, TCGv_i64 arg2);
 void tcg_gen_not_i64(TCGv_i64 ret, TCGv_i64 arg);
+void tcg_gen_ussub_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b);
 void tcg_gen_ext8s_i64(TCGv_i64 ret, TCGv_i64 arg);
 void tcg_gen_ext16s_i64(TCGv_i64 ret, TCGv_i64 arg);
 void tcg_gen_ext32s_i64(TCGv_i64 ret, TCGv_i64 arg);
diff --git a/tcg/tcg-op-gvec.c b/tcg/tcg-op-gvec.c
index bc323e2500..cd0a2fd83d 100644
--- a/tcg/tcg-op-gvec.c
+++ b/tcg/tcg-op-gvec.c
@@ -2399,14 +2399,14 @@ void tcg_gen_gvec_usadd(unsigned vece, uint32_t dofs, uint32_t aofs,
     tcg_gen_gvec_3(dofs, aofs, bofs, oprsz, maxsz, &g[vece]);
 }
 
-static void tcg_gen_ussub_i32(TCGv_i32 d, TCGv_i32 a, TCGv_i32 b)
+void tcg_gen_ussub_i32(TCGv_i32 d, TCGv_i32 a, TCGv_i32 b)
 {
     TCGv_i32 min = tcg_constant_i32(0);
     tcg_gen_sub_i32(d, a, b);
     tcg_gen_movcond_i32(TCG_COND_LTU, d, a, b, min, d);
 }
 
-static void tcg_gen_ussub_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b)
+void tcg_gen_ussub_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b)
 {
     TCGv_i64 min = tcg_constant_i64(0);
     tcg_gen_sub_i64(d, a, b);
-- 
2.52.0

[PATCH v2 06/50] Add helper-to-tcg subproject

[Anton Johansson via qemu development]

Adds a new bare bones subproject along with options to enable or disable
it.  The LLVM dependency is managed manually through `llvm-config` for
two reasons: meson struggles with multiple system LLVM versions anyway
and usually returns the newest one even if it doesn't satisfy the version
requirement, and; It allows the user to override the default if they
choose.  As such, an option is added to the subproject to manually
override `llvm-config`.

A new target config option TARGET_HELPER_TO_TCG is also introduced,
which will used in future commits to conditionally add or remove
remove code and memory required by helper-to-tcg.

Current meson option is limited to Hexagon, this might change in the
future if more uses are found on different frontends.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 meson.build                                 | 10 ++++
 meson_options.txt                           |  2 +
 scripts/meson-buildoptions.sh               |  5 ++
 subprojects/helper-to-tcg/meson.build       | 61 +++++++++++++++++++++
 subprojects/helper-to-tcg/meson_options.txt |  2 +
 5 files changed, 80 insertions(+)
 create mode 100644 subprojects/helper-to-tcg/meson.build
 create mode 100644 subprojects/helper-to-tcg/meson_options.txt

diff --git a/meson.build b/meson.build
index 49a5baf5b5..a1411b0eb6 100644
--- a/meson.build
+++ b/meson.build
@@ -3273,7 +3273,16 @@ host_kconfig = \
   (hv_balloon ? ['CONFIG_HV_BALLOON_POSSIBLE=y'] : []) + \
   (have_rust ? ['CONFIG_HAVE_RUST=y'] : [])
 
+helper_to_tcg = subproject('helper-to-tcg',
+  required: get_option('hexagon_helper_to_tcg') \
+    .disable_auto_if('hexagon-linux-user' not in target_dirs and \
+                     'hexagon-softmmu' not in target_dirs))
+
 ignored = [ 'TARGET_XML_FILES', 'TARGET_ABI_DIR' ]
+if not helper_to_tcg.found()
+   # do not define it if it is not usable
+   ignored += ['TARGET_HELPER_TO_TCG']
+endif
 
 default_targets = 'CONFIG_DEFAULT_TARGETS' in config_host
 actual_target_dirs = []
@@ -4266,6 +4275,7 @@ foreach target : target_dirs
   if host_os == 'linux'
     target_inc += include_directories('linux-headers', is_system: true)
   endif
+
   if target.endswith('-softmmu')
     target_type='system'
     if target_base_arch in target_system_arch
diff --git a/meson_options.txt b/meson_options.txt
index a07cb47d35..63309ceaff 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -377,6 +377,8 @@ option('qemu_ga_version', type: 'string', value: '',
 
 option('hexagon_idef_parser', type : 'boolean', value : true,
        description: 'use idef-parser to automatically generate TCG code for the Hexagon frontend')
+option('hexagon_helper_to_tcg', type : 'feature', value : 'enabled',
+       description: 'use the helper-to-tcg translator to automatically generate TCG code from helpers')
 
 option('x86_version', type : 'combo', choices : ['0', '1', '2', '3', '4'], value: '1',
        description: 'tweak required x86_64 architecture version beyond compiler default')
diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh
index c003985047..9951d5ad95 100644
--- a/scripts/meson-buildoptions.sh
+++ b/scripts/meson-buildoptions.sh
@@ -125,6 +125,9 @@ meson_options_help() {
   printf "%s\n" '  gtk             GTK+ user interface'
   printf "%s\n" '  guest-agent     Build QEMU Guest Agent'
   printf "%s\n" '  guest-agent-msi Build MSI package for the QEMU Guest Agent'
+  printf "%s\n" '  hexagon-helper-to-tcg'
+  printf "%s\n" '                  use the helper-to-tcg translator to automatically generate'
+  printf "%s\n" '                  TCG code from helpers'
   printf "%s\n" '  hv-balloon      hv-balloon driver (requires Glib 2.68+ GTree API)'
   printf "%s\n" '  hvf             HVF acceleration support'
   printf "%s\n" '  iconv           Font glyph conversion support'
@@ -337,6 +340,8 @@ _meson_option_parse() {
     --disable-guest-agent) printf "%s" -Dguest_agent=disabled ;;
     --enable-guest-agent-msi) printf "%s" -Dguest_agent_msi=enabled ;;
     --disable-guest-agent-msi) printf "%s" -Dguest_agent_msi=disabled ;;
+    --enable-hexagon-helper-to-tcg) printf "%s" -Dhexagon_helper_to_tcg=enabled ;;
+    --disable-hexagon-helper-to-tcg) printf "%s" -Dhexagon_helper_to_tcg=disabled ;;
     --enable-hexagon-idef-parser) printf "%s" -Dhexagon_idef_parser=true ;;
     --disable-hexagon-idef-parser) printf "%s" -Dhexagon_idef_parser=false ;;
     --enable-hv-balloon) printf "%s" -Dhv_balloon=enabled ;;
diff --git a/subprojects/helper-to-tcg/meson.build b/subprojects/helper-to-tcg/meson.build
new file mode 100644
index 0000000000..8ab58adb39
--- /dev/null
+++ b/subprojects/helper-to-tcg/meson.build
@@ -0,0 +1,61 @@
+##
+##  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+##
+##  This program is free software; you can redistribute it and/or modify
+##  it under the terms of the GNU General Public License as published by
+##  the Free Software Foundation; either version 2 of the License, or
+##  (at your option) any later version.
+##
+##  This program is distributed in the hope that it will be useful,
+##  but WITHOUT ANY WARRANTY; without even the implied warranty of
+##  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+##  GNU General Public License for more details.
+##
+##  You should have received a copy of the GNU General Public License
+##  along with this program; if not, see <http://www.gnu.org/licenses/>.
+##
+
+project('helper-to-tcg', ['cpp'],
+        version: '0.8',
+        default_options: ['cpp_std=c++17'])
+
+python = import('python').find_installation()
+
+# Find LLVM using llvm-config manually.  Needed as meson struggles when multiple
+# versions of LLVM are installed on the same system (always returns the most
+# recent).
+llvm_config = get_option('llvm_config_path')
+cpp_args = run_command(llvm_config, '--cxxflags').stdout().strip().split()
+bindir = run_command(llvm_config, '--bindir').stdout().strip()
+ldflags = run_command(llvm_config, '--ldflags').stdout().strip().split()
+libs = run_command(llvm_config, '--libs').stdout().strip().split()
+syslibs = run_command(llvm_config, '--system-libs').stdout().strip().split()
+incdir = run_command(llvm_config, '--includedir').stdout().strip().split()
+version = run_command(llvm_config, '--version').stdout().strip()
+version_major = version.split('.')[0].to_int()
+
+# Check LLVM version manually
+if version_major < 10 or version_major > 21
+    error('LLVM version', version, 'not supported.')
+endif
+
+sources = [
+]
+
+# NOTE: Add -Wno-template-id-cdtor for GCC versions >= 14.  This warning is
+# related to a change in the C++ standard in C++20, that also applies to C++14
+# for some reason. See defect report DR2237 and commit
+#   https://gcc.gnu.org/git/gitweb.cgi?p=gcc.git;h=4b38d56dbac6742b038551a36ec80200313123a1
+# (temporary)
+compiler_info = meson.get_compiler('cpp')
+compiler = compiler_info.get_id()
+compiler_version = compiler_info.version().split('-').get(0)
+compiler_version_major = compiler_version.split('.').get(0)
+if compiler == 'gcc' and compiler_version_major.to_int() >= 14
+    cpp_args += ['-Wno-template-id-cdtor', '-Wno-missing-template-keyword']
+endif
+
+pipeline = executable('helper-to-tcg', sources,
+                      include_directories: ['include'] + [incdir],
+                      link_args: [ldflags] + [libs] + [syslibs],
+                      cpp_args: cpp_args)
diff --git a/subprojects/helper-to-tcg/meson_options.txt b/subprojects/helper-to-tcg/meson_options.txt
new file mode 100644
index 0000000000..8a4b28a585
--- /dev/null
+++ b/subprojects/helper-to-tcg/meson_options.txt
@@ -0,0 +1,2 @@
+option('llvm_config_path', type : 'string', value : 'llvm-config',
+       description: 'override default llvm-config used for finding LLVM')
-- 
2.52.0

[PATCH v2 07/50] helper-to-tcg: Introduce get-llvm-ir.py

[Anton Johansson via qemu development]

Introduces a new python helper script to convert a set of QEMU .c files to
LLVM IR .ll using clang.  Compile flags are found by looking at
compile_commands.json, and llvm-link is used to link together all LLVM
modules into a single module.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 subprojects/helper-to-tcg/get-llvm-ir.py | 145 +++++++++++++++++++++++
 subprojects/helper-to-tcg/meson.build    |   8 ++
 2 files changed, 153 insertions(+)
 create mode 100755 subprojects/helper-to-tcg/get-llvm-ir.py

diff --git a/subprojects/helper-to-tcg/get-llvm-ir.py b/subprojects/helper-to-tcg/get-llvm-ir.py
new file mode 100755
index 0000000000..982b87f791
--- /dev/null
+++ b/subprojects/helper-to-tcg/get-llvm-ir.py
@@ -0,0 +1,145 @@
+#!/usr/bin/env python3
+
+##
+##  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+##
+##  This program is free software; you can redistribute it and/or modify
+##  it under the terms of the GNU General Public License as published by
+##  the Free Software Foundation; either version 2 of the License, or
+##  (at your option) any later version.
+##
+##  This program is distributed in the hope that it will be useful,
+##  but WITHOUT ANY WARRANTY; without even the implied warranty of
+##  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+##  GNU General Public License for more details.
+##
+##  You should have received a copy of the GNU General Public License
+##  along with this program; if not, see <http://www.gnu.org/licenses/>.
+##
+
+import argparse
+import json
+import os
+import shlex
+import sys
+import subprocess
+
+
+def log(msg):
+    print(msg, file=sys.stderr)
+
+
+def run_command(command):
+    proc = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+    out = proc.communicate()
+    if proc.wait() != 0:
+        log(f"Command: {' '.join(command)} exited with {proc.returncode}\n")
+        log(f"output:\n{out}\n")
+
+
+def find_compile_commands(compile_commands_path, clang_path, input_path, target):
+    with open(compile_commands_path, "r") as f:
+        compile_commands = json.load(f)
+        for compile_command in compile_commands:
+            path = compile_command["file"]
+            if os.path.basename(path) != os.path.basename(input_path):
+                continue
+
+            os.chdir(compile_command["directory"])
+            command = compile_command["command"]
+
+            # If building multiple targets there's a chance
+            # input files share the same path and name.
+            # This could cause us to find the wrong compile
+            # command, we use the target path to distinguish
+            # between these.
+            if not target in command:
+                continue
+
+            argv = shlex.split(command)
+            argv[0] = clang_path
+
+            return argv
+
+    raise ValueError(f"Unable to find compile command for {input_path}")
+
+
+def generate_llvm_ir(
+    compile_commands_path, clang_path, output_path, input_path, target
+):
+    command = find_compile_commands(
+        compile_commands_path, clang_path, input_path, target
+    )
+
+    flags_to_remove = {
+        "-ftrivial-auto-var-init=zero",
+        "-fzero-call-used-regs=used-gpr",
+        "-Wimplicit-fallthrough=2",
+        "-Wold-style-declaration",
+        "-Wno-psabi",
+        "-Wshadow=local",
+        "-c",
+    }
+
+    # Remove
+    #   - output of makefile rules (-MQ,-MF target);
+    #   - output of object files (-o target);
+    #   - excessive zero-initialization of block-scope variables
+    #     (-ftrivial-auto-var-init=zero);
+    #   - and any optimization flags (-O).
+    for i, arg in reversed(list(enumerate(command))):
+        if arg in {"-MQ", "-o", "-MF"}:
+            del command[i : i + 2]
+        elif arg.startswith("-O") or arg in flags_to_remove:
+            del command[i]
+
+    # Define a HELPER_TO_TCG macro for translation units wanting to
+    # conditionally include or exclude code during translation to TCG.
+    # Disable optimization (-O0) and make sure clang doesn't emit optnone
+    # attributes (-disable-O0-optnone) which inhibit further optimization.
+    # Optimization will be performed at a later stage in the helper-to-tcg
+    # pipeline.
+    command += [
+        "-S",
+        "-emit-llvm",
+        "-DHELPER_TO_TCG_IR_GEN",
+        "-O0",
+        "-g",
+        "-Xclang",
+        "-disable-O0-optnone",
+    ]
+    if output_path:
+        command += ["-o", output_path]
+
+    run_command(command)
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Produce the LLVM IR of a given .c file."
+    )
+    parser.add_argument(
+        "--compile-commands", required=True, help="Path to compile_commands.json"
+    )
+    parser.add_argument("--clang", default="clang", help="Path to clang.")
+    parser.add_argument("--llvm-link", default="llvm-link", help="Path to llvm-link.")
+    parser.add_argument("-o", "--output", required=True, help="Output .ll file path")
+    parser.add_argument(
+        "--target-path", help="Path to QEMU target dir. (e.q. target/i386)"
+    )
+    parser.add_argument("inputs", nargs="+", help=".c file inputs")
+    args = parser.parse_args()
+
+    outputs = []
+    for input in args.inputs:
+        output = os.path.basename(input) + ".ll"
+        generate_llvm_ir(
+            args.compile_commands, args.clang, output, input, args.target_path
+        )
+        outputs.append(output)
+
+    run_command([args.llvm_link] + outputs + ["-S", "-o", args.output])
+
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/subprojects/helper-to-tcg/meson.build b/subprojects/helper-to-tcg/meson.build
index 8ab58adb39..97bce186fe 100644
--- a/subprojects/helper-to-tcg/meson.build
+++ b/subprojects/helper-to-tcg/meson.build
@@ -42,6 +42,14 @@ endif
 sources = [
 ]
 
+clang = bindir / 'clang'
+llvm_link = bindir / 'llvm-link'
+
+get_llvm_ir_cmd = [python, meson.current_source_dir() / 'get-llvm-ir.py',
+                   '--compile-commands', 'compile_commands.json',
+                   '--clang', clang,
+                   '--llvm-link', llvm_link]
+
 # NOTE: Add -Wno-template-id-cdtor for GCC versions >= 14.  This warning is
 # related to a change in the C++ standard in C++20, that also applies to C++14
 # for some reason. See defect report DR2237 and commit
-- 
2.52.0

[PATCH v2 08/50] helper-to-tcg: Handle LLVM version compatibility

[Anton Johansson via qemu development]

Adds a translation unit with the sole purpose of handling inter-LLVM
code changes.  Instead of littering the code with #ifdefs, most of them
will be limited to LlvmCompat.[cpp|hpp] and a saner compat::*() function
is exposed in its place.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 .../helper-to-tcg/include/LlvmCompat.hpp      | 124 ++++++++++++++++++
 subprojects/helper-to-tcg/meson.build         |   1 +
 subprojects/helper-to-tcg/src/LlvmCompat.cpp  |  89 +++++++++++++
 3 files changed, 214 insertions(+)
 create mode 100644 subprojects/helper-to-tcg/include/LlvmCompat.hpp
 create mode 100644 subprojects/helper-to-tcg/src/LlvmCompat.cpp

diff --git a/subprojects/helper-to-tcg/include/LlvmCompat.hpp b/subprojects/helper-to-tcg/include/LlvmCompat.hpp
new file mode 100644
index 0000000000..78822ca87b
--- /dev/null
+++ b/subprojects/helper-to-tcg/include/LlvmCompat.hpp
@@ -0,0 +1,124 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#pragma once
+
+//
+// The purpose of this file is to both collect and hide most api-specific
+// changes of LLVM [10,14]. Hopefully making it easier to keep track of the
+// changes necessary to support our targeted versions.
+//
+// Note some #ifdefs still remain throughout the codebase for larger codeblocks
+// that are specific enough such that pulling them here would be more cumbersome
+// than it's worth.
+//
+
+#include <llvm/Analysis/TargetTransformInfo.h>
+#include <llvm/IR/DerivedTypes.h>
+#include <llvm/IR/Intrinsics.h>
+#include <llvm/IR/Module.h>
+#include <llvm/IR/PassManager.h>
+#include <llvm/IR/PatternMatch.h>
+#include <llvm/MC/TargetRegistry.h>
+#include <llvm/Passes/OptimizationLevel.h>
+#include <llvm/Passes/PassBuilder.h>
+#include <llvm/Support/FileSystem.h>
+#include <llvm/Transforms/Utils/UnifyFunctionExitNodes.h>
+
+#include <stdint.h>
+
+namespace compat {
+
+constexpr auto OpenFlags = llvm::sys::fs::OF_TextWithCRLF;
+
+using OptimizationLevel = llvm::OptimizationLevel;
+
+constexpr auto LTOPhase = llvm::ThinOrFullLTOPhase::None;
+
+inline llvm::PassBuilder createPassBuilder(llvm::TargetMachine *TM,
+                                           llvm::PipelineTuningOptions &PTO) {
+#if LLVM_VERSION_MAJOR >= 16
+    return llvm::PassBuilder(TM, PTO, std::nullopt);
+#else
+    return llvm::PassBuilder(TM, PTO, llvm::None);
+#endif
+}
+
+// LLVM 16 changes the getFixedSize() -> getFixedValue()
+size_t getTypeAllocSize(const llvm::DataLayout &DL, llvm::Type *Ty);
+
+// These rely on string comparison with QEMU names using StringRef::startswith,
+// which changed to StringRef::starts_with in LLVM 18.
+bool isFunctionQemuHelper(llvm::StringRef Name);
+bool isFunctionQemuLoadStore(llvm::StringRef Name);
+
+// LLVM 21 moved to initializing a TargetTransformInfo via unique_ptr,
+// instead of by value.
+template <typename T>
+auto makeTTI(llvm::TargetMachine *TM, const llvm::Function &F) {
+    // Use explicit TargetTransformInfo() contructor
+#if LLVM_VERSION_MAJOR >= 21
+    return llvm::TargetTransformInfo(std::make_unique<T>(TM, F));
+#else
+    return llvm::TargetTransformInfo(T(TM, F));
+#endif
+}
+
+// Note, since constexpr auto is used to alias a function, default
+// arguments won't work.
+//
+// LLVM 20 deprecated getDeclaration() in favour of getOrInsertDeclaration()
+namespace Intrinsic {
+#if LLVM_VERSION_MAJOR >= 20
+constexpr auto getOrInsertDeclaration = llvm::Intrinsic::getOrInsertDeclaration;
+#else
+constexpr auto getOrInsertDeclaration = llvm::Intrinsic::getDeclaration;
+#endif
+} // namespace Intrinsic
+
+// Wrapper to convert Function- to Module analysis manager
+template <typename T>
+inline const typename T::Result *
+getModuleAnalysisManagerProxyResult(llvm::FunctionAnalysisManager &FAM,
+                                    llvm::Function &F) {
+    auto &MAMProxy = FAM.getResult<llvm::ModuleAnalysisManagerFunctionProxy>(F);
+    return MAMProxy.getCachedResult<T>(*F.getParent());
+}
+
+llvm::TargetMachine *getTargetMachine(llvm::Triple &TheTriple);
+
+//
+// LLVM 11 and below does not define the UnifyFunctionExitNodes pass
+// for the new pass manager.  Copy over the definition and use it for
+// 11 and below.
+//
+using llvm::UnifyFunctionExitNodesPass;
+
+inline uint32_t getVectorElementCount(const llvm::VectorType *VecTy) {
+    auto ElementCount = VecTy->getElementCount();
+    return ElementCount.getFixedValue();
+}
+
+//
+// PatternMatch
+//
+
+#define compat_m_InsertElt llvm::PatternMatch::m_InsertElt
+#define compat_m_Shuffle llvm::PatternMatch::m_Shuffle
+#define compat_m_ZeroMask llvm::PatternMatch::m_ZeroMask
+
+} // namespace compat
diff --git a/subprojects/helper-to-tcg/meson.build b/subprojects/helper-to-tcg/meson.build
index 97bce186fe..965014c988 100644
--- a/subprojects/helper-to-tcg/meson.build
+++ b/subprojects/helper-to-tcg/meson.build
@@ -40,6 +40,7 @@ if version_major < 10 or version_major > 21
 endif
 
 sources = [
+    'src/LlvmCompat.cpp',
 ]
 
 clang = bindir / 'clang'
diff --git a/subprojects/helper-to-tcg/src/LlvmCompat.cpp b/subprojects/helper-to-tcg/src/LlvmCompat.cpp
new file mode 100644
index 0000000000..3a1995391c
--- /dev/null
+++ b/subprojects/helper-to-tcg/src/LlvmCompat.cpp
@@ -0,0 +1,89 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#include "LlvmCompat.hpp"
+
+#include <llvm/CodeGen/CommandFlags.h>
+#include <llvm/Support/CodeGen.h>
+
+#include <string>
+
+// Static variables required by LLVM
+//
+// Defining RegisterCodeGenFlags with static duration registers extra
+// codegen commandline flags for specifying the target arch.
+static llvm::codegen::RegisterCodeGenFlags CGF;
+static llvm::ExitOnError ExitOnErr;
+
+namespace compat {
+
+using namespace llvm;
+
+size_t getTypeAllocSize(const llvm::DataLayout &DL, llvm::Type *Ty) {
+#if LLVM_VERSION_MAJOR >= 16
+    return DL.getTypeAllocSize(Ty).getFixedValue();
+#else
+    return DL.getTypeAllocSize(Ty).getFixedSize();
+#endif
+}
+
+bool isFunctionQemuHelper(StringRef Name) {
+#if LLVM_VERSION_MAJOR >= 18
+    return Name.starts_with("helper_");
+#else
+    return Name.startswith("helper_");
+#endif
+}
+
+bool isFunctionQemuLoadStore(StringRef Name) {
+#if LLVM_VERSION_MAJOR >= 18
+    return Name.starts_with("cpu_ld") or Name.starts_with("cpu_st");
+#else
+    return Name.startswith("cpu_ld") or Name.startswith("cpu_st");
+#endif
+}
+
+llvm::TargetMachine *getTargetMachine(llvm::Triple &TheTriple) {
+    const TargetOptions Options{};
+    std::string Error;
+    const Target *TheTarget = llvm::TargetRegistry::lookupTarget(
+        llvm::codegen::getMArch(), TheTriple, Error);
+    // Some modules don't specify a triple, and this is okay.
+    if (!TheTarget) {
+        return nullptr;
+    }
+
+#if LLVM_VERSION_MAJOR >= 18
+    auto Level = llvm::CodeGenOptLevel::Aggressive;
+#else
+    auto Level = llvm::CodeGenOpt::Aggressive;
+#endif
+
+#if LLVM_VERSION_MAJOR >= 21
+    return TheTarget->createTargetMachine(
+        llvm::Triple(TheTriple.getTriple()), llvm::codegen::getCPUStr(),
+        llvm::codegen::getFeaturesStr(), Options,
+        llvm::codegen::getExplicitRelocModel(),
+        llvm::codegen::getExplicitCodeModel(), Level);
+#else
+    return TheTarget->createTargetMachine(
+        TheTriple.getTriple(), llvm::codegen::getCPUStr(),
+        llvm::codegen::getFeaturesStr(), Options, {}, {}, Level);
+#endif
+}
+
+} // namespace compat
-- 
2.52.0

[PATCH v2 09/50] helper-to-tcg: Introduce custom LLVM pipeline

[Anton Johansson via qemu development]

Adds a custom pipeline, similar to LLVM opt, with the goal of taking an
input LLVM IR module to an equivalent output .c file implementing
functions in TCG.

Initial LLVM boilerplate is added up until the creation of a
ModulePassManager.  A custom target derived from x64 is added, to ensure
consistent behaviour across different hosts.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 .../helper-to-tcg/include/CmdLineOptions.hpp  |  23 +++
 subprojects/helper-to-tcg/meson.build         |   1 +
 subprojects/helper-to-tcg/src/Pipeline.cpp    | 146 ++++++++++++++++++
 3 files changed, 170 insertions(+)
 create mode 100644 subprojects/helper-to-tcg/include/CmdLineOptions.hpp
 create mode 100644 subprojects/helper-to-tcg/src/Pipeline.cpp

diff --git a/subprojects/helper-to-tcg/include/CmdLineOptions.hpp b/subprojects/helper-to-tcg/include/CmdLineOptions.hpp
new file mode 100644
index 0000000000..93706b78c5
--- /dev/null
+++ b/subprojects/helper-to-tcg/include/CmdLineOptions.hpp
@@ -0,0 +1,23 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#pragma once
+
+#include <llvm/Support/CommandLine.h>
+
+// Options for pipeline
+extern llvm::cl::list<std::string> InputFiles;
diff --git a/subprojects/helper-to-tcg/meson.build b/subprojects/helper-to-tcg/meson.build
index 965014c988..ca46578cb3 100644
--- a/subprojects/helper-to-tcg/meson.build
+++ b/subprojects/helper-to-tcg/meson.build
@@ -41,6 +41,7 @@ endif
 
 sources = [
     'src/LlvmCompat.cpp',
+    'src/Pipeline.cpp',
 ]
 
 clang = bindir / 'clang'
diff --git a/subprojects/helper-to-tcg/src/Pipeline.cpp b/subprojects/helper-to-tcg/src/Pipeline.cpp
new file mode 100644
index 0000000000..f36f447fd2
--- /dev/null
+++ b/subprojects/helper-to-tcg/src/Pipeline.cpp
@@ -0,0 +1,146 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#include "CmdLineOptions.hpp"
+#include "LlvmCompat.hpp"
+
+#if LLVM_VERSION_MAJOR == 15
+#include <llvm/ADT/Triple.h>
+#else
+#include <llvm/TargetParser/Triple.h>
+#endif
+#include <llvm/Analysis/AliasAnalysis.h>
+#include <llvm/Analysis/CGSCCPassManager.h>
+#include <llvm/Analysis/LoopAnalysisManager.h>
+#include <llvm/Analysis/TargetTransformInfo.h>
+#include <llvm/CodeGen/BasicTTIImpl.h>
+#include <llvm/IR/LLVMContext.h>
+#include <llvm/IR/Module.h>
+#include <llvm/IR/PassManager.h>
+#include <llvm/IRReader/IRReader.h>
+#include <llvm/InitializePasses.h>
+#include <llvm/Linker/Linker.h>
+#include <llvm/PassRegistry.h>
+#include <llvm/Passes/PassBuilder.h>
+#include <llvm/Support/CommandLine.h>
+#include <llvm/Support/InitLLVM.h>
+#include <llvm/Support/SourceMgr.h>
+#include <llvm/Support/TargetSelect.h>
+#include <llvm/Target/TargetMachine.h>
+
+using namespace llvm;
+
+cl::OptionCategory Cat("helper-to-tcg Options");
+
+// Options for pipeline
+cl::opt<std::string> InputFile(cl::Positional, cl::desc("[input LLVM module]"),
+                               cl::cat(Cat));
+
+// Define a TargetTransformInfo (TTI) subclass, this allows for overriding
+// common per-llvm-target information expected by other LLVM passes, such
+// as the width of the largest scalar/vector registers.  Needed for consistent
+// behaviour across different hosts.
+class TcgTTI : public BasicTTIImplBase<TcgTTI> {
+    friend class BasicTTIImplBase<TcgTTI>;
+
+    // We need to provide ST, TLI, getST(), getTLI()
+    const TargetSubtargetInfo *ST;
+    const TargetLoweringBase *TLI;
+
+    const TargetSubtargetInfo *getST() const { return ST; }
+    const TargetLoweringBase *getTLI() const { return TLI; }
+
+  public:
+    // Initialize ST and TLI from the target machine, e.g. if we're
+    // targeting x86 we'll get the Subtarget and TargetLowering to
+    // match that architechture.
+    TcgTTI(TargetMachine *TM, Function const &F)
+        : BasicTTIImplBase(TM, F.getParent()->getDataLayout()),
+          ST(TM->getSubtargetImpl(F)), TLI(ST->getTargetLowering()) {}
+
+    TypeSize getRegisterBitWidth(TargetTransformInfo::RegisterKind K) const {
+        switch (K) {
+        case TargetTransformInfo::RGK_Scalar:
+            // We pretend we always support 64-bit registers
+            return TypeSize::getFixed(64);
+        case TargetTransformInfo::RGK_FixedWidthVector:
+            // We pretend we always support 2048-bit vector registers
+            return TypeSize::getFixed(2048);
+        case TargetTransformInfo::RGK_ScalableVector:
+            return TypeSize::getScalable(0);
+        default:
+            abort();
+        }
+    }
+};
+
+int main(int argc, char **argv) {
+    InitLLVM X(argc, argv);
+    cl::HideUnrelatedOptions(Cat);
+
+    InitializeAllTargets();
+    InitializeAllTargetMCs();
+    PassRegistry &Registry = *PassRegistry::getPassRegistry();
+    initializeCore(Registry);
+    initializeScalarOpts(Registry);
+    initializeVectorization(Registry);
+    initializeAnalysis(Registry);
+    initializeTransformUtils(Registry);
+    initializeInstCombine(Registry);
+    initializeTarget(Registry);
+
+    cl::ParseCommandLineOptions(argc, argv);
+
+    LLVMContext Context;
+
+    SMDiagnostic Err;
+    std::unique_ptr<Module> M = parseIRFile(InputFile, Err, Context);
+
+    // Create a new TargetMachine to represent a TCG target,
+    // we use x86_64 as a base and derive from that using a
+    // TargetTransformInfo to provide allowed scalar and vector
+    // register sizes.
+    Triple ModuleTriple("x86_64-pc-unknown");
+    assert(ModuleTriple.getArch());
+    TargetMachine *TM = compat::getTargetMachine(ModuleTriple);
+
+    PipelineTuningOptions PTO;
+    PassBuilder PB = compat::createPassBuilder(TM, PTO);
+    LoopAnalysisManager LAM;
+    FunctionAnalysisManager FAM;
+    CGSCCAnalysisManager CGAM;
+    ModuleAnalysisManager MAM;
+
+    // Register our TargetIrAnalysis pass using our own TTI
+    FAM.registerPass([&] {
+        return TargetIRAnalysis(
+            [&TM](const Function &F) -> TargetIRAnalysis::Result {
+                return compat::makeTTI<TcgTTI>(TM, F);
+            });
+    });
+
+    // Register other default LLVM Analyses
+    PB.registerFunctionAnalyses(FAM);
+    PB.registerModuleAnalyses(MAM);
+    PB.registerLoopAnalyses(LAM);
+    PB.registerCGSCCAnalyses(CGAM);
+    PB.crossRegisterProxies(LAM, FAM, CGAM, MAM);
+
+    ModulePassManager MPM;
+
+    return 0;
+}
-- 
2.52.0

[PATCH v2 10/50] helper-to-tcg: Add pipeline --debug and --debug-only

[Anton Johansson via qemu development]

These options are normally added by LLVM, however since we want a
minimal set of options in --help we have to readd them ourselves.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 subprojects/helper-to-tcg/src/Pipeline.cpp | 25 +++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/subprojects/helper-to-tcg/src/Pipeline.cpp b/subprojects/helper-to-tcg/src/Pipeline.cpp
index f36f447fd2..4b9523ad6f 100644
--- a/subprojects/helper-to-tcg/src/Pipeline.cpp
+++ b/subprojects/helper-to-tcg/src/Pipeline.cpp
@@ -37,11 +37,14 @@
 #include <llvm/PassRegistry.h>
 #include <llvm/Passes/PassBuilder.h>
 #include <llvm/Support/CommandLine.h>
+#include <llvm/Support/Debug.h>
 #include <llvm/Support/InitLLVM.h>
 #include <llvm/Support/SourceMgr.h>
 #include <llvm/Support/TargetSelect.h>
 #include <llvm/Target/TargetMachine.h>
 
+#define DEBUG_TYPE "pipeline"
+
 using namespace llvm;
 
 cl::OptionCategory Cat("helper-to-tcg Options");
@@ -50,6 +53,16 @@ cl::OptionCategory Cat("helper-to-tcg Options");
 cl::opt<std::string> InputFile(cl::Positional, cl::desc("[input LLVM module]"),
                                cl::cat(Cat));
 
+// Debug options
+#ifndef NDEBUG
+static cl::opt<bool> Debug("debug", cl::desc("Enable debug logging (slow)"),
+                           cl::init(false), cl::cat(Cat));
+static cl::opt<std::string>
+    DebugOnly("debug-only",
+              cl::desc("Enable debug logging for a specific pass"),
+              cl::init(""), cl::cat(Cat));
+#endif
+
 // Define a TargetTransformInfo (TTI) subclass, this allows for overriding
 // common per-llvm-target information expected by other LLVM passes, such
 // as the width of the largest scalar/vector registers.  Needed for consistent
@@ -105,8 +118,18 @@ int main(int argc, char **argv) {
 
     cl::ParseCommandLineOptions(argc, argv);
 
-    LLVMContext Context;
+    // Enable debug logging, -debug and -debug-only are normally provided by
+    // LLVMs default debug options, but since we want to hide most default
+    // options and keep them visible during a debug build it's easiest to add
+    // them back manually
+#ifndef NDEBUG
+    DebugFlag = Debug;
+    if (!DebugOnly.empty()) {
+        setCurrentDebugType(DebugOnly.c_str());
+    }
+#endif
 
+    LLVMContext Context;
     SMDiagnostic Err;
     std::unique_ptr<Module> M = parseIRFile(InputFile, Err, Context);
 
-- 
2.52.0

[PATCH v2 11/50] helper-to-tcg: Add simple error creation helper

[Anton Johansson via qemu development]

Simple function for creating Expected<> with nice error messages.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 subprojects/helper-to-tcg/include/Error.hpp | 38 +++++++++++++++++++++
 1 file changed, 38 insertions(+)
 create mode 100644 subprojects/helper-to-tcg/include/Error.hpp

diff --git a/subprojects/helper-to-tcg/include/Error.hpp b/subprojects/helper-to-tcg/include/Error.hpp
new file mode 100644
index 0000000000..835610d83e
--- /dev/null
+++ b/subprojects/helper-to-tcg/include/Error.hpp
@@ -0,0 +1,38 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#pragma once
+
+#include <llvm/IR/Value.h>
+#include <llvm/Support/Error.h>
+
+inline llvm::Error mkError(const llvm::StringRef Msg) {
+    return llvm::createStringError(llvm::inconvertibleErrorCode(), Msg);
+}
+
+// TODO: Usage of `mkError()` and `dbgs()` for serializing `Value`s is
+// __really__ slow, and should only occur for error reporting.
+// `ModuleSlotTracker` could be used which would speed up repeated serialization
+// of a `Value`.
+inline llvm::Error mkError(const llvm::StringRef Msg, const llvm::Value *V) {
+    std::string Str;
+    llvm::raw_string_ostream Stream(Str);
+    Stream << Msg;
+    Stream << *V;
+    Stream.flush();
+    return llvm::createStringError(llvm::inconvertibleErrorCode(), Str);
+}
-- 
2.52.0

[PATCH v2 12/50] helper-to-tcg: Introduce PrepareForOptPass

[Anton Johansson via qemu development]

Adds a new LLVM pass that runs early in the pipeline with the goal
of preparing the input module for optimization by doing some early
culling of functions and information gathering.

This commits sets up a new LLVM pass over the IR module and runs it from
the pipeline.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 .../include/PrepareForOptPass.hpp             | 34 +++++++++++++++++++
 subprojects/helper-to-tcg/meson.build         |  1 +
 subprojects/helper-to-tcg/src/Pipeline.cpp    | 27 +++++++++++++++
 .../PrepareForOptPass/PrepareForOptPass.cpp   | 29 ++++++++++++++++
 4 files changed, 91 insertions(+)
 create mode 100644 subprojects/helper-to-tcg/include/PrepareForOptPass.hpp
 create mode 100644 subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp

diff --git a/subprojects/helper-to-tcg/include/PrepareForOptPass.hpp b/subprojects/helper-to-tcg/include/PrepareForOptPass.hpp
new file mode 100644
index 0000000000..2b3694c536
--- /dev/null
+++ b/subprojects/helper-to-tcg/include/PrepareForOptPass.hpp
@@ -0,0 +1,34 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#pragma once
+
+#include <llvm/IR/PassManager.h>
+
+//
+// PrepareForOptPass
+//
+// Pass that performs either early information collection or basic culling of
+// the input module. simplify the module, or to allow for further optimization.
+//
+
+class PrepareForOptPass : public llvm::PassInfoMixin<PrepareForOptPass> {
+public:
+    PrepareForOptPass() {}
+    llvm::PreservedAnalyses run(llvm::Module &M,
+                                llvm::ModuleAnalysisManager &MAM);
+};
diff --git a/subprojects/helper-to-tcg/meson.build b/subprojects/helper-to-tcg/meson.build
index ca46578cb3..4fd0ddb778 100644
--- a/subprojects/helper-to-tcg/meson.build
+++ b/subprojects/helper-to-tcg/meson.build
@@ -42,6 +42,7 @@ endif
 sources = [
     'src/LlvmCompat.cpp',
     'src/Pipeline.cpp',
+    'src/PrepareForOptPass/PrepareForOptPass.cpp',
 ]
 
 clang = bindir / 'clang'
diff --git a/subprojects/helper-to-tcg/src/Pipeline.cpp b/subprojects/helper-to-tcg/src/Pipeline.cpp
index 4b9523ad6f..59de572bf6 100644
--- a/subprojects/helper-to-tcg/src/Pipeline.cpp
+++ b/subprojects/helper-to-tcg/src/Pipeline.cpp
@@ -17,6 +17,7 @@
 
 #include "CmdLineOptions.hpp"
 #include "LlvmCompat.hpp"
+#include "PrepareForOptPass.hpp"
 
 #if LLVM_VERSION_MAJOR == 15
 #include <llvm/ADT/Triple.h>
@@ -42,6 +43,7 @@
 #include <llvm/Support/SourceMgr.h>
 #include <llvm/Support/TargetSelect.h>
 #include <llvm/Target/TargetMachine.h>
+#include <llvm/Transforms/Scalar/SROA.h>
 
 #define DEBUG_TYPE "pipeline"
 
@@ -165,5 +167,30 @@ int main(int argc, char **argv) {
 
     ModulePassManager MPM;
 
+    //
+    // Start by Filtering out functions we don't want to translate,
+    // following by a pass that removes `noinline`s that are inserted
+    // by clang on -O0. We finally run a UnifyExitNodesPass to make sure
+    // the helpers we parse only has a single exit.
+    //
+
+    {
+        FunctionPassManager FPM;
+#if LLVM_VERSION_MAJOR >= 16
+        FPM.addPass(SROAPass(SROAOptions::ModifyCFG));
+#else
+        FPM.addPass(SROAPass());
+#endif
+        MPM.addPass(createModuleToFunctionPassAdaptor(std::move(FPM)));
+    }
+
+    MPM.addPass(PrepareForOptPass());
+
+    {
+        FunctionPassManager FPM;
+        FPM.addPass(compat::UnifyFunctionExitNodesPass());
+        MPM.addPass(createModuleToFunctionPassAdaptor(std::move(FPM)));
+    }
+
     return 0;
 }
diff --git a/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp b/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp
new file mode 100644
index 0000000000..4a7e82b7bd
--- /dev/null
+++ b/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp
@@ -0,0 +1,29 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#include "PrepareForOptPass.h"
+
+#include <llvm/Support/Debug.h>
+
+#define DEBUG_TYPE "prepare-for-opt"
+
+using namespace llvm;
+
+PreservedAnalyses PrepareForOptPass::run(Module &M,
+                                         ModuleAnalysisManager &MAM) {
+    return PreservedAnalyses::none();
+}
-- 
2.52.0

[PATCH v2 13/50] helper-to-tcg: PrepareForOptPass, demangle function names

[Anton Johansson via qemu development]

Since input LLVM IR might come from languages which mangle function
names, e.g. C++, take extra care to demangle all function names and save
them on the side for future lookup.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 .../PrepareForOptPass/PrepareForOptPass.cpp   | 43 ++++++++++++++++++-
 1 file changed, 42 insertions(+), 1 deletion(-)

diff --git a/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp b/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp
index 4a7e82b7bd..c15c0af6ea 100644
--- a/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp
+++ b/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp
@@ -15,15 +15,56 @@
 //  along with this program; if not, see <http://www.gnu.org/licenses/>.
 //
 
-#include "PrepareForOptPass.h"
+#include "PrepareForOptPass.hpp"
 
+#include <llvm/ADT/StringRef.h>
+#include <llvm/ADT/StringSet.h>
+#include <llvm/Demangle/Demangle.h>
 #include <llvm/Support/Debug.h>
 
 #define DEBUG_TYPE "prepare-for-opt"
 
 using namespace llvm;
 
+static void demangleFunctionNames(Module &M) {
+    StringSet DemangledNames;
+    for (Function &F : M) {
+        const std::string DemangledName = llvm::demangle(F.getName().str());
+        if (DemangledName == F.getName()) {
+            // Name not mangled
+            continue;
+        }
+        // The resulting demangled name might look something like
+        //
+        //   namespace::subnamespace::function(...)
+        //
+        // Extract the function name and use this to replace mangled name.  If
+        // we previously encountered the same name, give up and leave the
+        // mangled name.
+        std::string FunctionName;
+        size_t Index = 0;
+        // Remove namespaces
+        Index = DemangledName.find_last_of(':');
+        if (Index != std::string::npos) {
+            FunctionName = DemangledName.substr(Index + 1);
+        }
+        // Remove arguments
+        Index = FunctionName.find_first_of('(');
+        if (Index != std::string::npos) {
+            FunctionName = FunctionName.substr(0, Index);
+        }
+
+        if (DemangledNames.contains(FunctionName)) {
+            continue;
+        }
+        DemangledNames.insert(FunctionName);
+
+        F.setName(FunctionName);
+    }
+}
+
 PreservedAnalyses PrepareForOptPass::run(Module &M,
                                          ModuleAnalysisManager &MAM) {
+    demangleFunctionNames(M);
     return PreservedAnalyses::none();
 }
-- 
2.52.0

[PATCH v2 14/50] helper-to-tcg: PrepareForOptPass, map annotations

[Anton Johansson via qemu development]

In the LLVM IR module, function annotations are stored in one big global
array of strings.  Traverse this array and parse the data into a format
more useful for future passes.  A map between Functions * and an
`Annotations` structure is exposed.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 .../include/FunctionAnnotation.hpp            | 104 ++++++++++++++++++
 .../include/PrepareForOptPass.hpp             |   7 +-
 subprojects/helper-to-tcg/src/Pipeline.cpp    |   6 +-
 .../PrepareForOptPass/PrepareForOptPass.cpp   |  94 ++++++++++++++++
 4 files changed, 209 insertions(+), 2 deletions(-)
 create mode 100644 subprojects/helper-to-tcg/include/FunctionAnnotation.hpp

diff --git a/subprojects/helper-to-tcg/include/FunctionAnnotation.hpp b/subprojects/helper-to-tcg/include/FunctionAnnotation.hpp
new file mode 100644
index 0000000000..398dd53ef5
--- /dev/null
+++ b/subprojects/helper-to-tcg/include/FunctionAnnotation.hpp
@@ -0,0 +1,104 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#pragma once
+
+#include <llvm/ADT/DenseMap.h>
+#include <llvm/ADT/SmallVector.h>
+#include <llvm/Support/Format.h>
+#include <llvm/Support/raw_ostream.h>
+#include <stdint.h>
+
+namespace llvm {
+class Function;
+}
+
+// Different kind of function annotations which control the behaviour
+// of helper-to-tcg.
+enum class ArgumentAnnotation : uint8_t {
+    // Declares a list of arguments as immediates
+    Immediate = 1,
+    // Declares a list of arguments as vectors, represented by offsets into
+    // the CPU state
+    PtrToOffset = 2,
+};
+
+// Different kind of function annotations which control the behaviour
+// of helper-to-tcg.
+enum class FunctionAnnotation : uint8_t {
+    // Function should be translated
+    HelperToTcg = 1,
+    // Return value of function is an immediate
+    ReturnsImmediate = 2,
+};
+
+// Annotation data which may be attached to a function
+class Annotations {
+    // 8-bit flag for each argument in a function, fields defined by
+    // `ArgumentAnnotions`.
+    llvm::SmallVector<uint8_t, 4> ArgumentAnnotations;
+    // Flag of function annotations, fields defined by `FunctionsAnnotations`.
+    uint8_t FunctionAnnotations = 0;
+
+  public:
+    inline uint8_t getArgFlag(size_t Index) const {
+        if (Index >= ArgumentAnnotations.size()) {
+            return 0;
+        }
+        return ArgumentAnnotations[Index];
+    }
+
+    // Getters and setters for annotations flags.
+
+    inline void set(FunctionAnnotation FA) {
+        FunctionAnnotations |= (uint8_t)FA;
+    }
+
+    inline void set(size_t Index, ArgumentAnnotation AA) {
+        if (Index >= ArgumentAnnotations.size()) {
+            // Resizing will default initialize any new elements.
+            ArgumentAnnotations.resize(Index + 1);
+        }
+        ArgumentAnnotations[Index] |= (uint8_t)AA;
+    }
+
+    inline bool isSet(FunctionAnnotation FA) const {
+        return (FunctionAnnotations & (uint8_t)FA) != 0;
+    }
+
+    inline bool isSet(size_t Index, ArgumentAnnotation AA) const {
+        uint8_t Flag = getArgFlag(Index);
+        return (Flag & (uint8_t)AA) != 0;
+    }
+
+    // Pretty printing debug information
+    inline void dump(llvm::raw_ostream &Out) const {
+        Out << "Annotations:\n";
+        Out << "  Function: " << llvm::format_hex(FunctionAnnotations, 4)
+            << "\n";
+        for (size_t I = 0; I < ArgumentAnnotations.size(); ++I) {
+            const uint8_t Flag = ArgumentAnnotations[I];
+            Out << "  Argument[" << I << "]: " << llvm::format_hex(Flag, 4)
+                << "\n";
+        }
+    }
+};
+
+// Mapping from functions to annotations, this is the main structure to be used
+// by other parts of the codebase when referencing annotations, filled out by
+// `PrepareForOptPass`.
+using AnnotationMapTy = llvm::DenseMap<llvm::Function *, Annotations>;
diff --git a/subprojects/helper-to-tcg/include/PrepareForOptPass.hpp b/subprojects/helper-to-tcg/include/PrepareForOptPass.hpp
index 2b3694c536..e007243578 100644
--- a/subprojects/helper-to-tcg/include/PrepareForOptPass.hpp
+++ b/subprojects/helper-to-tcg/include/PrepareForOptPass.hpp
@@ -17,6 +17,7 @@
 
 #pragma once
 
+#include "FunctionAnnotation.hpp"
 #include <llvm/IR/PassManager.h>
 
 //
@@ -27,8 +28,12 @@
 //
 
 class PrepareForOptPass : public llvm::PassInfoMixin<PrepareForOptPass> {
+    AnnotationMapTy &ResultAnnotations;
 public:
-    PrepareForOptPass() {}
+    PrepareForOptPass(AnnotationMapTy &ResultAnnotations)
+        : ResultAnnotations(ResultAnnotations)
+    {
+    }
     llvm::PreservedAnalyses run(llvm::Module &M,
                                 llvm::ModuleAnalysisManager &MAM);
 };
diff --git a/subprojects/helper-to-tcg/src/Pipeline.cpp b/subprojects/helper-to-tcg/src/Pipeline.cpp
index 59de572bf6..051611b0f3 100644
--- a/subprojects/helper-to-tcg/src/Pipeline.cpp
+++ b/subprojects/helper-to-tcg/src/Pipeline.cpp
@@ -184,7 +184,11 @@ int main(int argc, char **argv) {
         MPM.addPass(createModuleToFunctionPassAdaptor(std::move(FPM)));
     }
 
-    MPM.addPass(PrepareForOptPass());
+    // TODO: Get pass results via dependencies instead? Adds more boiler-plate
+    // but is correlct in LLVM-terms.
+
+    AnnotationMapTy Annotations;
+    MPM.addPass(PrepareForOptPass(Annotations));
 
     {
         FunctionPassManager FPM;
diff --git a/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp b/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp
index c15c0af6ea..1228ac952f 100644
--- a/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp
+++ b/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp
@@ -16,10 +16,15 @@
 //
 
 #include "PrepareForOptPass.hpp"
+#include "Error.hpp"
 
 #include <llvm/ADT/StringRef.h>
 #include <llvm/ADT/StringSet.h>
 #include <llvm/Demangle/Demangle.h>
+#include <llvm/IR/Constants.h>
+#include <llvm/IR/Function.h>
+#include <llvm/IR/Instruction.h>
+#include <llvm/IR/Module.h>
 #include <llvm/Support/Debug.h>
 
 #define DEBUG_TYPE "prepare-for-opt"
@@ -63,8 +68,97 @@ static void demangleFunctionNames(Module &M) {
     }
 }
 
+static Error parseAnnotationStr(Annotations &Ann, StringRef Str,
+                                size_t NumArgs) {
+    Str = Str.trim();
+
+    // Function annotations
+    if (Str.consume_front("helper-to-tcg")) {
+        Ann.set(FunctionAnnotation::HelperToTcg);
+        return Error::success();
+    } else if (Str.consume_front("returns-immediate")) {
+        Ann.set(FunctionAnnotation::ReturnsImmediate);
+        return Error::success();
+    }
+
+    // Argument annotations
+    ArgumentAnnotation AA;
+    if (Str.consume_front("immediate")) {
+        AA = ArgumentAnnotation::Immediate;
+    } else if (Str.consume_front("ptr-to-offset")) {
+        AA = ArgumentAnnotation::PtrToOffset;
+    } else {
+        return mkError("Unknown annotation");
+    }
+
+    // An argument annotation looks like
+    //
+    //  "immediate: 0, 1, 2",
+    //
+    // parse the comma separated list of argument indices.
+    if (!Str.consume_front(":")) {
+        return mkError("Expected \":\"");
+    }
+    Str = Str.ltrim(' ');
+    do {
+        Str = Str.ltrim(' ');
+        size_t I = 0;
+        Str.consumeInteger(10, I);
+        if (I >= NumArgs) {
+            return mkError("Annotation has out of bounds argument index");
+        }
+        Ann.set(I, AA);
+    } while (Str.consume_front(","));
+
+    return Error::success();
+}
+
+static void collectAnnotations(Module &M, AnnotationMapTy &ResultAnnotations) {
+    // cast over dyn_cast is being used here to
+    // assert that the structure of
+    //
+    //     llvm.global.annotation
+    //
+    // is what we expect.
+
+    GlobalVariable *GA = M.getGlobalVariable("llvm.global.annotations");
+    if (!GA) {
+        return;
+    }
+
+    // Get the metadata which is stored in the first op
+    auto *CA = cast<ConstantArray>(GA->getOperand(0));
+    // Loop over metadata
+    for (Value *CAOp : CA->operands()) {
+        auto *Struct = cast<ConstantStruct>(CAOp);
+        assert(Struct->getNumOperands() >= 2);
+
+        Function *F = cast<Function>(Struct->getOperand(0));
+        ConstantDataArray *AnnData =
+            cast<ConstantDataArray>(Struct->getOperand(1)->getOperand(0));
+
+        StringRef AnnStr = AnnData->getAsString();
+        AnnStr = AnnStr.substr(0, AnnStr.size() - 1);
+        Annotations Ann = ResultAnnotations[F];
+        if (auto Err = parseAnnotationStr(Ann, AnnStr, F->arg_size()); Err) {
+            errs() << "Failed to parse annotation: \"" << Err
+                   << "\" for function " << F->getName() << "\n";
+            continue;
+        }
+        ResultAnnotations[F] = Ann;
+    }
+
+    LLVM_DEBUG({
+        for (auto &P : ResultAnnotations) {
+            dbgs() << "Annotations for " << P.first->getName() << "\n";
+            P.second.dump(dbgs());
+        }
+    });
+}
+
 PreservedAnalyses PrepareForOptPass::run(Module &M,
                                          ModuleAnalysisManager &MAM) {
     demangleFunctionNames(M);
+    collectAnnotations(M, ResultAnnotations);
     return PreservedAnalyses::none();
 }
-- 
2.52.0

[PATCH v2 15/50] helper-to-tcg: PrepareForOptPass, cull unused functions

[Anton Johansson via qemu development]

Make an early pass over all functions in the input module and filter out
functions with:

  1. Invalid return type, or;
  2. No helper-to-tcg annotation and not called by a function with such
     a annotation.

A commandline option is also added to force translation of all functions
starting with "helper_".

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 .../helper-to-tcg/include/CmdLineOptions.hpp  |  2 +
 .../include/PrepareForOptPass.hpp             |  7 +-
 subprojects/helper-to-tcg/src/Pipeline.cpp    |  5 ++
 .../PrepareForOptPass/PrepareForOptPass.cpp   | 86 +++++++++++++++++++
 4 files changed, 96 insertions(+), 4 deletions(-)

diff --git a/subprojects/helper-to-tcg/include/CmdLineOptions.hpp b/subprojects/helper-to-tcg/include/CmdLineOptions.hpp
index 93706b78c5..ca1cb59835 100644
--- a/subprojects/helper-to-tcg/include/CmdLineOptions.hpp
+++ b/subprojects/helper-to-tcg/include/CmdLineOptions.hpp
@@ -21,3 +21,5 @@
 
 // Options for pipeline
 extern llvm::cl::list<std::string> InputFiles;
+// Options for PrepareForOptPass
+extern llvm::cl::opt<bool> TranslateAllHelpers;
diff --git a/subprojects/helper-to-tcg/include/PrepareForOptPass.hpp b/subprojects/helper-to-tcg/include/PrepareForOptPass.hpp
index e007243578..08ca9a43bb 100644
--- a/subprojects/helper-to-tcg/include/PrepareForOptPass.hpp
+++ b/subprojects/helper-to-tcg/include/PrepareForOptPass.hpp
@@ -29,11 +29,10 @@
 
 class PrepareForOptPass : public llvm::PassInfoMixin<PrepareForOptPass> {
     AnnotationMapTy &ResultAnnotations;
-public:
+
+  public:
     PrepareForOptPass(AnnotationMapTy &ResultAnnotations)
-        : ResultAnnotations(ResultAnnotations)
-    {
-    }
+        : ResultAnnotations(ResultAnnotations) {}
     llvm::PreservedAnalyses run(llvm::Module &M,
                                 llvm::ModuleAnalysisManager &MAM);
 };
diff --git a/subprojects/helper-to-tcg/src/Pipeline.cpp b/subprojects/helper-to-tcg/src/Pipeline.cpp
index 051611b0f3..89637eaec6 100644
--- a/subprojects/helper-to-tcg/src/Pipeline.cpp
+++ b/subprojects/helper-to-tcg/src/Pipeline.cpp
@@ -65,6 +65,11 @@ static cl::opt<std::string>
               cl::init(""), cl::cat(Cat));
 #endif
 
+// Options for PrepareForOptPass
+cl::opt<bool> TranslateAllHelpers(
+    "translate-all-helpers", cl::init(false),
+    cl::desc("Translate all functions starting with helper_*"), cl::cat(Cat));
+
 // Define a TargetTransformInfo (TTI) subclass, this allows for overriding
 // common per-llvm-target information expected by other LLVM passes, such
 // as the width of the largest scalar/vector registers.  Needed for consistent
diff --git a/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp b/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp
index 1228ac952f..df6d9eeec8 100644
--- a/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp
+++ b/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp
@@ -16,17 +16,25 @@
 //
 
 #include "PrepareForOptPass.hpp"
+#include "CmdLineOptions.hpp"
 #include "Error.hpp"
+#include "FunctionAnnotation.hpp"
+#include "LlvmCompat.hpp"
 
+#include <llvm/ADT/SmallPtrSet.h>
 #include <llvm/ADT/StringRef.h>
 #include <llvm/ADT/StringSet.h>
 #include <llvm/Demangle/Demangle.h>
 #include <llvm/IR/Constants.h>
 #include <llvm/IR/Function.h>
 #include <llvm/IR/Instruction.h>
+#include <llvm/IR/Instructions.h>
 #include <llvm/IR/Module.h>
 #include <llvm/Support/Debug.h>
 
+#include <queue>
+#include <set>
+
 #define DEBUG_TYPE "prepare-for-opt"
 
 using namespace llvm;
@@ -156,9 +164,87 @@ static void collectAnnotations(Module &M, AnnotationMapTy &ResultAnnotations) {
     });
 }
 
+inline bool hasValidReturnTy(const Module &M, const Function *F) {
+    Type *RetTy = F->getReturnType();
+    return RetTy->isStructTy() || RetTy == Type::getVoidTy(F->getContext()) ||
+           RetTy == Type::getInt8Ty(M.getContext()) ||
+           RetTy == Type::getInt16Ty(M.getContext()) ||
+           RetTy == Type::getInt32Ty(M.getContext()) ||
+           RetTy == Type::getInt64Ty(M.getContext());
+}
+
+// Functions that should be removed:
+//   - No helper-to-tcg annotation (if TranslateAllHelpers == false);
+//   - Invalid (non-integer/void) return type
+static bool shouldRemoveFunction(const Module &M, const Function &F,
+                                 const AnnotationMapTy &AnnotationMap) {
+    if (F.isDeclaration()) {
+        return false;
+    }
+
+    if (!hasValidReturnTy(M, &F)) {
+        return true;
+    }
+
+    std::queue<const Function *> Worklist;
+    std::set<const Function *> Visited;
+    Worklist.push(&F);
+    while (!Worklist.empty()) {
+        const Function *F = Worklist.front();
+        Worklist.pop();
+        if (F->isDeclaration() or Visited.find(F) != Visited.end()) {
+            continue;
+        }
+        Visited.insert(F);
+
+        if (TranslateAllHelpers and
+            compat::isFunctionQemuHelper(F->getName())) {
+            // If --translate-all-helpers is provided and `F` starts with
+            // "helper_*", then don't skip it.
+            return false;
+        } else if (auto It = AnnotationMap.find(F); It != AnnotationMap.end()) {
+            // Otherwise check "helper-to-tcg" annotation.
+            const Annotations &Ann = It->second;
+            if (Ann.isSet(FunctionAnnotation::HelperToTcg)) {
+                return false;
+            }
+        }
+
+        // Push functions that call `F` to the worklist, this way we retain
+        // functions that are being called by functions with the "helper-to-tcg"
+        // annotation.
+        for (const User *U : F->users()) {
+            auto Call = dyn_cast<CallInst>(U);
+            if (!Call) {
+                continue;
+            }
+            const Function *ParentF = Call->getParent()->getParent();
+            Worklist.push(ParentF);
+        }
+    }
+
+    return true;
+}
+
+static void cullUnusedFunctions(Module &M, AnnotationMapTy &Annotations) {
+    SmallPtrSet<Function *, 16> FunctionsToRemove;
+    for (auto &F : M) {
+        if (shouldRemoveFunction(M, F, Annotations)) {
+            FunctionsToRemove.insert(&F);
+        }
+    }
+
+    for (Function *F : FunctionsToRemove) {
+        Annotations.erase(F);
+        F->setComdat(nullptr);
+        F->deleteBody();
+    }
+}
+
 PreservedAnalyses PrepareForOptPass::run(Module &M,
                                          ModuleAnalysisManager &MAM) {
     demangleFunctionNames(M);
     collectAnnotations(M, ResultAnnotations);
+    cullUnusedFunctions(M, ResultAnnotations);
     return PreservedAnalyses::none();
 }
-- 
2.52.0

[PATCH v2 16/50] helper-to-tcg: PrepareForOptPass, undef llvm.returnaddress

[Anton Johansson via qemu development]

Convert llvm.returnaddress arguments to cpu_[ld|st]*() to undef, causing
the LLVM optmizer to discard the intrinsics.  Needed as
llvm.returnadress is not representable in TCG, and usually results from
usage of GETPC() in helper functions.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 .../PrepareForOptPass/PrepareForOptPass.cpp   | 47 +++++++++++++++++++
 1 file changed, 47 insertions(+)

diff --git a/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp b/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp
index df6d9eeec8..157f8cd05e 100644
--- a/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp
+++ b/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp
@@ -29,6 +29,7 @@
 #include <llvm/IR/Function.h>
 #include <llvm/IR/Instruction.h>
 #include <llvm/IR/Instructions.h>
+#include <llvm/IR/Intrinsics.h>
 #include <llvm/IR/Module.h>
 #include <llvm/Support/Debug.h>
 
@@ -241,10 +242,56 @@ static void cullUnusedFunctions(Module &M, AnnotationMapTy &Annotations) {
     }
 }
 
+struct RetAddrReplaceInfo {
+    User *Parent;
+    unsigned OpIndex;
+    Type *Ty;
+};
+
+static void replaceRetaddrWithUndef(Module &M) {
+    // Replace uses of llvm.returnaddress arguments to cpu_ld* w. undef,
+    // and let optimizations remove it.  Needed as llvm.returnaddress is
+    // not reprensentable in TCG.
+    SmallVector<RetAddrReplaceInfo, 24> UsesToReplace;
+    Function *Retaddr = compat::Intrinsic::getOrInsertDeclaration(
+        &M, Intrinsic::returnaddress, {});
+    // Loop over all calls to llvm.returnaddress
+    for (auto *CallUser : Retaddr->users()) {
+        auto *Call = dyn_cast<CallInst>(CallUser);
+        if (!Call) {
+            continue;
+        }
+        for (auto *PtrToIntUser : Call->users()) {
+            auto *Cast = dyn_cast<PtrToIntInst>(PtrToIntUser);
+            if (!Cast) {
+                continue;
+            }
+            for (Use &U : Cast->uses()) {
+                auto *Call = dyn_cast<CallInst>(U.getUser());
+                Function *F = Call->getCalledFunction();
+                if (compat::isFunctionQemuLoadStore(F->getName())) {
+                    UsesToReplace.push_back({
+                        .Parent = U.getUser(),
+                        .OpIndex = U.getOperandNo(),
+                        .Ty = U->getType(),
+                    });
+                }
+            }
+        }
+    }
+
+    // Defer replacement to not invalidate iterators
+    for (RetAddrReplaceInfo &RI : UsesToReplace) {
+        auto *Undef = UndefValue::get(RI.Ty);
+        RI.Parent->setOperand(RI.OpIndex, Undef);
+    }
+}
+
 PreservedAnalyses PrepareForOptPass::run(Module &M,
                                          ModuleAnalysisManager &MAM) {
     demangleFunctionNames(M);
     collectAnnotations(M, ResultAnnotations);
     cullUnusedFunctions(M, ResultAnnotations);
+    replaceRetaddrWithUndef(M);
     return PreservedAnalyses::none();
 }
-- 
2.52.0

[PATCH v2 17/50] helper-to-tcg: PrepareForOptPass, fixup inline attributes

[Anton Johansson via qemu development]

When producing LLVM IR using clang -O0, a noinline attribute is added.
Remove this attribute to not inhibit future optimization.  Also try and
force functions returning struct type to be inlined so they might be
translated.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 .../src/PrepareForOptPass/PrepareForOptPass.cpp    | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp b/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp
index 157f8cd05e..a7bee53582 100644
--- a/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp
+++ b/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp
@@ -32,6 +32,7 @@
 #include <llvm/IR/Intrinsics.h>
 #include <llvm/IR/Module.h>
 #include <llvm/Support/Debug.h>
+#include <llvm/Transforms/Utils/Local.h>
 
 #include <queue>
 #include <set>
@@ -293,5 +294,18 @@ PreservedAnalyses PrepareForOptPass::run(Module &M,
     collectAnnotations(M, ResultAnnotations);
     cullUnusedFunctions(M, ResultAnnotations);
     replaceRetaddrWithUndef(M);
+    // Remove noinline function attributes automatically added by -O0, add
+    // alwaysinline attribute to functions with a struct return value, these
+    // can not be translated to TCG currently and we rely on inlining to
+    // hopefully get rid of them.
+    for (Function &F : M) {
+        if (F.hasFnAttribute(Attribute::AttrKind::NoInline)) {
+            F.removeFnAttr(Attribute::AttrKind::NoInline);
+        }
+        if (F.getReturnType()->isStructTy()) {
+            F.addFnAttr(Attribute::AttrKind::AlwaysInline);
+        }
+    }
+
     return PreservedAnalyses::none();
 }
-- 
2.52.0

[PATCH v2 18/50] helper-to-tcg: PrepareForOptPass, collect debuginfo

[Anton Johansson via qemu development]

Debug information is collected before any non-preserving optimization
pass is ran.  Currently, only variable definitions are parsed, mapping a
Value * to a name and typename.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 .../helper-to-tcg/include/DebugInfo.hpp       | 46 ++++++++++++++++
 .../include/PrepareForOptPass.hpp             | 10 ++--
 subprojects/helper-to-tcg/src/Pipeline.cpp    |  3 +-
 .../PrepareForOptPass/PrepareForOptPass.cpp   | 52 +++++++++++++++++++
 4 files changed, 107 insertions(+), 4 deletions(-)
 create mode 100644 subprojects/helper-to-tcg/include/DebugInfo.hpp

diff --git a/subprojects/helper-to-tcg/include/DebugInfo.hpp b/subprojects/helper-to-tcg/include/DebugInfo.hpp
new file mode 100644
index 0000000000..27e545c6b7
--- /dev/null
+++ b/subprojects/helper-to-tcg/include/DebugInfo.hpp
@@ -0,0 +1,46 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#pragma once
+
+#include <llvm/ADT/StringRef.h>
+#include <llvm/IR/ValueMap.h>
+
+namespace llvm {
+class Value;
+}
+
+// StringRefs will refer do debug metadata fields which have the same
+// lifetime as the LLVMContext and survive accross optimizations and
+// possible deletions of functions/variables.
+struct DebugInfo {
+    llvm::StringRef VarName;
+    llvm::StringRef BaseTypeName;
+};
+
+using DebugInfoMapTy = llvm::ValueMap<const llvm::Value *, DebugInfo>;
+
+// Helper to get the variable name from debug info associated with a particular
+// value, or default construct an empty name.
+inline llvm::StringRef getDebugVarName(const DebugInfoMapTy &DM,
+                                       const llvm::Value *V) {
+    auto It = DM.find(V);
+    if (It != DM.end()) {
+        return It->second.VarName;
+    }
+    return {};
+}
diff --git a/subprojects/helper-to-tcg/include/PrepareForOptPass.hpp b/subprojects/helper-to-tcg/include/PrepareForOptPass.hpp
index 08ca9a43bb..43cf77190a 100644
--- a/subprojects/helper-to-tcg/include/PrepareForOptPass.hpp
+++ b/subprojects/helper-to-tcg/include/PrepareForOptPass.hpp
@@ -17,6 +17,7 @@
 
 #pragma once
 
+#include "DebugInfo.hpp"
 #include "FunctionAnnotation.hpp"
 #include <llvm/IR/PassManager.h>
 
@@ -29,10 +30,13 @@
 
 class PrepareForOptPass : public llvm::PassInfoMixin<PrepareForOptPass> {
     AnnotationMapTy &ResultAnnotations;
+    DebugInfoMapTy &ResultDebugInfo;
 
-  public:
-    PrepareForOptPass(AnnotationMapTy &ResultAnnotations)
-        : ResultAnnotations(ResultAnnotations) {}
+public:
+    PrepareForOptPass(AnnotationMapTy &ResultAnnotations,
+                      DebugInfoMapTy &ResultDebugInfo)
+        : ResultAnnotations(ResultAnnotations),
+          ResultDebugInfo(ResultDebugInfo) {}
     llvm::PreservedAnalyses run(llvm::Module &M,
                                 llvm::ModuleAnalysisManager &MAM);
 };
diff --git a/subprojects/helper-to-tcg/src/Pipeline.cpp b/subprojects/helper-to-tcg/src/Pipeline.cpp
index 89637eaec6..a11c5fd353 100644
--- a/subprojects/helper-to-tcg/src/Pipeline.cpp
+++ b/subprojects/helper-to-tcg/src/Pipeline.cpp
@@ -193,7 +193,8 @@ int main(int argc, char **argv) {
     // but is correlct in LLVM-terms.
 
     AnnotationMapTy Annotations;
-    MPM.addPass(PrepareForOptPass(Annotations));
+    DebugInfoMapTy DebugInfo;
+    MPM.addPass(PrepareForOptPass(Annotations, DebugInfo));
 
     {
         FunctionPassManager FPM;
diff --git a/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp b/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp
index a7bee53582..7a9b954d7c 100644
--- a/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp
+++ b/subprojects/helper-to-tcg/src/PrepareForOptPass/PrepareForOptPass.cpp
@@ -26,6 +26,9 @@
 #include <llvm/ADT/StringSet.h>
 #include <llvm/Demangle/Demangle.h>
 #include <llvm/IR/Constants.h>
+#if LLVM_VERSION_MAJOR >= 19
+#include <llvm/IR/DebugProgramInstruction.h>
+#endif
 #include <llvm/IR/Function.h>
 #include <llvm/IR/Instruction.h>
 #include <llvm/IR/Instructions.h>
@@ -288,6 +291,53 @@ static void replaceRetaddrWithUndef(Module &M) {
     }
 }
 
+static void collectDebugInfo(Function &F, DebugInfoMapTy &ResultDebugInfo) {
+    StringSet EncounteredNames;
+    for (auto &BB : F) {
+        for (Instruction &I : BB) {
+#if LLVM_VERSION_MAJOR >= 19
+            for (DbgVariableRecord &DVR :
+                 filterDbgVars(I.getDbgRecordRange())) {
+                if (!DVR.isDbgValue()) {
+                    continue;
+                }
+                StringRef BaseType{};
+                StringRef VarName{};
+                if (auto *Derived =
+                        dyn_cast<DIDerivedType>(DVR.getVariable()->getType())) {
+                    BaseType = Derived->getBaseType()->getName();
+                }
+                VarName = DVR.getVariable()->getName();
+                if (ResultDebugInfo.find(DVR.getValue(0)) ==
+                        ResultDebugInfo.end() and
+                    !EncounteredNames.contains(VarName.data())) {
+                    ResultDebugInfo[DVR.getValue(0)] = {VarName, BaseType};
+                    EncounteredNames.insert(VarName.data());
+                }
+            }
+#else
+            if (I.isDebugOrPseudoInst()) {
+                if (const auto *Dbg = dyn_cast<DbgValueInst>(&I)) {
+                    StringRef BaseType{};
+                    StringRef VarName{};
+                    if (auto *Derived = dyn_cast<DIDerivedType>(
+                            Dbg->getVariable()->getType())) {
+                        BaseType = Derived->getBaseType()->getName();
+                    }
+                    VarName = Dbg->getVariable()->getName();
+                    if (ResultDebugInfo.find(Dbg->getValue(0)) ==
+                            ResultDebugInfo.end() and
+                        !EncounteredNames.contains(VarName.data())) {
+                        ResultDebugInfo[Dbg->getValue(0)] = {VarName, BaseType};
+                        EncounteredNames.insert(VarName.data());
+                    }
+                }
+            }
+#endif
+        }
+    }
+}
+
 PreservedAnalyses PrepareForOptPass::run(Module &M,
                                          ModuleAnalysisManager &MAM) {
     demangleFunctionNames(M);
@@ -305,6 +355,8 @@ PreservedAnalyses PrepareForOptPass::run(Module &M,
         if (F.getReturnType()->isStructTy()) {
             F.addFnAttr(Attribute::AttrKind::AlwaysInline);
         }
+        // Populate variable and type names for `Value`s from debug info.
+        collectDebugInfo(F, ResultDebugInfo);
     }
 
     return PreservedAnalyses::none();
-- 
2.52.0

[PATCH v2 19/50] helper-to-tcg: Pipeline, run optimization pass

[Anton Johansson via qemu development]

Run a standard LLVM -Oz optimization pass, which makes up the bulk of
optimizations in helper-to-tcg.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 subprojects/helper-to-tcg/src/Pipeline.cpp | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/subprojects/helper-to-tcg/src/Pipeline.cpp b/subprojects/helper-to-tcg/src/Pipeline.cpp
index a11c5fd353..571276de24 100644
--- a/subprojects/helper-to-tcg/src/Pipeline.cpp
+++ b/subprojects/helper-to-tcg/src/Pipeline.cpp
@@ -202,5 +202,17 @@ int main(int argc, char **argv) {
         MPM.addPass(createModuleToFunctionPassAdaptor(std::move(FPM)));
     }
 
+    //
+    // Run a -Oz optimization pass.  In general -Oz will prefer loop
+    // vectorization over unrolling, as compared to -O3.  In TCG, this
+    // translates to more utilization of gvec and possibly smaller TBs.
+    //
+
+    // Optimization passes
+    MPM.addPass(PB.buildModuleSimplificationPipeline(
+        compat::OptimizationLevel::Oz, compat::LTOPhase));
+    MPM.addPass(
+        PB.buildModuleOptimizationPipeline(compat::OptimizationLevel::Oz, {}));
+
     return 0;
 }
-- 
2.52.0

[PATCH v2 20/50] helper-to-tcg: Introduce pseudo instructions

[Anton Johansson via qemu development]

"pseudo" instructions makes it easy to add custom instructions to
LLVM IR in the form of calls to undefined functions.  These will be used
in future commits to express functionality present in TCG that is missing
from LLVM IR (certain vector ops.), or to simplify the backend by
collecting similar instruction mappings into a single opcode
(idendity mapping).

Mapping from a call instructions in LLVM IR to an enum representing the
pseudo instruction is also handled, this avoids string comparisons in
the backend, and is easy to switch over.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 .../helper-to-tcg/include/PseudoInst.hpp      |  75 +++++++
 .../helper-to-tcg/include/PseudoInst.inc      |  78 ++++++++
 subprojects/helper-to-tcg/meson.build         |   1 +
 subprojects/helper-to-tcg/src/PseudoInst.cpp  | 183 ++++++++++++++++++
 4 files changed, 337 insertions(+)
 create mode 100644 subprojects/helper-to-tcg/include/PseudoInst.hpp
 create mode 100644 subprojects/helper-to-tcg/include/PseudoInst.inc
 create mode 100644 subprojects/helper-to-tcg/src/PseudoInst.cpp

diff --git a/subprojects/helper-to-tcg/include/PseudoInst.hpp b/subprojects/helper-to-tcg/include/PseudoInst.hpp
new file mode 100644
index 0000000000..ef0b03cb5b
--- /dev/null
+++ b/subprojects/helper-to-tcg/include/PseudoInst.hpp
@@ -0,0 +1,75 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#pragma once
+
+#include <stdint.h>
+
+#include <llvm/ADT/ArrayRef.h>
+#include <llvm/IR/IRBuilder.h>
+#include <llvm/IR/Module.h>
+#include <llvm/IR/Value.h>
+
+// Pseudo instructions refers to extra LLVM instructions implemented as
+// calls to undefined functions.  They are useful for amending LLVM IR to
+// simplify mapping to TCG in the backend, e.g.
+//
+//   %2 = call i32 @IdentityMap.i32.i16(i16 %1)
+//
+// is a pseudo opcode used to communicate that %1 and %2 should be mapped
+// to the same value in TCG.
+
+enum PseudoInstArg {
+    ArgAny,
+    ArgInt,
+    ArgVec,
+    ArgPtr,
+    ArgLabel,
+    ArgVoid,
+};
+
+// Define an enum `PseudoInst` consisting of the names of pseudo instructions
+// defined in "PseudoInst.inc".
+#define PSEUDO_INST_DEF(name, ret, args) name
+enum PseudoInst : uint8_t {
+#include "PseudoInst.inc"
+};
+#undef PSEUDO_INST_DEF
+
+// Retrieve string representation and argument counts for a given
+// pseudo instruction.
+const char *pseudoInstName(PseudoInst Inst);
+uint8_t pseudoInstArgCount(PseudoInst Inst);
+llvm::ArrayRef<PseudoInstArg> pseudoInstArgTypes(PseudoInst Inst);
+
+// Maps `PseudoInst`, return type, and argument types, to a `FunctionCallee`
+// that can be called.  Provided argument types are matched against
+// "PseudoInst.inc" for verification and an assert triggers on failure.
+llvm::FunctionCallee pseudoInstFunction(llvm::Module &M, PseudoInst Inst,
+                                        llvm::Type *RetType,
+                                        llvm::ArrayRef<llvm::Type *> ArgTypes);
+
+// Convenience function that gets a pseudo instruction via
+// `pseudoInstFunction()` and creates a call to it via the provided `IRBuilder`.
+llvm::CallInst *createPseudoInstCall(llvm::Module &M,
+                                     llvm::IRBuilder<> &Builder,
+                                     PseudoInst Inst, llvm::Type *RetType,
+                                     llvm::ArrayRef<llvm::Value *> Values);
+
+// Reverse mapping of above, takes a call instruction and attempts to map the
+// callee to a `PseudoInst`.
+PseudoInst getPseudoInstFromCall(const llvm::CallInst *Call);
diff --git a/subprojects/helper-to-tcg/include/PseudoInst.inc b/subprojects/helper-to-tcg/include/PseudoInst.inc
new file mode 100644
index 0000000000..003d3fb186
--- /dev/null
+++ b/subprojects/helper-to-tcg/include/PseudoInst.inc
@@ -0,0 +1,78 @@
+PSEUDO_INST_DEF(InvalidPseudoInst,  ArgVoid, PSEUDO_INST_ARGVEC()),
+// Identity mapping
+PSEUDO_INST_DEF(IdentityMap,        ArgAny, PSEUDO_INST_ARGVEC(ArgAny)),
+// Pointer arithmetic
+PSEUDO_INST_DEF(PtrAdd,             ArgPtr, PSEUDO_INST_ARGVEC(ArgPtr, ArgInt)),
+// Global accesses
+PSEUDO_INST_DEF(AccessGlobalArray,  ArgInt, PSEUDO_INST_ARGVEC(ArgInt, ArgInt, ArgInt)),
+PSEUDO_INST_DEF(AccessGlobalValue,  ArgInt, PSEUDO_INST_ARGVEC(ArgInt, ArgInt)),
+// Conditional branch
+PSEUDO_INST_DEF(Brcond,             ArgVoid, PSEUDO_INST_ARGVEC(ArgInt, ArgInt, ArgInt, ArgInt, ArgLabel, ArgLabel)),
+// Conditional move
+PSEUDO_INST_DEF(Movcond,            ArgInt, PSEUDO_INST_ARGVEC(ArgInt, ArgInt, ArgInt, ArgInt, ArgInt)),
+// Vector creation ops
+PSEUDO_INST_DEF(VecSplat,           ArgVec, PSEUDO_INST_ARGVEC(ArgInt)),
+PSEUDO_INST_DEF(VecConstant,        ArgVec, PSEUDO_INST_ARGVEC(ArgVec)),
+// Vector unary ops
+PSEUDO_INST_DEF(VecNot,             ArgVec, PSEUDO_INST_ARGVEC(ArgVec)),
+// Vector scalar binary ops
+PSEUDO_INST_DEF(VecAddScalar,       ArgVec, PSEUDO_INST_ARGVEC(ArgVec, ArgInt)),
+PSEUDO_INST_DEF(VecSubScalar,       ArgVec, PSEUDO_INST_ARGVEC(ArgVec, ArgInt)),
+PSEUDO_INST_DEF(VecMulScalar,       ArgVec, PSEUDO_INST_ARGVEC(ArgVec, ArgInt)),
+PSEUDO_INST_DEF(VecXorScalar,       ArgVec, PSEUDO_INST_ARGVEC(ArgVec, ArgInt)),
+PSEUDO_INST_DEF(VecOrScalar,        ArgVec, PSEUDO_INST_ARGVEC(ArgVec, ArgInt)),
+PSEUDO_INST_DEF(VecAndScalar,       ArgVec, PSEUDO_INST_ARGVEC(ArgVec, ArgInt)),
+PSEUDO_INST_DEF(VecShlScalar,       ArgVec, PSEUDO_INST_ARGVEC(ArgVec, ArgInt)),
+PSEUDO_INST_DEF(VecLShrScalar,      ArgVec, PSEUDO_INST_ARGVEC(ArgVec, ArgInt)),
+PSEUDO_INST_DEF(VecAShrScalar,      ArgVec, PSEUDO_INST_ARGVEC(ArgVec, ArgInt)),
+// Vector unary ops that stores to pointer
+PSEUDO_INST_DEF(VecNotStore,        ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec)),
+// Vector binary ops that stores to pointer
+PSEUDO_INST_DEF(VecAddStore,        ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgVec)),
+PSEUDO_INST_DEF(VecSubStore,        ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgVec)),
+PSEUDO_INST_DEF(VecMulStore,        ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgVec)),
+PSEUDO_INST_DEF(VecXorStore,        ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgVec)),
+PSEUDO_INST_DEF(VecOrStore,         ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgVec)),
+PSEUDO_INST_DEF(VecAndStore,        ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgVec)),
+PSEUDO_INST_DEF(VecShlStore,        ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgVec)),
+PSEUDO_INST_DEF(VecLShrStore,       ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgVec)),
+PSEUDO_INST_DEF(VecAShrStore,       ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgVec)),
+PSEUDO_INST_DEF(VecAddScalarStore,  ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgInt)),
+PSEUDO_INST_DEF(VecSubScalarStore,  ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgInt)),
+PSEUDO_INST_DEF(VecMulScalarStore,  ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgInt)),
+PSEUDO_INST_DEF(VecXorScalarStore,  ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgInt)),
+PSEUDO_INST_DEF(VecOrScalarStore,   ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgInt)),
+PSEUDO_INST_DEF(VecAndScalarStore,  ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgInt)),
+PSEUDO_INST_DEF(VecShlScalarStore,  ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgInt)),
+PSEUDO_INST_DEF(VecLShrScalarStore, ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgInt)),
+PSEUDO_INST_DEF(VecAShrScalarStore, ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgInt)),
+// Host memory operations
+//                                                      vaddr,  value   sign    size    endian
+PSEUDO_INST_DEF(GuestLoad,  ArgInt,  PSEUDO_INST_ARGVEC(ArgInt,         ArgInt, ArgInt, ArgInt)),
+PSEUDO_INST_DEF(GuestStore, ArgVoid, PSEUDO_INST_ARGVEC(ArgInt, ArgInt,         ArgInt, ArgInt)),
+// ...
+PSEUDO_INST_DEF(VecTruncStore,        ArgVoid, PSEUDO_INST_ARGVEC(ArgInt, ArgPtr, ArgVec)),
+PSEUDO_INST_DEF(VecZExtStore,         ArgVoid, PSEUDO_INST_ARGVEC(ArgInt, ArgPtr, ArgVec)),
+PSEUDO_INST_DEF(VecSExtStore,         ArgVoid, PSEUDO_INST_ARGVEC(ArgInt, ArgPtr, ArgVec)),
+PSEUDO_INST_DEF(VecSignedSatAddStore, ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgVec)),
+PSEUDO_INST_DEF(VecSignedSatSubStore, ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgVec)),
+PSEUDO_INST_DEF(VecSelectStore,       ArgVoid, PSEUDO_INST_ARGVEC(ArgInt, ArgPtr, ArgVec, ArgVec, ArgVec)),
+PSEUDO_INST_DEF(VecFunnelShrStore,    ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgVec, ArgVec)),
+PSEUDO_INST_DEF(VecAbsStore,          ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec)),
+PSEUDO_INST_DEF(VecSignedMaxStore,    ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgVec)),
+PSEUDO_INST_DEF(VecUnsignedMaxStore,  ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgVec)),
+PSEUDO_INST_DEF(VecSignedMinStore,    ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgVec)),
+PSEUDO_INST_DEF(VecUnsignedMinStore,  ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgVec)),
+PSEUDO_INST_DEF(VecCtlzStore,         ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec)),
+PSEUDO_INST_DEF(VecCttzStore,         ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec)),
+PSEUDO_INST_DEF(VecCtpopStore,        ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec)),
+PSEUDO_INST_DEF(VecWideCondBitsel,    ArgVec, PSEUDO_INST_ARGVEC(ArgVec, ArgVec, ArgVec)),
+PSEUDO_INST_DEF(VecWideCondBitselStore,    ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgVec, ArgVec, ArgVec)),
+PSEUDO_INST_DEF(VecCompare,           ArgVec, PSEUDO_INST_ARGVEC(ArgInt, ArgVec, ArgVec)),
+PSEUDO_INST_DEF(VecSelect,            ArgVec, PSEUDO_INST_ARGVEC(ArgInt, ArgVec, ArgVec)),
+
+PSEUDO_INST_DEF(SignExtract,          ArgInt, PSEUDO_INST_ARGVEC(ArgInt, ArgInt, ArgInt)),
+PSEUDO_INST_DEF(Extract,              ArgInt, PSEUDO_INST_ARGVEC(ArgInt, ArgInt, ArgInt)),
+
+PSEUDO_INST_DEF(Exception,            ArgVoid, PSEUDO_INST_ARGVEC(ArgPtr, ArgInt)),
+PSEUDO_INST_DEF(GetPC,                ArgInt,  PSEUDO_INST_ARGVEC()),
diff --git a/subprojects/helper-to-tcg/meson.build b/subprojects/helper-to-tcg/meson.build
index 4fd0ddb778..b3a3de6297 100644
--- a/subprojects/helper-to-tcg/meson.build
+++ b/subprojects/helper-to-tcg/meson.build
@@ -43,6 +43,7 @@ sources = [
     'src/LlvmCompat.cpp',
     'src/Pipeline.cpp',
     'src/PrepareForOptPass/PrepareForOptPass.cpp',
+    'src/PseudoInst.cpp',
 ]
 
 clang = bindir / 'clang'
diff --git a/subprojects/helper-to-tcg/src/PseudoInst.cpp b/subprojects/helper-to-tcg/src/PseudoInst.cpp
new file mode 100644
index 0000000000..91e0f25e3d
--- /dev/null
+++ b/subprojects/helper-to-tcg/src/PseudoInst.cpp
@@ -0,0 +1,183 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#include "PseudoInst.hpp"
+#include "LlvmCompat.hpp"
+
+#include <llvm/ADT/DenseMap.h>
+#include <llvm/ADT/Twine.h>
+#include <llvm/IR/Function.h>
+#include <llvm/IR/Instructions.h>
+#include <llvm/Support/Casting.h>
+
+using namespace llvm;
+
+// Define an array `PseudoInstName[]`, indexed by `PseudoInst` and mapping
+// to a string representation of the enum.
+#define PSEUDO_INST_DEF(name, ret, args) #name
+static const char *PseudoInstName[] = {
+#include "PseudoInst.inc"
+};
+#undef PSEUDO_INST_DEF
+
+// Define an array `PseudoInstArgTypes[]` indexed by `PseudoInst` and
+// mapping to an array of `PseudoInstArg` representing allowed argument types.
+#define PSEUDO_INST_ARGVEC(...) {__VA_ARGS__}
+
+#define PSEUDO_INST_DEF(name, ret, args) args
+static const SmallVector<PseudoInstArg, 6> PseudoInstArgTypes[] = {
+#include "PseudoInst.inc"
+};
+#undef PSEUDO_INST_DEF
+#undef PSEUDO_INST_ARGVEC
+
+// In order to map from a `Function *` to a `PseudoInst`, we keep a map
+// of all functions created, this simplifies mapping of callees to
+// a `PseudoInst` that can be switched over.
+static DenseMap<Function *, PseudoInst> MapFuncToInst;
+
+// Converts llvm `Type`s to a string representation
+// that can be embedded in function names for basic overloading.
+//
+// Ex.
+//
+//      [8 x i8] -> "a8xi8"
+//      <128 x i8> -> "v128xi8"
+//
+// LLVM has an implementation of a similar function used by intrinsics,
+// called `getMangledTypeStr`, but it's not exposed.
+inline std::string getMangledTypeStr(Type *Ty) {
+    std::string TypeStr = "";
+    llvm::raw_string_ostream TypeStream(TypeStr);
+    switch (Ty->getTypeID()) {
+    case Type::ArrayTyID: {
+        auto *ArrayTy = cast<ArrayType>(Ty);
+        std::string ElementStr = getMangledTypeStr(ArrayTy->getElementType());
+        TypeStream << "a" << ArrayTy->getNumElements() << "x" << ElementStr;
+    } break;
+    case Type::FixedVectorTyID: {
+        auto *VecTy = cast<VectorType>(Ty);
+        uint32_t ElementCount = compat::getVectorElementCount(VecTy);
+        std::string ElementStr = getMangledTypeStr(VecTy->getElementType());
+        TypeStream << "v" << ElementCount << "x" << ElementStr;
+    } break;
+    case Type::StructTyID: {
+        auto *StructTy = cast<StructType>(Ty);
+        TypeStream << StructTy->getName();
+    } break;
+    case Type::IntegerTyID: {
+        auto *IntTy = cast<IntegerType>(Ty);
+        TypeStream << "i" << IntTy->getBitWidth();
+    } break;
+    case Type::PointerTyID: {
+        TypeStream << "p";
+    } break;
+    default:
+        abort();
+    }
+    return TypeStream.str();
+}
+
+// Access functions into the static defined above.
+
+const char *pseudoInstName(PseudoInst Inst) { return PseudoInstName[Inst]; }
+
+uint8_t pseudoInstArgCount(PseudoInst Inst) {
+    return PseudoInstArgTypes[Inst].size();
+}
+
+llvm::ArrayRef<PseudoInstArg> pseudoInstArgTypes(PseudoInst Inst) {
+    return PseudoInstArgTypes[Inst];
+}
+
+// Match LLVM type against `PseudoIntsArg`.
+static void assertMatchingType(PseudoInstArg Expected, Type *Ty) {
+    const Type::TypeID Id = Ty->getTypeID();
+    switch (Expected) {
+    case ArgAny:
+        return;
+    case ArgInt:
+        assert(Id == Type::IntegerTyID);
+        return;
+    case ArgVec:
+        assert(Id == Type::FixedVectorTyID);
+        return;
+    case ArgPtr:
+        assert(Id == Type::PointerTyID);
+        return;
+    case ArgLabel:
+        assert(Id == Type::LabelTyID);
+        return;
+    case ArgVoid:
+        assert(Id == Type::VoidTyID);
+        return;
+    default:
+        abort();
+    };
+}
+
+llvm::FunctionCallee pseudoInstFunction(llvm::Module &M, PseudoInst Inst,
+                                        llvm::Type *RetType,
+                                        llvm::ArrayRef<llvm::Type *> ArgTypes) {
+    ArrayRef<PseudoInstArg> PArgTypes = pseudoInstArgTypes(Inst);
+    assert(PArgTypes.size() == ArgTypes.size());
+    for (size_t I = 0; I < ArgTypes.size(); ++I) {
+        assertMatchingType(PArgTypes[I], ArgTypes[I]);
+    }
+
+    auto *FT = llvm::FunctionType::get(RetType, ArgTypes, false);
+
+    std::string FnName{PseudoInstName[Inst]};
+    if (!RetType->isVoidTy()) {
+        FnName += ".";
+        FnName += getMangledTypeStr(RetType);
+    }
+    for (llvm::Type *Ty : ArgTypes) {
+        if (Ty->isLabelTy()) {
+            continue;
+        }
+        FnName += ".";
+        FnName += getMangledTypeStr(Ty);
+    }
+
+    llvm::FunctionCallee Fn = M.getOrInsertFunction(FnName, FT);
+    auto *F = cast<Function>(Fn.getCallee());
+    MapFuncToInst.insert({F, Inst});
+
+    return Fn;
+}
+
+llvm::CallInst *createPseudoInstCall(llvm::Module &M, IRBuilder<> &Builder,
+                                     PseudoInst Inst, llvm::Type *RetType,
+                                     llvm::ArrayRef<llvm::Value *> Values) {
+    SmallVector<Type *, 8> ArgTypes;
+    for (Value *V : Values) {
+        ArgTypes.push_back(V->getType());
+    }
+
+    FunctionCallee Fn = pseudoInstFunction(M, Inst, RetType, ArgTypes);
+    return Builder.CreateCall(Fn, Values);
+}
+
+PseudoInst getPseudoInstFromCall(const CallInst *Call) {
+    Function *F = Call->getCalledFunction();
+    auto It = MapFuncToInst.find(F);
+    if (It == MapFuncToInst.end()) {
+        return InvalidPseudoInst;
+    }
+    return It->second;
+}
-- 
2.52.0

[PATCH v2 21/50] helper-to-tcg: Add guest vector layout description

[Anton Johansson via qemu development]

Vectors are represented as arrays of host-endian blocks at least 64 bits in
size, the size of which is configurable.  Lane 0 in each block may be
placed either at the most- or least-significant bytes.

This covers a wide range of QEMU targets' vector representations.

Vector layout information is used later when mapping constant
expressions such as constant vectors, and when emitting helper functions
for performing size-changing operations.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 .../helper-to-tcg/include/VectorLayout.hpp    | 116 ++++++++++++++++++
 1 file changed, 116 insertions(+)
 create mode 100644 subprojects/helper-to-tcg/include/VectorLayout.hpp

diff --git a/subprojects/helper-to-tcg/include/VectorLayout.hpp b/subprojects/helper-to-tcg/include/VectorLayout.hpp
new file mode 100644
index 0000000000..e85f5029d5
--- /dev/null
+++ b/subprojects/helper-to-tcg/include/VectorLayout.hpp
@@ -0,0 +1,116 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#pragma once
+
+#include <llvm/ADT/ArrayRef.h>
+#include <stddef.h>
+#include <stdint.h>
+
+// Vector Layout
+//
+// Vector layouts among QEMU guests is quite diverse, this header aims to be
+// simple but still capture a large subset of targets.
+//
+// Guest vectors are assumed to be divided into host-endian blocks of a
+// configurable size (>= 64-bit and power-of-two), with block 0 always being at
+// the lowest address.  Blocks in turn are divided into 64-bit columns which are
+// used when interfacing with QEMUs `gvec` API.
+//
+// Below is a 32-byte <32 x i8> guest vector consisting of 16-byte blocks,
+// 8-byte columns, and 1-byte lanes for each element in the vector.  Elements
+// count from 1-32 in hex to indicate their position in memory.  Note this
+// example is for a little-endian host with `LeastSignificant` lane preference
+// meaning lane 0 is at the least significant bytes of each block.
+//
+//   Guest vector <32 x i8> {1, 2, 3, ..., 32}
+//   +---------------------------------+---------------------------------+
+//   |              BLOCK 0            |              BLOCK 1            |
+//   +----------------+----------------+----------------+----------------+
+//   |    COLUMN 0    |    COLUMN 1    |    COLUMN 2    |    COLUMN 3    |
+//   |0102030405060708|090A0B0C0D0E0F10|1112131415161718|191A1B1C1D1E1F20|
+//
+//   LOW ADDRESS                                              HIGH ADDRESS
+//
+// On a big-endian host, the vector would instead be expressed in memory as
+//
+//   Guest vector <32 x i8> {1, 2, 3, ..., 32}
+//   +---------------------------------+---------------------------------+
+//   |              BLOCK 0            |              BLOCK 1            |
+//   +----------------+----------------+----------------+----------------+
+//   |    COLUMN 1    |    COLUMN 0    |    COLUMN 3    |    COLUMN 2    |
+//   |100F0E0D0C0B0A09|0807060504030201|201F1E1D1C1B1A19|1817161514131211|
+//
+//   LOW ADDRESS                                              HIGH ADDRESS
+//
+// with the column order having shifted to make sure the host-endian blocks
+// retain the same value.  Blocks are also allowed to be equal or greater in
+// size that the guest vector, at which point the entire memory view of the
+// vector would reverse when going from a little to big-endian host.
+//
+// The lane preference may also be changed to `MostSignificant` which for the
+// little-endian example places lane 0 and the highest address of block 0.
+// Note, that `MostSignificant` on little-endian and `LeastSignificant` on
+// big-endian hosts are equivalent in their memory representation, but will
+// differ in how they are emitted in C:
+//
+//   // MostSignificant, little endian
+//   uint64_t vec[] = {0x90A0B0C0D0E0F10, 0x102030405060708, ...};
+//
+//   // LeastSignificant, big endian
+//   uint64_t vec[] = {0x100F0E0D0C0B0A09, 0x807060504030201, ...};
+//
+
+enum LanePreference {
+    LeastSignificant,
+    MostSignificant,
+};
+
+struct VectorLayout {
+    bool HostBigEndian;
+    LanePreference Lane0;
+    size_t BlockBytes;
+
+    inline size_t index(bool Reverse, size_t Count, size_t I, size_t J) const {
+        if (Reverse) {
+            return Count * I + (J ^ (Count - 1));
+        } else {
+            return Count * I + J;
+        }
+    }
+
+    // Return the index into a linear array of lane L in column C
+    inline size_t indexLane(size_t LanesPerColumn, size_t C, size_t L) const {
+        const bool Reverse = (Lane0 == MostSignificant);
+        return index(Reverse, LanesPerColumn, C, L);
+    }
+
+    // Return the index into a linear array of
+    inline size_t indexBlock(size_t BlocksPerVec, size_t I) const {
+        const bool Reverse = (HostBigEndian ^ (Lane0 == MostSignificant));
+        return index(Reverse, BlocksPerVec, 0, I);
+    }
+
+    //inline uint64_t column(llvm::ArrayRef<uint64_t> Lanes) {
+    //    uint64_t Column = 0;
+    //    const size_t LanesPerColumn = 64 / ElementSize;
+    //    for (size_t L = 0; L < LanesPerColumn; ++L) {
+    //        const size_t Index = VL.indexLane(LanesPerColumn, C, L);
+    //        Column |= Ints[Index] << (ElementSize * L);
+    //    }
+    //}
+};
-- 
2.52.0

[PATCH v2 22/50] helper-to-tcg: Introduce PrepareForTcgPass

[Anton Johansson via qemu development]

Adds a new pass over the LLVM module which runs post-optimization with
the end-goal of:

  * Culling functions which aren't worth translating;
  * Canonicalizing the IR to something closer to TCG, and;
  * Extracting information which may be useful in the backend pass.

The bulk of IR transformations occur in the canonicalization phase, with
a handful occuring later in the packend. This commits sets up a new LLVM
pass over the IR module and runs it from the pipeline.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 .../helper-to-tcg/include/CmdLineOptions.hpp  |  2 ++
 .../include/PrepareForTcgPass.hpp             | 27 +++++++++++++++++++
 subprojects/helper-to-tcg/meson.build         |  1 +
 subprojects/helper-to-tcg/src/Pipeline.cpp    | 24 +++++++++++++++++
 .../PrepareForTcgPass/PrepareForTcgPass.cpp   | 25 +++++++++++++++++
 5 files changed, 79 insertions(+)
 create mode 100644 subprojects/helper-to-tcg/include/PrepareForTcgPass.hpp
 create mode 100644 subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp

diff --git a/subprojects/helper-to-tcg/include/CmdLineOptions.hpp b/subprojects/helper-to-tcg/include/CmdLineOptions.hpp
index ca1cb59835..bfdd3ebe41 100644
--- a/subprojects/helper-to-tcg/include/CmdLineOptions.hpp
+++ b/subprojects/helper-to-tcg/include/CmdLineOptions.hpp
@@ -23,3 +23,5 @@
 extern llvm::cl::list<std::string> InputFiles;
 // Options for PrepareForOptPass
 extern llvm::cl::opt<bool> TranslateAllHelpers;
+// Options for PrepareForTcgPass
+extern llvm::cl::opt<std::string> TcgGlobalMappingsName;
diff --git a/subprojects/helper-to-tcg/include/PrepareForTcgPass.hpp b/subprojects/helper-to-tcg/include/PrepareForTcgPass.hpp
new file mode 100644
index 0000000000..90bc9402cb
--- /dev/null
+++ b/subprojects/helper-to-tcg/include/PrepareForTcgPass.hpp
@@ -0,0 +1,27 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#pragma once
+
+#include <llvm/IR/PassManager.h>
+
+class PrepareForTcgPass : public llvm::PassInfoMixin<PrepareForTcgPass> {
+public:
+    PrepareForTcgPass() {}
+    llvm::PreservedAnalyses run(llvm::Module &M,
+                                llvm::ModuleAnalysisManager &MAM);
+};
diff --git a/subprojects/helper-to-tcg/meson.build b/subprojects/helper-to-tcg/meson.build
index b3a3de6297..c2926652f3 100644
--- a/subprojects/helper-to-tcg/meson.build
+++ b/subprojects/helper-to-tcg/meson.build
@@ -44,6 +44,7 @@ sources = [
     'src/Pipeline.cpp',
     'src/PrepareForOptPass/PrepareForOptPass.cpp',
     'src/PseudoInst.cpp',
+    'src/PrepareForTcgPass/PrepareForTcgPass.cpp',
 ]
 
 clang = bindir / 'clang'
diff --git a/subprojects/helper-to-tcg/src/Pipeline.cpp b/subprojects/helper-to-tcg/src/Pipeline.cpp
index 571276de24..6058718a9c 100644
--- a/subprojects/helper-to-tcg/src/Pipeline.cpp
+++ b/subprojects/helper-to-tcg/src/Pipeline.cpp
@@ -18,6 +18,7 @@
 #include "CmdLineOptions.hpp"
 #include "LlvmCompat.hpp"
 #include "PrepareForOptPass.hpp"
+#include "PrepareForTcgPass.hpp"
 
 #if LLVM_VERSION_MAJOR == 15
 #include <llvm/ADT/Triple.h>
@@ -32,6 +33,7 @@
 #include <llvm/IR/LLVMContext.h>
 #include <llvm/IR/Module.h>
 #include <llvm/IR/PassManager.h>
+#include <llvm/IR/Verifier.h>
 #include <llvm/IRReader/IRReader.h>
 #include <llvm/InitializePasses.h>
 #include <llvm/Linker/Linker.h>
@@ -43,6 +45,7 @@
 #include <llvm/Support/SourceMgr.h>
 #include <llvm/Support/TargetSelect.h>
 #include <llvm/Target/TargetMachine.h>
+#include <llvm/Transforms/Scalar/DCE.h>
 #include <llvm/Transforms/Scalar/SROA.h>
 
 #define DEBUG_TYPE "pipeline"
@@ -70,6 +73,13 @@ cl::opt<bool> TranslateAllHelpers(
     "translate-all-helpers", cl::init(false),
     cl::desc("Translate all functions starting with helper_*"), cl::cat(Cat));
 
+// Options for PrepareForTcgPass
+cl::opt<std::string> TcgGlobalMappingsName(
+    "tcg-global-mappings",
+    cl::desc("<Name of global cpu_mappings[] used for mapping accesses"
+             "into a struct to TCG globals>"),
+    cl::Required, cl::cat(Cat));
+
 // Define a TargetTransformInfo (TTI) subclass, this allows for overriding
 // common per-llvm-target information expected by other LLVM passes, such
 // as the width of the largest scalar/vector registers.  Needed for consistent
@@ -214,5 +224,19 @@ int main(int argc, char **argv) {
     MPM.addPass(
         PB.buildModuleOptimizationPipeline(compat::OptimizationLevel::Oz, {}));
 
+    //
+    // Next, we run our final transformations, including removing phis and our
+    // own instruction combining that prioritizes instructions that map more
+    // easily to TCG.
+    //
+
+    MPM.addPass(PrepareForTcgPass());
+    MPM.addPass(VerifierPass());
+    {
+        FunctionPassManager FPM;
+        FPM.addPass(DCEPass());
+        MPM.addPass(createModuleToFunctionPassAdaptor(std::move(FPM)));
+    }
+
     return 0;
 }
diff --git a/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp b/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp
new file mode 100644
index 0000000000..cea6573e41
--- /dev/null
+++ b/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp
@@ -0,0 +1,25 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#include "PrepareForTcgPass.hpp"
+
+using namespace llvm;
+
+PreservedAnalyses PrepareForTcgPass::run(Module &M, ModuleAnalysisManager &MAM)
+{
+    return PreservedAnalyses::none();
+}
-- 
2.52.0

[PATCH v2 23/50] helper-to-tcg: PrepareForTcgPass, remove functions with cycles

[Anton Johansson via qemu development]

Functions with cycles are removed for two primary reasons:

  * As a simplifying assumption for register allocation which occurs down
    the line, and;

  * If a function contains cycles post-optimization neither unrolling or
    loop vectorization were deemed beneficial, and the function _might_ be
    better suited as a helper anyway.

Cycles are detected by iterating over Strongly Connected Components
(SCCs) which imply the existence of cycles if a SCC contains more than one
node, or it has a self-edge.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 .../PrepareForTcgPass/PrepareForTcgPass.cpp   | 34 +++++++++++++++++--
 1 file changed, 32 insertions(+), 2 deletions(-)

diff --git a/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp b/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp
index cea6573e41..41f317ed0b 100644
--- a/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp
+++ b/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp
@@ -17,9 +17,39 @@
 
 #include "PrepareForTcgPass.hpp"
 
+#include <llvm/ADT/SCCIterator.h>
+#include <llvm/ADT/SmallPtrSet.h>
+#include <llvm/IR/Function.h>
+#include <llvm/IR/Module.h>
+
 using namespace llvm;
 
-PreservedAnalyses PrepareForTcgPass::run(Module &M, ModuleAnalysisManager &MAM)
-{
+static void removeFunctionsWithLoops(Module &M, ModuleAnalysisManager &MAM) {
+    // Iterate over all Strongly Connected Components (SCCs), a SCC implies
+    // the existence of loops if:
+    //   - it has more than one node, or;
+    //   - it has a self-edge.
+    SmallPtrSet<Function *, 16> FunctionsToRemove;
+    for (Function &F : M) {
+        if (F.isDeclaration()) {
+            continue;
+        }
+        for (auto It = scc_begin(&F); !It.isAtEnd(); ++It) {
+            if (It.hasCycle()) {
+                FunctionsToRemove.insert(&F);
+                break;
+            }
+        }
+    }
+
+    for (Function *F : FunctionsToRemove) {
+        F->setComdat(nullptr);
+        F->deleteBody();
+    }
+}
+
+PreservedAnalyses PrepareForTcgPass::run(Module &M,
+                                         ModuleAnalysisManager &MAM) {
+    removeFunctionsWithLoops(M, MAM);
     return PreservedAnalyses::none();
 }
-- 
2.52.0

[PATCH v2 24/50] helper-to-tcg: PrepareForTcgPass, demote PHI nodes

[Anton Johansson via qemu development]

PHI nodes have no clear analogue in TCG, this commits converts them to
stack accesses using a built-in LLVM transformation.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 .../PrepareForTcgPass/PrepareForTcgPass.cpp   | 23 +++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp b/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp
index 41f317ed0b..d8530406ad 100644
--- a/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp
+++ b/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp
@@ -20,7 +20,10 @@
 #include <llvm/ADT/SCCIterator.h>
 #include <llvm/ADT/SmallPtrSet.h>
 #include <llvm/IR/Function.h>
+#include <llvm/IR/InstIterator.h>
+#include <llvm/IR/Instructions.h>
 #include <llvm/IR/Module.h>
+#include <llvm/Transforms/Utils/Local.h>
 
 using namespace llvm;
 
@@ -48,8 +51,28 @@ static void removeFunctionsWithLoops(Module &M, ModuleAnalysisManager &MAM) {
     }
 }
 
+inline void demotePhis(Function &F) {
+    if (F.isDeclaration()) {
+        return;
+    }
+
+    SmallVector<PHINode *, 10> Phis;
+    for (auto &I : instructions(F)) {
+        if (auto *Phi = dyn_cast<PHINode>(&I)) {
+            Phis.push_back(Phi);
+        }
+    }
+
+    for (auto *Phi : Phis) {
+        DemotePHIToStack(Phi);
+    }
+}
+
 PreservedAnalyses PrepareForTcgPass::run(Module &M,
                                          ModuleAnalysisManager &MAM) {
     removeFunctionsWithLoops(M, MAM);
+    for (Function &F : M) {
+        demotePhis(F);
+    }
     return PreservedAnalyses::none();
 }
-- 
2.52.0

[PATCH v2 25/50] helper-to-tcg: PrepareForTcgPass, map TCG globals

[Anton Johansson via qemu development]

The input LLVM module may define an array of cpu_mapping structs,
describing the mapping between fields in a specified struct (usually
CPUArchState) and TCG globals.

Create a map between offsets into the specified struct and TCG globals
(name, size, number of elements, stride) by iterating over the global
cpu_mapping array.  The name of this array is configurable via
the --tcg-global-mappings flag.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 .../include/PrepareForTcgPass.hpp             |  6 +-
 .../helper-to-tcg/include/TcgGlobalMap.hpp    | 40 ++++++++++++
 subprojects/helper-to-tcg/src/Pipeline.cpp    |  9 +--
 .../PrepareForTcgPass/PrepareForTcgPass.cpp   | 62 +++++++++++++++++++
 4 files changed, 112 insertions(+), 5 deletions(-)
 create mode 100644 subprojects/helper-to-tcg/include/TcgGlobalMap.hpp

diff --git a/subprojects/helper-to-tcg/include/PrepareForTcgPass.hpp b/subprojects/helper-to-tcg/include/PrepareForTcgPass.hpp
index 90bc9402cb..3e0679b46c 100644
--- a/subprojects/helper-to-tcg/include/PrepareForTcgPass.hpp
+++ b/subprojects/helper-to-tcg/include/PrepareForTcgPass.hpp
@@ -17,11 +17,15 @@
 
 #pragma once
 
+#include "TcgGlobalMap.hpp"
 #include <llvm/IR/PassManager.h>
 
 class PrepareForTcgPass : public llvm::PassInfoMixin<PrepareForTcgPass> {
+    TcgGlobalMap &ResultTcgGlobalMap;
+
 public:
-    PrepareForTcgPass() {}
+    PrepareForTcgPass(TcgGlobalMap &ResultTcgGlobalMap)
+        : ResultTcgGlobalMap(ResultTcgGlobalMap) {}
     llvm::PreservedAnalyses run(llvm::Module &M,
                                 llvm::ModuleAnalysisManager &MAM);
 };
diff --git a/subprojects/helper-to-tcg/include/TcgGlobalMap.hpp b/subprojects/helper-to-tcg/include/TcgGlobalMap.hpp
new file mode 100644
index 0000000000..ad7ac54608
--- /dev/null
+++ b/subprojects/helper-to-tcg/include/TcgGlobalMap.hpp
@@ -0,0 +1,40 @@
+#pragma once
+
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#include <llvm/ADT/DenseMap.h>
+#include <llvm/ADT/SmallVector.h>
+#include <llvm/ADT/StringRef.h>
+#include <stdint.h>
+
+// `TcgGlobal` describes a field in a struct which has a mapping to a TCG
+// global.  `Size`, `NumElements`, and `Stride` describe the mapped type, all
+// types are assumed to be integer or arrays of integers.  `Code` is the
+// expression to be emitted when accessing the mapped global, usually the
+// variable name.
+struct TcgGlobal {
+    llvm::StringRef Code;
+    uint64_t Size;
+    uint64_t NumElements;
+    uint64_t Stride;
+};
+
+// Array of maps between offsets into a mapped struct to the resulting global
+// type, outer array is indexed by the base struct type to handle multiple
+// struct-to-global mappings.
+using TcgGlobalMap = llvm::SmallVector<llvm::DenseMap<uint32_t, TcgGlobal>, 1>;
diff --git a/subprojects/helper-to-tcg/src/Pipeline.cpp b/subprojects/helper-to-tcg/src/Pipeline.cpp
index 6058718a9c..f24bb67f0c 100644
--- a/subprojects/helper-to-tcg/src/Pipeline.cpp
+++ b/subprojects/helper-to-tcg/src/Pipeline.cpp
@@ -76,9 +76,9 @@ cl::opt<bool> TranslateAllHelpers(
 // Options for PrepareForTcgPass
 cl::opt<std::string> TcgGlobalMappingsName(
     "tcg-global-mappings",
-    cl::desc("<Name of global cpu_mappings[] used for mapping accesses"
-             "into a struct to TCG globals>"),
-    cl::Required, cl::cat(Cat));
+    cl::desc("Name of global cpu_mappings[] used for mapping accesses"
+             "into a struct to TCG globals"),
+    cl::init("mappings"), cl::cat(Cat));
 
 // Define a TargetTransformInfo (TTI) subclass, this allows for overriding
 // common per-llvm-target information expected by other LLVM passes, such
@@ -230,7 +230,8 @@ int main(int argc, char **argv) {
     // easily to TCG.
     //
 
-    MPM.addPass(PrepareForTcgPass());
+    TcgGlobalMap TcgGlobals;
+    MPM.addPass(PrepareForTcgPass(TcgGlobals));
     MPM.addPass(VerifierPass());
     {
         FunctionPassManager FPM;
diff --git a/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp b/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp
index d8530406ad..69add43529 100644
--- a/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp
+++ b/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp
@@ -16,9 +16,12 @@
 //
 
 #include "PrepareForTcgPass.hpp"
+#include "CmdLineOptions.hpp"
 
 #include <llvm/ADT/SCCIterator.h>
 #include <llvm/ADT/SmallPtrSet.h>
+#include <llvm/ADT/StringMap.h>
+#include <llvm/IR/Constants.h>
 #include <llvm/IR/Function.h>
 #include <llvm/IR/InstIterator.h>
 #include <llvm/IR/Instructions.h>
@@ -68,11 +71,70 @@ inline void demotePhis(Function &F) {
     }
 }
 
+static StringMap<size_t> collectTcgGlobals(Module &M, TcgGlobalMap &ResultTcgGlobalMap) {
+    auto *Map = M.getGlobalVariable(TcgGlobalMappingsName);
+    if (!Map) {
+        return {};
+    }
+
+    // In case the `tcg_global_mappings` array is empty,
+    // casting to `ConstantArray` will fail, even though it's a
+    // `[0 x %struct.cpu_tcg_mapping]`.
+    auto *MapElems = dyn_cast<ConstantArray>(Map->getOperand(0));
+    if (!MapElems) {
+        return {};
+    }
+
+    StringMap<size_t> TypeIndexMap;
+
+    for (auto Row : MapElems->operand_values()) {
+        auto *ConstRow = cast<ConstantStruct>(Row);
+
+        // Get code string
+        auto *CodePtr = ConstRow->getOperand(0);
+        StringRef CodeStr =
+            cast<ConstantDataArray>(CodePtr->getOperand(0))->getAsString();
+        CodeStr = CodeStr.rtrim('\0');
+
+        // Get base type name
+        auto *TypeNamePtr = ConstRow->getOperand(2);
+        StringRef TypeNameStr =
+            cast<ConstantDataArray>(TypeNamePtr->getOperand(0))->getAsString();
+        TypeNameStr = TypeNameStr.rtrim('\0');
+
+        // Get offset in cpu env
+        auto *Offset = cast<ConstantInt>(ConstRow->getOperand(4));
+        // Get size of variable in cpu env
+        auto *SizeInBytes = cast<ConstantInt>(ConstRow->getOperand(5));
+        unsigned SizeInBits = 8 * SizeInBytes->getLimitedValue();
+
+        auto *Stride = cast<ConstantInt>(ConstRow->getOperand(6));
+        auto *NumElements = cast<ConstantInt>(ConstRow->getOperand(7));
+
+        if (auto It = TypeIndexMap.find(TypeNameStr);
+            It == TypeIndexMap.end()) {
+            TypeIndexMap[TypeNameStr] = ResultTcgGlobalMap.size();
+            ResultTcgGlobalMap.emplace_back();
+        }
+
+        const size_t Index = TypeIndexMap[TypeNameStr];
+        ResultTcgGlobalMap[Index][Offset->getLimitedValue()] = {
+            CodeStr,
+            SizeInBits,
+            NumElements->getLimitedValue(),
+            Stride->getLimitedValue(),
+        };
+    }
+
+    return TypeIndexMap;
+}
+
 PreservedAnalyses PrepareForTcgPass::run(Module &M,
                                          ModuleAnalysisManager &MAM) {
     removeFunctionsWithLoops(M, MAM);
     for (Function &F : M) {
         demotePhis(F);
     }
+    const StringMap<size_t> TypeIndexMap = collectTcgGlobals(M, ResultTcgGlobalMap);
     return PreservedAnalyses::none();
 }
-- 
2.52.0

[PATCH v2 26/50] helper-to-tcg: PrepareForTcgPass, transform GEPs

[Anton Johansson via qemu development]

getelementpointer (GEP) instructions in LLVM IR represent general pointer
arithmetic (struct field access, array indexing, ...).  From the
perspective of TCG, three distinct cases are important and are
transformed into pseudo instructions respectively:

  * Struct accesses whose offset into the struct map to a TCG global are
    transformed into `call @AccessGlobalValue(structindex, offset)`;

  * Struct accesses whose offset into the struct map to an array of TCG
    globals are transformed into `call @AccessGlobalArray(structindex, offset, index)`;

  * Otherwise, access is converted to general pointer arithmetic in LLVM IR using
    `call @PtrAdd(...)`.

These three cases are treated differently in the backend and all other
GEPs are considered erroneous.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 .../include/PrepareForTcgPass.hpp             |   7 +-
 subprojects/helper-to-tcg/meson.build         |   1 +
 subprojects/helper-to-tcg/src/Pipeline.cpp    |   2 +-
 .../PrepareForTcgPass/PrepareForTcgPass.cpp   |   4 +
 .../src/PrepareForTcgPass/TransformGEPs.cpp   | 355 ++++++++++++++++++
 .../src/PrepareForTcgPass/TransformGEPs.hpp   |  45 +++
 6 files changed, 411 insertions(+), 3 deletions(-)
 create mode 100644 subprojects/helper-to-tcg/src/PrepareForTcgPass/TransformGEPs.cpp
 create mode 100644 subprojects/helper-to-tcg/src/PrepareForTcgPass/TransformGEPs.hpp

diff --git a/subprojects/helper-to-tcg/include/PrepareForTcgPass.hpp b/subprojects/helper-to-tcg/include/PrepareForTcgPass.hpp
index 3e0679b46c..d0cd6e3cbd 100644
--- a/subprojects/helper-to-tcg/include/PrepareForTcgPass.hpp
+++ b/subprojects/helper-to-tcg/include/PrepareForTcgPass.hpp
@@ -17,15 +17,18 @@
 
 #pragma once
 
+#include "DebugInfo.hpp"
 #include "TcgGlobalMap.hpp"
 #include <llvm/IR/PassManager.h>
 
 class PrepareForTcgPass : public llvm::PassInfoMixin<PrepareForTcgPass> {
     TcgGlobalMap &ResultTcgGlobalMap;
+    const DebugInfoMapTy &DebugInfo;
 
 public:
-    PrepareForTcgPass(TcgGlobalMap &ResultTcgGlobalMap)
-        : ResultTcgGlobalMap(ResultTcgGlobalMap) {}
+    PrepareForTcgPass(TcgGlobalMap &ResultTcgGlobalMap,
+                      const DebugInfoMapTy &DebugInfo)
+        : ResultTcgGlobalMap(ResultTcgGlobalMap), DebugInfo(DebugInfo) {}
     llvm::PreservedAnalyses run(llvm::Module &M,
                                 llvm::ModuleAnalysisManager &MAM);
 };
diff --git a/subprojects/helper-to-tcg/meson.build b/subprojects/helper-to-tcg/meson.build
index c2926652f3..564191d328 100644
--- a/subprojects/helper-to-tcg/meson.build
+++ b/subprojects/helper-to-tcg/meson.build
@@ -45,6 +45,7 @@ sources = [
     'src/PrepareForOptPass/PrepareForOptPass.cpp',
     'src/PseudoInst.cpp',
     'src/PrepareForTcgPass/PrepareForTcgPass.cpp',
+    'src/PrepareForTcgPass/TransformGEPs.cpp',
 ]
 
 clang = bindir / 'clang'
diff --git a/subprojects/helper-to-tcg/src/Pipeline.cpp b/subprojects/helper-to-tcg/src/Pipeline.cpp
index f24bb67f0c..9212bba29f 100644
--- a/subprojects/helper-to-tcg/src/Pipeline.cpp
+++ b/subprojects/helper-to-tcg/src/Pipeline.cpp
@@ -231,7 +231,7 @@ int main(int argc, char **argv) {
     //
 
     TcgGlobalMap TcgGlobals;
-    MPM.addPass(PrepareForTcgPass(TcgGlobals));
+    MPM.addPass(PrepareForTcgPass(TcgGlobals, DebugInfo));
     MPM.addPass(VerifierPass());
     {
         FunctionPassManager FPM;
diff --git a/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp b/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp
index 69add43529..10129fa245 100644
--- a/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp
+++ b/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp
@@ -17,6 +17,7 @@
 
 #include "PrepareForTcgPass.hpp"
 #include "CmdLineOptions.hpp"
+#include "TransformGEPs.hpp"
 
 #include <llvm/ADT/SCCIterator.h>
 #include <llvm/ADT/SmallPtrSet.h>
@@ -136,5 +137,8 @@ PreservedAnalyses PrepareForTcgPass::run(Module &M,
         demotePhis(F);
     }
     const StringMap<size_t> TypeIndexMap = collectTcgGlobals(M, ResultTcgGlobalMap);
+    for (Function &F : M) {
+        transformGEPs(M, F, ResultTcgGlobalMap, TypeIndexMap, DebugInfo);
+    }
     return PreservedAnalyses::none();
 }
diff --git a/subprojects/helper-to-tcg/src/PrepareForTcgPass/TransformGEPs.cpp b/subprojects/helper-to-tcg/src/PrepareForTcgPass/TransformGEPs.cpp
new file mode 100644
index 0000000000..f4c4a5b22d
--- /dev/null
+++ b/subprojects/helper-to-tcg/src/PrepareForTcgPass/TransformGEPs.cpp
@@ -0,0 +1,355 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#include "TransformGEPs.hpp"
+#include "DebugInfo.hpp"
+#include "LlvmCompat.hpp"
+#include "Error.hpp"
+#include "PseudoInst.hpp"
+
+#include <llvm/ADT/SmallSet.h>
+#include <llvm/ADT/SmallVector.h>
+#include <llvm/ADT/iterator_range.h>
+#include <llvm/IR/DerivedTypes.h>
+#include <llvm/IR/Function.h>
+#include <llvm/IR/IRBuilder.h>
+#include <llvm/IR/InstIterator.h>
+#include <llvm/IR/Module.h>
+#include <llvm/IR/Operator.h>
+#include <llvm/IR/Value.h>
+#include <llvm/Support/Debug.h>
+
+#define DEBUG_TYPE "transform-geps"
+
+using namespace llvm;
+
+// collectIndices will, given a getelementptr (GEP) instruction, construct an
+// array of GepIndex structs keeping track of the total offset into the struct
+// along with some access information.  For instance,
+//
+//   struct SubS {
+//      uint8_t a;
+//      uint8_t b;
+//      uint8_t c;
+//   };
+//
+//   struct S {
+//      uint64_t i;
+//      struct SubS sub[3];
+//   };
+//
+//   void f(struct S *s, int idx) {
+//      S->sub[idx].a = ...
+//      S->sub[idx].b = ...
+//      S->sub[idx].c = ...
+//   }
+//
+// would correspond to the following GEPs
+//
+//   getelementptr %struct.S, %struct.S* %s, i64 0, i32 1, %idx, i32 0
+//   getelementptr %struct.S, %struct.S* %s, i64 0, i32 1, %idx, i32 1
+//   getelementptr %struct.S, %struct.S* %s, i64 0, i32 1, %idx, i32 2
+//
+// or the following GepIndex's
+//
+//   GepIndex{Size=0,false}, GepIndex{Size=8,false}, GepIndex{Size=4,true},
+//   GepIndex{Size=0,false} GepIndex{Size=0,false}, GepIndex{Size=8,false},
+//   GepIndex{Size=4,true}, GepIndex{Size=1,false} GepIndex{Size=0,false},
+//   GepIndex{Size=8,false}, GepIndex{Size=4,true}, GepIndex{Size=2,false}
+//
+
+struct GepIndex {
+    Value *V;
+    uint64_t Size;
+    bool IsArrayAccess = false;
+};
+
+using GepIndices = SmallVector<GepIndex, 2>;
+
+struct GlobalAccessInfo {
+    uint64_t Offset;
+    StringRef BaseTypeName;
+    Value *LastArrayIndex;
+};
+
+using GlobalAccessMap = DenseMap<Value *, GlobalAccessInfo>;
+
+static Expected<GepIndices> collectIndices(const DataLayout &DL,
+                                           GEPOperator *Gep) {
+    Type *PtrOpTy = Gep->getPointerOperandType();
+    if (!PtrOpTy->isPointerTy()) {
+        return mkError("GEPs on vectors are not handled!");
+    }
+    Type *InternalTy = Type::getIntNTy(Gep->getContext(), 64);
+    auto *One = ConstantInt::get(InternalTy, 1u);
+
+    GepIndices Result;
+    Type *CurrentTy = PtrOpTy;
+
+    // Handle initial pointer dereference
+    auto Begin = Gep->idx_begin();
+    {
+        CurrentTy = Gep->getSourceElementType();
+        const size_t FixedSize = compat::getTypeAllocSize(DL, CurrentTy);
+        Result.push_back(GepIndex{*Begin, FixedSize});
+        ++Begin;
+    }
+
+    for (auto &Arg : make_range(Begin, Gep->idx_end())) {
+        switch (CurrentTy->getTypeID()) {
+        case Type::ArrayTyID: {
+            CurrentTy = cast<ArrayType>(CurrentTy)->getElementType();
+            const size_t FixedSize = compat::getTypeAllocSize(DL, CurrentTy);
+            Result.push_back(
+                GepIndex{Arg.get(), FixedSize, /* IsArrayAccess= */ true});
+        } break;
+        case Type::StructTyID: {
+            auto *StructTy = cast<StructType>(CurrentTy);
+            auto *Constant = dyn_cast<ConstantInt>(Arg.get());
+            if (Constant->getBitWidth() > DL.getPointerSizeInBits()) {
+                return mkError(
+                    "GEP to struct with unsupported index bit width!");
+            }
+            uint64_t ConstantValue = Constant->getZExtValue();
+            uint64_t ElementOffset =
+                DL.getStructLayout(StructTy)->getElementOffset(ConstantValue);
+            CurrentTy = StructTy->getTypeAtIndex(ConstantValue);
+            Result.push_back(GepIndex{One, ElementOffset});
+        } break;
+        default:
+            return mkError("GEP unsupported index type: ");
+        }
+    }
+
+    return Result;
+}
+
+// Takes indices associated with a getelementpointer instruction and expands
+// it into pointer math.
+static void replaceGEPWithPointerMath(Module &M, Instruction *ParentInst,
+                                      GEPOperator *Gep,
+                                      const GepIndices &Indices) {
+    assert(Indices.size() > 0);
+    IRBuilder<> Builder(ParentInst);
+    Value *PtrOp = Gep->getPointerOperand();
+
+    // Sum indices to get the total offset from the base pointer
+    Value *PrevV = nullptr;
+    for (auto &Index : Indices) {
+        Value *Mul = Builder.CreateMul(
+            Index.V, ConstantInt::get(Index.V->getType(), Index.Size));
+        if (PrevV) {
+            uint32_t BitWidthLeft =
+                cast<IntegerType>(PrevV->getType())->getIntegerBitWidth();
+            uint32_t BitWidthRight =
+                cast<IntegerType>(Mul->getType())->getIntegerBitWidth();
+            if (BitWidthLeft < BitWidthRight) {
+                PrevV = Builder.CreateZExt(PrevV, Mul->getType());
+            } else if (BitWidthLeft > BitWidthRight) {
+                Mul = Builder.CreateZExt(Mul, PrevV->getType());
+            }
+            PrevV = Builder.CreateAdd(PrevV, Mul);
+        } else {
+            PrevV = Mul;
+        }
+    }
+
+    Gep->replaceAllUsesWith(createPseudoInstCall(
+        M, Builder, PtrAdd, Gep->getType(), {PtrOp, PrevV}));
+}
+
+// Takes indices associated with a getelementpointer instruction and expands
+// it into pointer math.
+static Value *replaceGEPWithGlobalAccess(Module &M, Instruction *ParentInst,
+                                         GEPOperator *Gep, size_t TypeIndex,
+                                         uint64_t BaseOffset,
+                                         Value *ArrayIndex) {
+    CallInst *Call;
+    IRBuilder<> Builder(ParentInst);
+    Type *IndexTy = Type::getIntNTy(M.getContext(), 64);
+    auto *ConstBaseOffset = ConstantInt::get(IndexTy, BaseOffset);
+    auto *ConstTypeIndex = ConstantInt::get(IndexTy, TypeIndex);
+    if (ArrayIndex) {
+        Call =
+            createPseudoInstCall(M, Builder, AccessGlobalArray, Gep->getType(),
+                                 {ConstTypeIndex, ConstBaseOffset, ArrayIndex});
+    } else {
+        Call =
+            createPseudoInstCall(M, Builder, AccessGlobalValue, Gep->getType(),
+                                 {ConstTypeIndex, ConstBaseOffset});
+    }
+    Gep->replaceAllUsesWith(Call);
+    return cast<Value>(Call);
+}
+
+static bool transformGEP(Module &M, const TcgGlobalMap &TcgGlobals,
+                         GlobalAccessMap &GAMap,
+                         const StringMap<size_t> &TypeIndexMap,
+                         const DebugInfoMapTy &DebugInfo,
+                         const GepIndices &Indices, Instruction *ParentInst,
+                         GEPOperator *Gep) {
+    GlobalAccessInfo Info{};
+    uint32_t NumArrayAccesses = 0;
+    for (const GepIndex &Index : Indices) {
+        if (Index.IsArrayAccess) {
+            Info.LastArrayIndex = Index.V;
+            ++NumArrayAccesses;
+        } else if (auto *Const = dyn_cast<ConstantInt>(Index.V)) {
+            Info.Offset += Const->getZExtValue() * Index.Size;
+        }
+    }
+
+    Value *PtrOp = Gep->getPointerOperand();
+    bool IsGlobalAccess = isa<Argument>(PtrOp);
+    if (auto It = GAMap.find(PtrOp); It != GAMap.end()) {
+        Info.Offset += It->second.Offset;
+        Info.BaseTypeName = It->second.BaseTypeName;
+        assert(!Info.LastArrayIndex or !It->second.LastArrayIndex);
+        if (It->second.LastArrayIndex) {
+            Info.LastArrayIndex = It->second.LastArrayIndex;
+        }
+        IsGlobalAccess = true;
+    } else if (auto It = DebugInfo.find(PtrOp); It != DebugInfo.end()) {
+        Info.BaseTypeName = It->second.BaseTypeName;
+    }
+
+    bool PtrHasMapping =
+        !Info.BaseTypeName.empty() and TypeIndexMap.count(Info.BaseTypeName);
+
+    LLVM_DEBUG({
+        dbgs() << "For " << *Gep << "\n";
+        dbgs() << "  has mapping: " << PtrHasMapping << "\n";
+        dbgs() << "  is global access: " << IsGlobalAccess << "\n";
+    });
+
+    if (IsGlobalAccess and PtrHasMapping and NumArrayAccesses <= 1) {
+
+        if (!isa<ConstantExpr>(Gep)) {
+            bool HasOnlyGEPUsers = true;
+            for (auto *U : cast<Instruction>(Gep)->users()) {
+                if (!isa<GEPOperator>(U)) {
+                    HasOnlyGEPUsers = false;
+                    break;
+                }
+            }
+            if (HasOnlyGEPUsers) {
+                GAMap[cast<Value>(Gep)] = Info;
+                return true;
+            }
+        }
+
+        const size_t Index = TypeIndexMap.lookup(Info.BaseTypeName);
+
+        // Array accesses, particularly those to an array at the beginning of a
+        // mapped structs, e.g. `uint32_t v = env->gpr[40]`, might appear either
+        // as
+        //
+        //   getelementptr struct.CPUArchState, ptr %0, i64 0, i64 40,
+        //
+        // or
+        //
+        //   getelementptr i8, ptr %0, i64 160.
+        //
+        // The former and simpler one being more common in older LLVM versions.
+        // For the latter we need to manually compute the array index.
+        for (auto &P : TcgGlobals[Index]) {
+            const TcgGlobal Global = P.second;
+            if ((Global.NumElements == 1 and Info.Offset == P.first) or
+                (Info.Offset >= P.first and
+                 Info.Offset < P.first + Global.NumElements * Global.Stride)) {
+                Value *ArrayIndexV = Info.LastArrayIndex;
+                if (Global.NumElements > 1 and !Info.LastArrayIndex) {
+                    assert(Global.Stride > 0);
+                    uint64_t ArrayIndex =
+                        (Info.Offset - P.first) / Global.Stride;
+                    Info.Offset = P.first;
+                    ArrayIndexV = ConstantInt::get(
+                        Type::getInt64Ty(M.getContext()), ArrayIndex);
+                }
+
+                LLVM_DEBUG(dbgs() << "  replacing with global access\n");
+                Value *Access = replaceGEPWithGlobalAccess(
+                    M, ParentInst, Gep, Index, Info.Offset, ArrayIndexV);
+                GAMap[Access] = Info;
+                return !isa<ConstantExpr>(Gep);
+            }
+        }
+    }
+
+    LLVM_DEBUG(dbgs() << "  replacing with pointer math\n");
+    replaceGEPWithPointerMath(M, ParentInst, Gep, Indices);
+    return !isa<ConstantExpr>(Gep);
+}
+
+static GEPOperator *getGEPOperator(Instruction *I) {
+    // If the instructions is directly a GEP, simply return it.
+    auto *GEP = dyn_cast<GEPOperator>(I);
+    if (GEP) {
+        return GEP;
+    }
+
+    // Hard-code handling of GEPs that appear as an inline operand to loads
+    // and stores.
+    if (isa<LoadInst>(I)) {
+        auto *Load = cast<LoadInst>(I);
+        auto *ConstExpr = dyn_cast<ConstantExpr>(Load->getPointerOperand());
+        if (ConstExpr) {
+            return dyn_cast<GEPOperator>(ConstExpr);
+        }
+    } else if (isa<StoreInst>(I)) {
+        auto *Store = dyn_cast<StoreInst>(I);
+        auto *ConstExpr = dyn_cast<ConstantExpr>(Store->getPointerOperand());
+        if (ConstExpr) {
+            return dyn_cast<GEPOperator>(ConstExpr);
+        }
+    }
+
+    return nullptr;
+}
+
+void transformGEPs(Module &M, Function &F, const TcgGlobalMap &TcgGlobals,
+                   const StringMap<size_t> &TypeIndexMap,
+                   const DebugInfoMapTy &DebugInfo) {
+    SmallSet<Instruction *, 8> InstToErase;
+    GlobalAccessMap GAMap;
+
+    LLVM_DEBUG(dbgs() << "Transforming GEPs for:" << F.getName() << "\n");
+
+    for (auto &I : instructions(F)) {
+        GEPOperator *GEP = getGEPOperator(&I);
+        if (!GEP) {
+            continue;
+        }
+
+        Expected<GepIndices> Indices = collectIndices(M.getDataLayout(), GEP);
+        if (!Indices) {
+            dbgs() << "Failed collecting GEP indices for:\n\t" << I << "\n";
+            dbgs() << "Reason: " << Indices.takeError();
+            abort();
+        }
+
+        bool ShouldErase = transformGEP(M, TcgGlobals, GAMap, TypeIndexMap,
+                                        DebugInfo, Indices.get(), &I, GEP);
+        if (ShouldErase) {
+            InstToErase.insert(&I);
+        }
+    }
+
+    for (auto *I : InstToErase) {
+        I->eraseFromParent();
+    }
+}
diff --git a/subprojects/helper-to-tcg/src/PrepareForTcgPass/TransformGEPs.hpp b/subprojects/helper-to-tcg/src/PrepareForTcgPass/TransformGEPs.hpp
new file mode 100644
index 0000000000..8430a65fa0
--- /dev/null
+++ b/subprojects/helper-to-tcg/src/PrepareForTcgPass/TransformGEPs.hpp
@@ -0,0 +1,45 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#pragma once
+
+#include "DebugInfo.hpp"
+#include "TcgGlobalMap.hpp"
+#include <llvm/ADT/StringMap.h>
+#include <llvm/IR/Function.h>
+#include <llvm/IR/Module.h>
+
+//
+// Transform of module that converts `getelementptr` (GEP) operators to
+// pseudo instructions, either:
+//
+//   1. `call @AccessGlobalArray(StructIndex, Offset, Index)`
+//      if `Offset` is mapped to a global TCG array;
+//
+//   2. `call @AccessGlobalValue(StructIndex, Offset)`
+//      if `Offset` is mapped to a global TCG value;
+//
+//   3. Pointer math, if above fails.
+//
+// `StructIndex` is a an increasing integer assigned to each mapped struct type,
+// usually only `CPUArchState` is mapped, so `StructIndex` would be 0.
+//
+
+void transformGEPs(llvm::Module &M, llvm::Function &F,
+                   const TcgGlobalMap &TcgGlobals,
+                   const llvm::StringMap<size_t> &TypeIndexMap,
+                   const DebugInfoMapTy &DebugInfo);
-- 
2.52.0

[PATCH v2 27/50] helper-to-tcg: PrepareForTcgPass, canonicalize IR

[Anton Johansson via qemu development]

Iterates over the IR with the goal of converting it to a form closer to
TCG, taking care of IR discrepancies between LLVM and TCG.  This also
simplifies the backend by containing the bulk of custom IR
transformations.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 .../helper-to-tcg/include/CmdLineOptions.hpp  |    3 +
 .../include/PrepareForTcgPass.hpp             |    7 +-
 subprojects/helper-to-tcg/meson.build         |    1 +
 subprojects/helper-to-tcg/src/Pipeline.cpp    |   63 +-
 .../src/PrepareForTcgPass/CanonicalizeIR.cpp  | 1193 +++++++++++++++++
 .../src/PrepareForTcgPass/CanonicalizeIR.hpp  |   24 +
 .../PrepareForTcgPass/PrepareForTcgPass.cpp   |    2 +
 7 files changed, 1290 insertions(+), 3 deletions(-)
 create mode 100644 subprojects/helper-to-tcg/src/PrepareForTcgPass/CanonicalizeIR.cpp
 create mode 100644 subprojects/helper-to-tcg/src/PrepareForTcgPass/CanonicalizeIR.hpp

diff --git a/subprojects/helper-to-tcg/include/CmdLineOptions.hpp b/subprojects/helper-to-tcg/include/CmdLineOptions.hpp
index bfdd3ebe41..3a827f8ab7 100644
--- a/subprojects/helper-to-tcg/include/CmdLineOptions.hpp
+++ b/subprojects/helper-to-tcg/include/CmdLineOptions.hpp
@@ -25,3 +25,6 @@ extern llvm::cl::list<std::string> InputFiles;
 extern llvm::cl::opt<bool> TranslateAllHelpers;
 // Options for PrepareForTcgPass
 extern llvm::cl::opt<std::string> TcgGlobalMappingsName;
+extern llvm::cl::opt<std::string> UserPCRelBranchFunc;
+extern llvm::cl::opt<std::string> UserPCRelBranchFallthroughFunc;
+extern llvm::cl::opt<std::string> UserPCRelBranchConditionalFunc;
diff --git a/subprojects/helper-to-tcg/include/PrepareForTcgPass.hpp b/subprojects/helper-to-tcg/include/PrepareForTcgPass.hpp
index d0cd6e3cbd..078477ba2f 100644
--- a/subprojects/helper-to-tcg/include/PrepareForTcgPass.hpp
+++ b/subprojects/helper-to-tcg/include/PrepareForTcgPass.hpp
@@ -19,16 +19,19 @@
 
 #include "DebugInfo.hpp"
 #include "TcgGlobalMap.hpp"
+#include "VectorLayout.hpp"
 #include <llvm/IR/PassManager.h>
 
 class PrepareForTcgPass : public llvm::PassInfoMixin<PrepareForTcgPass> {
     TcgGlobalMap &ResultTcgGlobalMap;
     const DebugInfoMapTy &DebugInfo;
+    const VectorLayout &VL;
 
 public:
     PrepareForTcgPass(TcgGlobalMap &ResultTcgGlobalMap,
-                      const DebugInfoMapTy &DebugInfo)
-        : ResultTcgGlobalMap(ResultTcgGlobalMap), DebugInfo(DebugInfo) {}
+                      const DebugInfoMapTy &DebugInfo, const VectorLayout &VL)
+        : ResultTcgGlobalMap(ResultTcgGlobalMap), DebugInfo(DebugInfo), VL(VL) {
+    }
     llvm::PreservedAnalyses run(llvm::Module &M,
                                 llvm::ModuleAnalysisManager &MAM);
 };
diff --git a/subprojects/helper-to-tcg/meson.build b/subprojects/helper-to-tcg/meson.build
index 564191d328..79ce0c6dd8 100644
--- a/subprojects/helper-to-tcg/meson.build
+++ b/subprojects/helper-to-tcg/meson.build
@@ -46,6 +46,7 @@ sources = [
     'src/PseudoInst.cpp',
     'src/PrepareForTcgPass/PrepareForTcgPass.cpp',
     'src/PrepareForTcgPass/TransformGEPs.cpp',
+    'src/PrepareForTcgPass/CanonicalizeIR.cpp',
 ]
 
 clang = bindir / 'clang'
diff --git a/subprojects/helper-to-tcg/src/Pipeline.cpp b/subprojects/helper-to-tcg/src/Pipeline.cpp
index 9212bba29f..9b45664bdc 100644
--- a/subprojects/helper-to-tcg/src/Pipeline.cpp
+++ b/subprojects/helper-to-tcg/src/Pipeline.cpp
@@ -19,6 +19,7 @@
 #include "LlvmCompat.hpp"
 #include "PrepareForOptPass.hpp"
 #include "PrepareForTcgPass.hpp"
+#include "VectorLayout.hpp"
 
 #if LLVM_VERSION_MAJOR == 15
 #include <llvm/ADT/Triple.h>
@@ -42,6 +43,7 @@
 #include <llvm/Support/CommandLine.h>
 #include <llvm/Support/Debug.h>
 #include <llvm/Support/InitLLVM.h>
+#include <llvm/Support/MathExtras.h>
 #include <llvm/Support/SourceMgr.h>
 #include <llvm/Support/TargetSelect.h>
 #include <llvm/Target/TargetMachine.h>
@@ -80,6 +82,51 @@ cl::opt<std::string> TcgGlobalMappingsName(
              "into a struct to TCG globals"),
     cl::init("mappings"), cl::cat(Cat));
 
+cl::opt<std::string> UserPCRelBranchFunc(
+    "user-pcrel-branch-func",
+    cl::desc("Specify a function name in the input which represents a "
+             "PC-relative jump operation in QEMU."),
+    cl::init(""), cl::cat(Cat));
+
+cl::opt<std::string> UserPCRelBranchConditionalFunc(
+    "user-pcrel-branch-conditional-func",
+    cl::desc("Specify a function name in QEMU which represents a "
+             "PC-relative conditional jump operation.  Will be"
+             " emitted when the function supplied with -user-pcrel-branch-func"
+             " is called from inside a conditional block."),
+    cl::init(""), cl::cat(Cat));
+
+cl::opt<std::string> UserPCRelBranchFallthroughFunc(
+    "user-pcrel-branch-fallthrough-func",
+    cl::desc("Specify a function name in QEMU which represents a "
+             "PC-relative fallthrough operation.  Will be emitted when the"
+             " function supplied with -user-pcrel-branch-func"
+             " is called"
+             " from inside a conditional block, but that condition isn't taken."
+             " and we instead need to fallthrough to the next instruction."),
+    cl::init(""), cl::cat(Cat));
+ 
+static llvm::cl::opt<bool> VecForceBigEndian(
+    "vec-force-big-endian",
+    cl::desc("Force emission of vector operations as if on a big-endian host."),
+    cl::init(false), cl::cat(Cat));
+
+static llvm::cl::opt<size_t>
+    VecBlockBytes("vec-block-bytes",
+                  cl::desc("Sets the size of the host-endian blocks vector "
+                           "used to represent target vectors."),
+                  cl::init(64), cl::cat(Cat));
+
+static llvm::cl::opt<LanePreference> VecLane0(
+    "vec-lane-0",
+    cl::desc("Set whether lane 0 is treated as the high- or low-end "
+             "bytes of the vector blocks ([most|least]-significant)."),
+    cl::values(clEnumVal(LeastSignificant,
+                         "Vector lane 0 is in the least-significant bytes"),
+               clEnumVal(MostSignificant,
+                         "Vector lane 0 is in the most-significant bytes")),
+    cl::init(LeastSignificant), cl::cat(Cat));
+
 // Define a TargetTransformInfo (TTI) subclass, this allows for overriding
 // common per-llvm-target information expected by other LLVM passes, such
 // as the width of the largest scalar/vector registers.  Needed for consistent
@@ -146,10 +193,24 @@ int main(int argc, char **argv) {
     }
 #endif
 
+    if (!isPowerOf2_32(VecBlockBytes) or VecBlockBytes < 8) {
+        errs() << "--" << VecBlockBytes.ArgStr
+               << " must be a power-of-two and at least 8 bytes.";
+        return 1;
+    }
+
     LLVMContext Context;
     SMDiagnostic Err;
     std::unique_ptr<Module> M = parseIRFile(InputFile, Err, Context);
 
+    // After we have a module we can get the data layout and determine host
+    // endianness.
+    VectorLayout VL = {
+        .HostBigEndian = VecForceBigEndian or M->getDataLayout().isBigEndian(),
+        .Lane0 = VecLane0,
+        .BlockBytes = VecBlockBytes,
+    };
+
     // Create a new TargetMachine to represent a TCG target,
     // we use x86_64 as a base and derive from that using a
     // TargetTransformInfo to provide allowed scalar and vector
@@ -231,7 +292,7 @@ int main(int argc, char **argv) {
     //
 
     TcgGlobalMap TcgGlobals;
-    MPM.addPass(PrepareForTcgPass(TcgGlobals, DebugInfo));
+    MPM.addPass(PrepareForTcgPass(TcgGlobals, DebugInfo, VL));
     MPM.addPass(VerifierPass());
     {
         FunctionPassManager FPM;
diff --git a/subprojects/helper-to-tcg/src/PrepareForTcgPass/CanonicalizeIR.cpp b/subprojects/helper-to-tcg/src/PrepareForTcgPass/CanonicalizeIR.cpp
new file mode 100644
index 0000000000..54fdcade4e
--- /dev/null
+++ b/subprojects/helper-to-tcg/src/PrepareForTcgPass/CanonicalizeIR.cpp
@@ -0,0 +1,1193 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#include "CanonicalizeIR.hpp"
+#include "CmdLineOptions.hpp"
+#include "LlvmCompat.hpp"
+#include "PseudoInst.hpp"
+#include "VectorLayout.hpp"
+
+#include <llvm/ADT/PostOrderIterator.h>
+#include <llvm/ADT/SmallPtrSet.h>
+#include <llvm/ADT/SmallSet.h>
+#include <llvm/ADT/SmallVector.h>
+#include <llvm/Analysis/RegionInfo.h>
+#include <llvm/Analysis/VectorUtils.h>
+#include <llvm/IR/Constants.h>
+#include <llvm/IR/DerivedTypes.h>
+#include <llvm/IR/Dominators.h>
+#include <llvm/IR/Function.h>
+#include <llvm/IR/IRBuilder.h>
+#include <llvm/IR/InstIterator.h>
+#include <llvm/IR/InstrTypes.h>
+#include <llvm/IR/Instruction.h>
+#include <llvm/IR/Instructions.h>
+#include <llvm/IR/Intrinsics.h>
+#include <llvm/IR/Module.h>
+#include <llvm/IR/PatternMatch.h>
+#include <llvm/Support/Casting.h>
+
+#include <algorithm> // for std::max/min
+
+using namespace llvm;
+using namespace PatternMatch;
+
+// Needed to track and remove instructions not handled by a subsequent dead code
+// elimination, this applies to calls to pseudo instructions in particular.
+using EraseInstVec = SmallVector<Instruction *, 16>;
+using UsageCountMap = DenseMap<Value *, uint16_t>;
+using ExitSet = SmallSet<BasicBlock *, 2>;
+
+// Helper function to remove an instruction only if all uses have been removed.
+// This way we can keep track instruction uses without having to modify the IR,
+// or without having to iterate over all uses everytime we wish to remove an
+// instruction.
+static void addToEraseVectorIfUnused(EraseInstVec &InstToErase,
+                                     UsageCountMap &UsageMap, Value *V) {
+    auto *I = dyn_cast<Instruction>(V);
+    if (!I) {
+        return;
+    }
+
+    // Add V to map if not there
+    if (UsageMap.count(V) == 0) {
+        UsageMap[V] = V->getNumUses();
+    }
+
+    // Erase if count reaches zero
+    if (--UsageMap[V] == 0) {
+        InstToErase.push_back(I);
+        UsageMap.erase(V);
+    }
+}
+
+// Forward declarations of IR transformations used in canonicalizing the IR
+static void upcastAshr(Instruction *I);
+static void convertInsertShuffleToSplat(EraseInstVec &InstToErase,
+                                        UsageCountMap &UsageMap, Module &M,
+                                        Instruction *I);
+
+static void
+defineVectorConstants(Module &M, const VectorLayout &VL, Instruction *I,
+                      DenseMap<Constant *, CallInst *> &Replacements);
+
+static void simplifyVecBinOpWithSplat(EraseInstVec &InstToErase,
+                                      UsageCountMap &UsageMap, Module &M,
+                                      BinaryOperator *BinOp);
+
+static void convertSelectICmp(Module &M, SelectInst *Select, ICmpInst *ICmp);
+
+static void convertQemuLoadStoreToPseudoInst(Module &M, CallInst *Call,
+                                             EraseInstVec &InstToErase,
+                                             UsageCountMap &UsageMap);
+static void convertExceptionCallsToPseudoInst(Module &M, CallInst *Call);
+static void convertReturnAddrToPseudoInst(Module &M, CallInst *Call);
+static void convertImmediateSelectAccessGlobal(EraseInstVec &InstToErase,
+                                               Module &M, CallInst *Call);
+static void convertImmediateDeclCall(EraseInstVec &InstToErase, Module &M,
+                                     CallInst *Call);
+static void convertUserBranchInstructions(Module &M,
+                                          DominatorTreeAnalysis::Result &DT,
+                                          const ExitSet &Exits, CallInst *Call);
+static void convertVecStoreToPseudoInst(EraseInstVec &InstToErase, Module &M,
+                                        StoreInst *Store);
+
+void canonicalizeIR(Module &M, ModuleAnalysisManager &MAM,
+                    const VectorLayout &VL) {
+    auto &FAM =
+        MAM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
+    const bool HasUserBranches = !UserPCRelBranchFunc.empty() and
+                                 !UserPCRelBranchConditionalFunc.empty() and
+                                 !UserPCRelBranchFallthroughFunc.empty();
+    for (Function &F : M) {
+        if (F.isDeclaration()) {
+            continue;
+        }
+
+        EraseInstVec InstToErase;
+        UsageCountMap UsageMap;
+        ExitSet Exits;
+        auto &DT = FAM.getResult<DominatorTreeAnalysis>(F);
+
+        // Collect basic blocks which exit the function, needed to support
+        // user-supplied branch calls, and correctly handle conditional branches
+        if (HasUserBranches) {
+            for (auto &BB : F) {
+                if (isa<ReturnInst>(BB.getTerminator())) {
+                    Exits.insert(&BB);
+                }
+            }
+        }
+
+        SmallPtrSet<Constant *, 8> ReplacedVectorConstants;
+
+        // Perform a first pass over all instructions in the function and apply
+        // IR transformations sequentially.  NOTE: order matters here.
+        for (Instruction &I : instructions(F)) {
+            if (I.isArithmeticShift()) {
+                upcastAshr(&I);
+            }
+
+            convertInsertShuffleToSplat(InstToErase, UsageMap, M, &I);
+
+            // Depends on convertInsertShuffleToSplat for @VecSplat instructions
+            if (auto *BinOp = dyn_cast<BinaryOperator>(&I)) {
+                simplifyVecBinOpWithSplat(InstToErase, UsageMap, M, BinOp);
+            }
+
+            // Independent of above
+            if (auto *ICmp = dyn_cast<ICmpInst>(&I)) {
+                for (auto *U : ICmp->users()) {
+                    auto *Select = dyn_cast<SelectInst>(U);
+                    if (Select and Select->getCondition() == ICmp) {
+                        convertSelectICmp(M, Select, ICmp);
+                    }
+                }
+            }
+
+            // Independent of above, can run at any point
+            if (auto *Call = dyn_cast<CallInst>(&I)) {
+                convertQemuLoadStoreToPseudoInst(M, Call, InstToErase,
+                                                 UsageMap);
+                convertExceptionCallsToPseudoInst(M, Call);
+                convertReturnAddrToPseudoInst(M, Call);
+                convertImmediateSelectAccessGlobal(InstToErase, M, Call);
+                convertImmediateDeclCall(InstToErase, M, Call);
+                if (HasUserBranches) {
+                    convertUserBranchInstructions(M, DT, Exits, Call);
+                }
+            }
+
+            // Depends on other vector conversions performed above, needs to
+            // run last
+            if (auto *Store = dyn_cast<StoreInst>(&I)) {
+                convertVecStoreToPseudoInst(InstToErase, M, Store);
+            }
+        }
+
+        // Finally clean up instructions we need to remove manually
+        for (Instruction *I : InstToErase) {
+            I->eraseFromParent();
+        }
+
+        DenseMap<Constant *, CallInst *> Replacements;
+        for (Instruction &I : instructions(F)) {
+            defineVectorConstants(M, VL, &I, Replacements);
+        }
+    }
+}
+
+static Value *upcastInt(IRBuilder<> &Builder, IntegerType *FinalIntTy,
+                        Value *V) {
+    if (auto *ConstInt = dyn_cast<ConstantInt>(V)) {
+        return ConstantInt::get(FinalIntTy, ConstInt->getZExtValue());
+    } else {
+        return Builder.CreateSExt(V, FinalIntTy);
+    }
+}
+
+// Convert
+//
+//   %2 = ashr i[8|16] %1, %0
+//
+// to
+//
+//   %2 = zext i[8|16] %1 to i32
+//   %3 = zext i[8|16] %2 to i32
+//   %2 = ashr i32 %2, %3
+//
+static void upcastAshr(Instruction *I) {
+    // Only care about scalar shifts < on less than 32-bit integers
+    auto *IntTy = dyn_cast<IntegerType>(I->getType());
+    if (!IntTy or IntTy->getBitWidth() >= 32) {
+        return;
+    }
+
+    IRBuilder<> Builder(I);
+
+    Value *Op1 = I->getOperand(0);
+    Value *Op2 = I->getOperand(1);
+    auto *UpcastIntTy = Builder.getInt32Ty();
+    Op1 = upcastInt(Builder, UpcastIntTy, Op1);
+    Op2 = upcastInt(Builder, UpcastIntTy, Op2);
+
+    auto *AShr = Builder.CreateAShr(Op1, Op2);
+    auto *Trunc = Builder.CreateTrunc(AShr, I->getType());
+    I->replaceAllUsesWith(Trunc);
+}
+
+// Convert vector intrinsics
+//
+//   %0 = insertelement ...
+//   %1 = shuffle ...
+//
+// to
+//
+//   %0 = call @VecSplat.*
+//
+static void convertInsertShuffleToSplat(EraseInstVec &InstToErase,
+                                        UsageCountMap &UsageMap, Module &M,
+                                        Instruction *I) {
+    Value *SplatV;
+    if (match(I, compat_m_Shuffle(compat_m_InsertElt(m_Value(), m_Value(SplatV),
+                                                     m_ZeroInt()),
+                                  m_Value(), compat_m_ZeroMask()))) {
+        auto *VecTy = cast<VectorType>(I->getType());
+        IRBuilder<> Builder(I);
+        I->replaceAllUsesWith(
+            createPseudoInstCall(M, Builder, VecSplat, VecTy, {SplatV}));
+        addToEraseVectorIfUnused(InstToErase, UsageMap, I->getOperand(0));
+        InstToErase.push_back(I);
+    }
+}
+
+static void
+defineVectorConstants(Module &M, const VectorLayout &VL, Instruction *I,
+                      DenseMap<Constant *, CallInst *> &Replacements) {
+    for (size_t J = 0; J < I->getNumOperands(); ++J) {
+        Value *Op = I->getOperand(J);
+        auto *Const = dyn_cast<Constant>(Op);
+        auto *VecTy = dyn_cast<VectorType>(Op->getType());
+        if (!Const or !VecTy) {
+            // Only care about non-splatted constant vectors, skip
+            // everything else.
+            continue;
+        }
+
+        if (Replacements.count(Const)) {
+            I->setOperand(J, Replacements[Const]);
+            continue;
+        }
+
+        if (Value *Splat = Const->getSplatValue()) {
+            auto *VecTy = cast<VectorType>(Const->getType());
+            IRBuilder<> Builder(I);
+            CallInst *Call =
+                createPseudoInstCall(M, Builder, VecSplat, VecTy, {Splat});
+            I->setOperand(J, Call);
+            Replacements[Const] = Call;
+        } else {
+            // Constant non-splatted vector, attempt to combine elements
+            // to make it splattable.
+            SmallVector<uint64_t, 16> Ints;
+
+            // Copy over elements to a vector
+            const unsigned ElementCount = compat::getVectorElementCount(VecTy);
+            const unsigned ElementSize =
+                VecTy->getElementType()->getIntegerBitWidth();
+
+            for (unsigned I = 0; I < ElementCount; ++I) {
+                Constant *Element = Const->getAggregateElement(I);
+                uint64_t Value = Element->getUniqueInteger().getZExtValue();
+                Ints.push_back(Value);
+            }
+
+            // When combining adjacent elements, the maximum size supported
+            // by TCG is 64-bit.  MaxNumElements is the maximum amount of
+            // elements to attempt to merge
+            size_t PatternLen = 0;
+            const unsigned MaxNumElements = 8 * sizeof(uint64_t) / ElementSize;
+            for (unsigned N = MaxNumElements; N > 1; N /= 2) {
+                // Attempt to combine N elements by checking if the first
+                // N elements tile the vector.
+                bool Match = true;
+                for (unsigned J = 0; J < ElementCount; ++J) {
+                    if (Ints[J % N] != Ints[J]) {
+                        Match = false;
+                        break;
+                    }
+                }
+                // If tiling succeeded, break out
+                if (Match) {
+                    PatternLen = N;
+                    break;
+                }
+            }
+
+            if (PatternLen > 0) {
+                // Managed to tile vector with splattable element, compute
+                // final splattable value
+                // TODO: Move to VectorLayout.hpp
+                uint64_t Column = 0;
+                for (unsigned I = 0; I < PatternLen; ++I) {
+                    // Flips the indices if lane 0 is in the most-significant
+                    // bits.
+                    const size_t Index = VL.indexLane(PatternLen, 0, I);
+                    Column |= Ints[Index] << I * ElementSize;
+                }
+                IRBuilder<> Builder(I);
+                CallInst *Call = createPseudoInstCall(
+                    M, Builder, VecSplat, VecTy, {Builder.getInt64(Column)});
+                I->setOperand(J, Call);
+                Replacements[Const] = Call;
+            } else {
+                // Tiling failed, fall back to emitting an array copy from
+                // C to a gvec vector.
+                // TODO: Move to VectorLayout.hpp
+                IRBuilder<> Builder(I);
+                SmallVector<Constant *, 8> Columns;
+                const size_t VectorSize =
+                    (ElementCount * (ElementSize / 8) + 7);
+                const size_t ColumnCount = VectorSize / 8;
+                Columns.resize(ColumnCount);
+                for (size_t C = 0; C < ColumnCount; ++C) {
+                    uint64_t Column = 0;
+                    const size_t LanesPerColumn = 64 / ElementSize;
+                    for (size_t L = 0; L < LanesPerColumn; ++L) {
+                        const size_t Index = VL.indexLane(LanesPerColumn, C, L);
+                        Column |= Ints[Index] << (ElementSize * L);
+                    }
+                    assert(VL.BlockBytes >= 8);
+                    const size_t Size =
+                        std::min(ColumnCount, VL.BlockBytes / 8);
+                    const size_t Index = VL.indexBlock(Size, C);
+                    Columns[Index] = Builder.getInt64(Column);
+                }
+                CallInst *Call =
+                    createPseudoInstCall(M, Builder, VecConstant, VecTy,
+                                         {ConstantVector::get(Columns)});
+                I->setOperand(J, Call);
+                Replacements[Const] = Call;
+            }
+        }
+    }
+}
+
+// Convert
+//
+//   %1 = @VecSplat(%0)
+//   %2 = <NxM> ... op <NxM> %1
+//
+// to
+//
+//   %2 = call @Vec[op]Scalar(..., %0)
+//
+// which more closely matches TCG gvec operations.
+static void simplifyVecBinOpWithSplat(EraseInstVec &InstToErase,
+                                      UsageCountMap &UsageMap, Module &M,
+                                      BinaryOperator *BinOp) {
+    Value *Lhs = BinOp->getOperand(0);
+    Value *Rhs = BinOp->getOperand(1);
+    if (!Lhs->getType()->isVectorTy() or !Rhs->getType()->isVectorTy()) {
+        return;
+    }
+
+    // Get splat value from constant or @VecSplat call
+    Value *SplatValue = nullptr;
+    if (auto *Const = dyn_cast<Constant>(Rhs)) {
+        SplatValue = Const->getSplatValue();
+    } else if (auto *Call = dyn_cast<CallInst>(Rhs)) {
+        if (getPseudoInstFromCall(Call) == VecSplat) {
+            SplatValue = Call->getOperand(0);
+        }
+    }
+
+    if (SplatValue == nullptr) {
+        return;
+    }
+
+    auto *VecTy = cast<VectorType>(Lhs->getType());
+    auto *ConstInt = dyn_cast<ConstantInt>(SplatValue);
+    bool ConstIsNegOne = ConstInt and ConstInt->getSExtValue() == -1;
+    bool IsNot = BinOp->getOpcode() == Instruction::Xor and ConstIsNegOne;
+    if (IsNot) {
+        IRBuilder<> Builder(BinOp);
+        BinOp->replaceAllUsesWith(
+            createPseudoInstCall(M, Builder, VecNot, VecTy, {Lhs}));
+    } else {
+        PseudoInst Inst;
+        switch (BinOp->getOpcode()) {
+        case Instruction::Add:
+            Inst = VecAddScalar;
+            break;
+        case Instruction::Sub:
+            Inst = VecSubScalar;
+            break;
+        case Instruction::Mul:
+            Inst = VecMulScalar;
+            break;
+        case Instruction::Xor:
+            Inst = VecXorScalar;
+            break;
+        case Instruction::Or:
+            Inst = VecOrScalar;
+            break;
+        case Instruction::And:
+            Inst = VecAndScalar;
+            break;
+        case Instruction::Shl:
+            Inst = VecShlScalar;
+            break;
+        case Instruction::LShr:
+            Inst = VecLShrScalar;
+            break;
+        case Instruction::AShr:
+            Inst = VecAShrScalar;
+            break;
+        default:
+            abort();
+        }
+
+        IRBuilder<> Builder(BinOp);
+        // Scalar gvec shift operations uses 32-bit scalars, whereas arithmetic
+        // operations uses 64-bit scalars.
+        uint32_t SplatSize = SplatValue->getType()->getIntegerBitWidth();
+        if (BinOp->isShift()) {
+            if (SplatSize > 32) {
+                SplatValue =
+                    Builder.CreateTrunc(SplatValue, Builder.getInt32Ty());
+            }
+        } else {
+            if (SplatSize < 64) {
+                SplatValue =
+                    Builder.CreateZExt(SplatValue, Builder.getInt64Ty());
+            }
+        }
+        BinOp->replaceAllUsesWith(
+            createPseudoInstCall(M, Builder, Inst, VecTy, {Lhs, SplatValue}));
+    }
+
+    InstToErase.push_back(BinOp);
+    addToEraseVectorIfUnused(InstToErase, UsageMap, Rhs);
+}
+
+// Convert
+//
+//   %2 = icmp [sgt|ugt|slt|ult] %0, %1
+//   %5 = select %2, %3, %4
+//
+// to
+//
+//   %5 = [s|u][max|min] %0, %1
+//
+// if possible.  Results in cleaner IR, particularly useful for vector
+// instructions.
+static bool convertSelectICmpToMinMax(Module &M, SelectInst *Select,
+                                      ICmpInst *ICmp, ICmpInst::Predicate &Pred,
+                                      Value *ICmpOp0, Value *ICmpOp1,
+                                      Value *SelectOp0, Value *SelectOp1) {
+    if (ICmpOp0 != SelectOp0 or ICmpOp1 != SelectOp1) {
+        return false;
+    }
+
+    Intrinsic::ID Intrin;
+    switch (Pred) {
+    case ICmpInst::ICMP_SGT:
+        Intrin = Intrinsic::smax;
+        break;
+    case ICmpInst::ICMP_UGT:
+        Intrin = Intrinsic::umax;
+        break;
+    case ICmpInst::ICMP_SLT:
+        Intrin = Intrinsic::smin;
+        break;
+    case ICmpInst::ICMP_ULT:
+        Intrin = Intrinsic::umin;
+        break;
+    default:
+        return false;
+    }
+
+    auto Ty = Select->getType();
+    auto MaxMinF = compat::Intrinsic::getOrInsertDeclaration(&M, Intrin, {Ty});
+
+    IRBuilder<> Builder(Select);
+    auto Call = Builder.CreateCall(MaxMinF, {ICmpOp0, ICmpOp1});
+    Select->replaceAllUsesWith(Call);
+
+    return true;
+}
+
+// In LLVM, icmp on vectors returns a vector on i1s whereas TCGs gvec_cmp
+// returns a vector of the element type of its operands.  This can result in
+// some subtle bugs.  Convert
+//
+//   icmp -> call @VecCompare
+//   select -> call @VecWideCondBitsel
+//
+static bool convertSelectICmpToVecBitsel(Module &M, SelectInst *Select,
+                                         ICmpInst *ICmp,
+                                         ICmpInst::Predicate &Pred,
+                                         Value *ICmpOp0, Value *ICmpOp1,
+                                         Value *SelectOp0, Value *SelectOp1) {
+    auto *ICmpVecTy = dyn_cast<VectorType>(ICmpOp0->getType());
+    auto *SelectVecTy = dyn_cast<VectorType>(Select->getType());
+    if (!ICmpVecTy or !SelectVecTy) {
+        return false;
+    }
+
+    Instruction *Cmp = ICmp;
+    {
+        IRBuilder<> Builder(Cmp);
+        ICmpInst::Predicate Pred = ICmp->getPredicate();
+        CallInst *Call = createPseudoInstCall(
+            M, Builder, VecCompare, ICmpVecTy,
+            {ConstantInt::get(Builder.getInt32Ty(), Pred), ICmpOp0, ICmpOp1});
+        Cmp = Call;
+    }
+
+    unsigned SrcWidth = ICmpVecTy->getElementType()->getIntegerBitWidth();
+    unsigned DstWidth = SelectVecTy->getElementType()->getIntegerBitWidth();
+
+    IRBuilder<> Builder(Select);
+    if (SrcWidth < DstWidth) {
+        Cmp = cast<Instruction>(Builder.CreateSExt(Cmp, SelectVecTy));
+    } else if (SrcWidth > DstWidth) {
+        Cmp = cast<Instruction>(Builder.CreateTrunc(Cmp, SelectVecTy));
+    }
+    Select->replaceAllUsesWith(
+        createPseudoInstCall(M, Builder, VecWideCondBitsel, SelectVecTy,
+                             {Cmp, SelectOp0, SelectOp1}));
+
+    return true;
+}
+
+// Convert
+//
+//   %2 = icmp [sgt|ugt|slt|ult] %0, %1
+//   %5 = select %2, %3, %4
+//
+// to
+//
+//   5 = call @Movcond.[cond].*(%1, %0, %3, %4)
+//
+// to more closely match TCG semantics.
+static bool convertSelectICmpToMovcond(Module &M, SelectInst *Select,
+                                       ICmpInst *ICmp,
+                                       ICmpInst::Predicate &Pred,
+                                       Value *ICmpOp0, Value *ICmpOp1,
+                                       Value *SelectOp0, Value *SelectOp1) {
+    // We only handle integers, we have no movcond equivalent in gvec
+    auto *IntTy = dyn_cast<IntegerType>(Select->getType());
+    if (!IntTy) {
+        return false;
+    }
+
+    // If the type of the comparison does not match the return type of the
+    // select statement, we cannot do anything so skip
+    if (ICmpOp0->getType() != IntTy) {
+        return false;
+    }
+
+    IRBuilder<> Builder(Select);
+    if (cast<IntegerType>(ICmpOp0->getType())->getBitWidth() <
+        IntTy->getBitWidth()) {
+        if (ICmp->isSigned(Pred)) {
+            ICmpOp0 = Builder.CreateSExt(ICmpOp0, IntTy);
+            ICmpOp1 = Builder.CreateSExt(ICmpOp1, IntTy);
+        } else {
+            ICmpOp0 = Builder.CreateZExt(ICmpOp0, IntTy);
+            ICmpOp1 = Builder.CreateZExt(ICmpOp1, IntTy);
+        }
+    }
+
+    // Create @Movcond.[slt|...].* function
+    Select->replaceAllUsesWith(
+        createPseudoInstCall(M, Builder, Movcond, IntTy,
+                             {ConstantInt::get(IntTy, Pred), ICmpOp0, ICmpOp1,
+                              SelectOp0, SelectOp1}));
+
+    return true;
+}
+
+// Specialize
+//
+//   %2 = icmp [sgt|ugt|slt|ult] %0, %1
+//   %5 = select %2, %3, %4
+//
+// to either maximum/minimum, vector operations matching TCG, or a conditional
+// move that also matches TCG in sematics.
+static void convertSelectICmp(Module &M, SelectInst *Select, ICmpInst *ICmp) {
+    // Given
+    //   %2 = icmp [sgt|ugt|slt|ult] %0, %1
+    //   %5 = select %2, %3, %4
+    assert(Select->getCondition() == ICmp);
+    Value *ICmpOp0 = ICmp->getOperand(0);
+    Value *ICmpOp1 = ICmp->getOperand(1);
+    Value *SelectOp0 = Select->getTrueValue();
+    Value *SelectOp1 = Select->getFalseValue();
+    ICmpInst::Predicate Pred = ICmp->getPredicate();
+
+    // First try to convert to min/max
+    //   %5 = [s|u][max|min] %0, %1
+    if (convertSelectICmpToMinMax(M, Select, ICmp, Pred, ICmpOp0, ICmpOp1,
+                                  SelectOp0, SelectOp1)) {
+        return;
+    }
+
+    // Secondly try convert icmp -> @VecCompare, select -> @VecWideCondBitsel
+    if (convertSelectICmpToVecBitsel(M, Select, ICmp, Pred, ICmpOp0, ICmpOp1,
+                                     SelectOp0, SelectOp1)) {
+        return;
+    }
+
+    // If min/max and vector conversion failed we fallback to a movcond
+    //   %5 = call @Movcond.[cond].*(%1, %0, %3, %4)
+    convertSelectICmpToMovcond(M, Select, ICmp, Pred, ICmpOp0, ICmpOp1,
+                               SelectOp0, SelectOp1);
+}
+
+// Convert QEMU guest loads/stores represented by calls such as
+//
+//   call cpu_ldsw_be*(),
+//   call cpu_stq_le*(),
+//
+// and friends, to pseudo instructions
+//
+//   %5 = call @GuestLoad.*(%addr, %sign, %size, %endian);
+//   %5 = call @GuestStore.*(%addr, %value, %size, %endian);
+//
+// Makes the backend agnostic to what instructions or calls are used to
+// represent loads and stores.
+static void convertQemuLoadStoreToPseudoInst(Module &M, CallInst *Call,
+                                             EraseInstVec &InstToErase,
+                                             UsageCountMap &UsageMap) {
+    Function *F = Call->getCalledFunction();
+    StringRef Name = F->getName();
+    if (Name.consume_front("cpu_")) {
+        bool IsLoad = Name.consume_front("ld");
+        bool IsStore = !IsLoad and Name.consume_front("st");
+        if (IsLoad or IsStore) {
+            bool Signed = !Name.consume_front("u");
+
+            uint8_t Size = 0;
+            switch (Name[0]) {
+            case 'b':
+                Size = 1;
+                break;
+            case 'w':
+                Size = 2;
+                break;
+            case 'l':
+                Size = 4;
+                break;
+            case 'q':
+                Size = 8;
+                break;
+            default:
+                abort();
+            }
+
+            uint8_t Endianness = 0;
+            if (Size > 1 and Size < 8) {
+                Name = Name.drop_front(2);
+                switch (Name[0]) {
+                case 'l':
+                    Endianness = 1;
+                    break;
+                case 'b':
+                    Endianness = 2;
+                    break;
+                default:
+                    // TODO: we need to parse the `MemOpIndex` in the second to
+                    // last argument, for now just assume little endian, same
+                    // goes for signedness.
+                    Endianness = 1;
+                }
+            }
+
+            IRBuilder<> Builder(Call);
+            Value *AddrOp = Call->getArgOperand(1);
+            IntegerType *FlagTy = Builder.getInt8Ty();
+            Value *SizeOp = ConstantInt::get(FlagTy, Size);
+            Value *EndianOp = ConstantInt::get(FlagTy, Endianness);
+            CallInst *NewCall;
+            if (IsLoad) {
+                Value *SignOp = ConstantInt::get(FlagTy, Signed);
+                IntegerType *RetTy = cast<IntegerType>(Call->getType());
+                NewCall =
+                    createPseudoInstCall(M, Builder, GuestLoad, RetTy,
+                                         {AddrOp, SignOp, SizeOp, EndianOp});
+            } else {
+                Value *ValueOp = Call->getArgOperand(2);
+                NewCall = createPseudoInstCall(
+                    M, Builder, GuestStore, Builder.getVoidTy(),
+                    {AddrOp, ValueOp, SizeOp, EndianOp});
+            }
+            Call->replaceAllUsesWith(NewCall);
+            InstToErase.push_back(Call);
+        }
+    }
+}
+
+// Convert QEMU exception calls
+//
+//   call raise_exception_ra(...),
+//   ...
+//
+// to a pseudo instruction
+//
+//   %5 = call @Exception.*(...);
+//
+// Makes the backend agnostic to what instructions or calls are used to
+// represent exceptions, and the list of sources can be expanded here.
+static void convertExceptionCallsToPseudoInst(Module &M, CallInst *Call) {
+    Function *F = Call->getCalledFunction();
+    if (F->getName() == "raise_exception_ra") {
+        IRBuilder<> Builder(Call);
+        Value *Op0 = Call->getArgOperand(0);
+        Value *Op1 = Call->getArgOperand(1);
+        Call->replaceAllUsesWith(createPseudoInstCall(
+            M, Builder, Exception, Builder.getVoidTy(), {Op0, Op1}));
+    }
+}
+
+// Convert QEMU exception calls
+//
+//   %0 = call @llvm.returnaddr(...),
+//   %1 = ptrtoint %0
+//   ...
+//
+// to a pseudo instruction
+//
+//   %0 = call @getpc(...);
+//
+static void convertReturnAddrToPseudoInst(Module &M, CallInst *Call) {
+    Function *F = Call->getCalledFunction();
+    if (!F->isIntrinsic() or F->getIntrinsicID() != Intrinsic::returnaddress) {
+        return;
+    }
+    if (!Call->hasOneUse()) {
+        return;
+    }
+
+    auto *PtrToInt = dyn_cast<PtrToIntInst>(*Call->user_begin());
+    if (!PtrToInt) {
+        return;
+    }
+
+    IRBuilder<> Builder(PtrToInt);
+    PtrToInt->replaceAllUsesWith(
+        createPseudoInstCall(M, Builder, GetPC, PtrToInt->getType(), {}));
+}
+
+// Convert QEMU exception calls
+//
+//   %0 = call @llvm.returnaddr(...),
+//   %1 = ptrtoint %0
+//   ...
+//
+// to a pseudo instruction
+//
+//   %0 = call @getpc(...);
+//
+static void convertImmediateSelectAccessGlobal(EraseInstVec &InstToErase,
+                                               Module &M, CallInst *Call) {
+    PseudoInst PI = getPseudoInstFromCall(Call);
+    if (PI != AccessGlobalArray) {
+        return;
+    }
+
+    if (!Call->hasOneUser()) {
+        return;
+    }
+
+    auto *Load = dyn_cast<LoadInst>(*Call->user_begin());
+    if (!Load) {
+        return;
+    }
+
+    auto *Zext = dyn_cast<ZExtInst>(Call->getArgOperand(1));
+    if (!Zext) {
+        return;
+    }
+
+    auto *Select = dyn_cast<SelectInst>(Zext->getOperand(0));
+    if (!Select) {
+        return;
+    }
+
+    if (!Zext->hasOneUse() or !Select->hasOneUse()) {
+        return;
+    }
+
+    Value *Cond = Select->getCondition();
+    Value *Arg1 = Select->getTrueValue();
+    Value *Arg2 = Select->getFalseValue();
+
+    if (!isa<Argument>(Arg1) or !isa<Argument>(Arg2)) {
+        return;
+    }
+
+    IRBuilder<> Builder(Select);
+    Function *AccessGlobalFn = Call->getCalledFunction();
+    Value *TypeIndex = Call->getArgOperand(0);
+    Value *Offset = Call->getArgOperand(1);
+    Value *Arg1Zext = Builder.CreateZExt(Arg1, Zext->getType());
+    Value *Arg2Zext = Builder.CreateZExt(Arg2, Zext->getType());
+    CallInst *Access1 =
+        Builder.CreateCall(AccessGlobalFn, {TypeIndex, Offset, Arg1Zext});
+    CallInst *Access2 =
+        Builder.CreateCall(AccessGlobalFn, {TypeIndex, Offset, Arg2Zext});
+    LoadInst *Load1 = Builder.CreateLoad(Load->getType(), Access1);
+    LoadInst *Load2 = Builder.CreateLoad(Load->getType(), Access2);
+    Value *NewSelect = Builder.CreateSelect(Cond, Load1, Load2);
+
+    InstToErase.push_back(Load);
+    InstToErase.push_back(Call);
+
+    Load->replaceAllUsesWith(NewSelect);
+}
+
+struct DeclReplaceInfo {
+    bool Valid = false;
+    Value *Zext = nullptr;
+    Value *Select = nullptr;
+    Value *Cond = nullptr;
+    Value *Arg1 = nullptr;
+    Value *Arg2 = nullptr;
+};
+
+static DeclReplaceInfo isReplaceableDeclCall(Value *A) {
+    auto *Zext = dyn_cast<ZExtInst>(A);
+    if (!Zext) {
+        return {};
+    }
+
+    auto *Select = dyn_cast<SelectInst>(Zext->getOperand(0));
+    if (!Select) {
+        return {};
+    }
+
+    if (!Zext->hasOneUse() or !Select->hasOneUse()) {
+        return {};
+    }
+
+    Value *Cond = Select->getCondition();
+    Value *Arg1 = Select->getTrueValue();
+    Value *Arg2 = Select->getFalseValue();
+
+    if (!isa<Argument>(Arg1) or !isa<Argument>(Arg2)) {
+        return {};
+    }
+
+    return {true, Zext, Select, Cond, Arg1, Arg2};
+}
+
+static void convertImmediateDeclCall(EraseInstVec &InstToErase, Module &M,
+                                     CallInst *Call) {
+    if (!Call->getCalledFunction()->isDeclaration()) {
+        return;
+    }
+
+    if (!Call->hasOneUser()) {
+        return;
+    }
+
+    SmallVector<Value *, 8> Args1;
+    SmallVector<Value *, 8> Args2;
+    DeclReplaceInfo Info{};
+    IRBuilder<> Builder(Call);
+    for (auto &A : Call->args()) {
+        if (!Info.Valid) {
+            Info = isReplaceableDeclCall(A);
+            if (Info.Valid) {
+                Value *Arg1Zext =
+                    Builder.CreateZExt(Info.Arg1, Info.Zext->getType());
+                Value *Arg2Zext =
+                    Builder.CreateZExt(Info.Arg2, Info.Zext->getType());
+                Args1.push_back(Arg1Zext);
+                Args2.push_back(Arg2Zext);
+                continue;
+            }
+        }
+
+        Args1.push_back(A);
+        Args2.push_back(A);
+    }
+
+    if (!Info.Valid) {
+        return;
+    }
+
+    Function *F = Call->getCalledFunction();
+    CallInst *Access1 = Builder.CreateCall(F, Args1);
+    CallInst *Access2 = Builder.CreateCall(F, Args2);
+    Value *NewSelect = Builder.CreateSelect(Info.Cond, Access1, Access2);
+    Access1->setDebugLoc(Call->getDebugLoc());
+    Access2->setDebugLoc(Call->getDebugLoc());
+
+    // InstToErase.push_back(Load);
+    InstToErase.push_back(Call);
+    InstToErase.push_back(cast<Instruction>(Info.Zext));
+
+    Call->replaceAllUsesWith(NewSelect);
+}
+
+static void convertUserBranchInstructions(Module &M,
+                                          DominatorTreeAnalysis::Result &DT,
+                                          const ExitSet &Exits,
+                                          CallInst *Call) {
+    Function *F = Call->getCalledFunction();
+    if (F->getName() != UserPCRelBranchFunc) {
+        return;
+    }
+
+    // Make sure the user-supplied functions exist and have the correct
+    // signature.
+    Function *BranchF = M.getFunction(UserPCRelBranchFunc);
+    Function *CondF = M.getFunction(UserPCRelBranchConditionalFunc);
+    Function *FallF = M.getFunction(UserPCRelBranchFallthroughFunc);
+    if (!CondF or !FallF or BranchF->getType() != CondF->getType() or
+        FallF->getType()->getFunctionNumParams()) {
+        return;
+    }
+
+    BasicBlock *BB = Call->getParent();
+    bool DominatesAllExits = true;
+    for (BasicBlock *E : Exits) {
+        if (!DT.dominates(BB, E)) {
+            DominatesAllExits = false;
+        }
+    }
+    // If the call to the user-supplied pc-relative jump function dominates all
+    // exit blocks, then leave it be, it's a single relative jump with one exit.
+    if (DominatesAllExits) {
+        return;
+    }
+
+    // Otherwise add a fallthrough operation to each exit block.
+    for (BasicBlock *E : Exits) {
+        IRBuilder<> Builder(E->getTerminator());
+        Builder.CreateCall(FallF, {});
+    }
+
+    // And replace the current jump operation with a "conditinal" one, so
+    // QEMU knows how to chain the generated code.
+    Call->setCalledFunction(CondF);
+}
+
+//
+// Following functions help with converting between different types of
+// instructions to pseudo instructions, particularly ones that write
+// to a pointer, aka the Vec*Store pseudo instructions
+//
+
+static PseudoInst instructionToStorePseudoInst(unsigned Opcode) {
+    switch (Opcode) {
+    case Instruction::Trunc:
+        return VecTruncStore;
+    case Instruction::ZExt:
+        return VecZExtStore;
+    case Instruction::SExt:
+        return VecSExtStore;
+    case Instruction::Select:
+        return VecSelectStore;
+    case Instruction::Add:
+        return VecAddStore;
+    case Instruction::Sub:
+        return VecSubStore;
+    case Instruction::Mul:
+        return VecMulStore;
+    case Instruction::Xor:
+        return VecXorStore;
+    case Instruction::Or:
+        return VecOrStore;
+    case Instruction::And:
+        return VecAndStore;
+    case Instruction::Shl:
+        return VecShlStore;
+    case Instruction::LShr:
+        return VecLShrStore;
+    case Instruction::AShr:
+        return VecAShrStore;
+    default:
+        abort();
+    }
+}
+
+static PseudoInst pseudoInstToStorePseudoInst(PseudoInst Inst) {
+    switch (Inst) {
+    case VecNot:
+        return VecNotStore;
+    case VecAddScalar:
+        return VecAddScalarStore;
+    case VecSubScalar:
+        return VecSubScalarStore;
+    case VecMulScalar:
+        return VecMulScalarStore;
+    case VecXorScalar:
+        return VecXorScalarStore;
+    case VecOrScalar:
+        return VecOrScalarStore;
+    case VecAndScalar:
+        return VecAndScalarStore;
+    case VecShlScalar:
+        return VecShlScalarStore;
+    case VecLShrScalar:
+        return VecLShrScalarStore;
+    case VecAShrScalar:
+        return VecAShrScalarStore;
+    case VecWideCondBitsel:
+        return VecWideCondBitselStore;
+    default:
+        abort();
+    }
+}
+
+static PseudoInst intrinsicToStorePseudoInst(unsigned IntrinsicID) {
+    switch (IntrinsicID) {
+    case Intrinsic::sadd_sat:
+        return VecSignedSatAddStore;
+    case Intrinsic::ssub_sat:
+        return VecSignedSatSubStore;
+    case Intrinsic::fshr:
+        return VecFunnelShrStore;
+    case Intrinsic::abs:
+        return VecAbsStore;
+    case Intrinsic::smax:
+        return VecSignedMaxStore;
+    case Intrinsic::umax:
+        return VecUnsignedMaxStore;
+    case Intrinsic::smin:
+        return VecSignedMinStore;
+    case Intrinsic::umin:
+        return VecUnsignedMinStore;
+    case Intrinsic::ctlz:
+        return VecCtlzStore;
+    case Intrinsic::cttz:
+        return VecCttzStore;
+    case Intrinsic::ctpop:
+        return VecCtpopStore;
+    default:
+        abort();
+    }
+}
+
+// For binary/unary ops on vectors where the result is stored to a
+// pointer
+//
+//   %3 = <NxM> %1 [op] <NxM> %2
+//   %4 = bitcast i8* %0 to <NxM>*
+//   store <NxM> %3, <NxM>* %4
+//
+// to
+//
+//   call @Vec[Op]Store.*(%0, %1, %2)
+//
+// This deals with the duality of pointers and vectors, and
+// simplifies the backend.  We previously kept a map on the
+// side to propagate "vector"-ness from %3 to %4 via the store,
+// no longer!
+static void convertVecStoreToPseudoInst(EraseInstVec &InstToErase, Module &M,
+                                        StoreInst *Store) {
+    Value *ValueOp = Store->getValueOperand();
+    Type *ValueTy = ValueOp->getType();
+    if (!ValueTy->isVectorTy()) {
+        return;
+    }
+
+    // Ensure store and binary op. are in the same basic
+    // block since the op. is moved to the store.
+    bool InSameBB =
+        cast<Instruction>(ValueOp)->getParent() == Store->getParent();
+    if (!InSameBB) {
+        return;
+    }
+
+    SmallVector<Value *, 3> Args;
+    Value *PtrOp = Store->getPointerOperand();
+    if (auto *BinOp = dyn_cast<BinaryOperator>(ValueOp)) {
+        Instruction *Inst = cast<Instruction>(ValueOp);
+        PseudoInst NewInst = instructionToStorePseudoInst(BinOp->getOpcode());
+        IRBuilder<> Builder(Store);
+        // Add one to account for extra store pointer
+        // argument of Vec*Store pseudo instructions.
+        const uint8_t SharedArgCount = pseudoInstArgCount(NewInst) - 1;
+        Args.push_back(PtrOp);
+        for (unsigned I = 0; I < SharedArgCount; ++I) {
+            Value *Op = Inst->getOperand(I);
+            Args.push_back(Op);
+        }
+        createPseudoInstCall(M, Builder, NewInst, Builder.getVoidTy(), Args);
+    } else if (auto *Call = dyn_cast<CallInst>(ValueOp)) {
+        Function *F = Call->getCalledFunction();
+        PseudoInst OldInst = getPseudoInstFromCall(Call);
+        if (OldInst != InvalidPseudoInst) {
+            // Map scalar vector pseudo instructions to
+            // store variants
+            PseudoInst NewInst = pseudoInstToStorePseudoInst(OldInst);
+            IRBuilder<> Builder(Store);
+            Args.push_back(PtrOp);
+            for (Value *Op : Call->args()) {
+                Args.push_back(Op);
+            }
+            createPseudoInstCall(M, Builder, NewInst, Builder.getVoidTy(),
+                                 Args);
+        } else if (F->isIntrinsic()) {
+            Instruction *Inst = cast<Instruction>(ValueOp);
+            PseudoInst NewInst =
+                intrinsicToStorePseudoInst(F->getIntrinsicID());
+            // Add one to account for extra store pointer
+            // argument of Vec*Store pseudo instructions.
+            const uint8_t SharedArgCount = pseudoInstArgCount(NewInst) - 1;
+            IRBuilder<> Builder(Store);
+            Args.push_back(PtrOp);
+            for (unsigned I = 0; I < SharedArgCount; ++I) {
+                Args.push_back(Inst->getOperand(I));
+            }
+            createPseudoInstCall(M, Builder, NewInst, Builder.getVoidTy(),
+                                 Args);
+        }
+    } else if (auto *Load = dyn_cast<LoadInst>(ValueOp)) {
+        auto *VecTy = cast<VectorType>(Load->getType());
+        auto *IntTy = cast<IntegerType>(VecTy->getElementType());
+        uint32_t LlvmSize = IntTy->getBitWidth();
+        uint32_t VectorElements = compat::getVectorElementCount(VecTy);
+        IRBuilder<> Builder(Store);
+        auto *Size = Builder.getInt64(LlvmSize * VectorElements);
+        Builder.CreateMemCpy(Store->getPointerOperand(),
+                             Store->getPointerAlignment(M.getDataLayout()),
+                             Load->getPointerOperand(),
+                             Load->getPointerAlignment(M.getDataLayout()),
+                             Size);
+        // Remove load if possible, won't get cleaned up by DCE
+        if (Load->hasOneUse()) {
+            InstToErase.push_back(cast<Instruction>(Load));
+        }
+    } else {
+        Instruction *Inst = cast<Instruction>(ValueOp);
+
+        PseudoInst NewInst = instructionToStorePseudoInst(Inst->getOpcode());
+        // Add one to account for extra store pointer
+        // argument of Vec*Store pseudo instructions.
+        const uint8_t SharedArgCount = pseudoInstArgCount(NewInst) - 2;
+
+        assert(SharedArgCount > 0 and
+               SharedArgCount <= (uint8_t)Inst->getNumOperands());
+        IRBuilder<> Builder(Store);
+        SmallVector<Value *, 8> Args;
+        auto SizeTy = Type::getInt8Ty(M.getContext());
+        Args.push_back(ConstantInt::get(
+            SizeTy,
+            cast<VectorType>(ValueTy)->getElementType()->getIntegerBitWidth()));
+        Args.push_back(PtrOp);
+        for (uint8_t I = 0; I < SharedArgCount; ++I) {
+            Args.push_back(Inst->getOperand(I));
+        }
+        createPseudoInstCall(M, Builder, NewInst, Builder.getVoidTy(), Args);
+    }
+
+    // Remove store instruction, this ensures DCE
+    // can cleanup the rest, we also remove ValueOp
+    // here since it's a call and won't get cleaned
+    // by DCE.
+    if (!isa<LoadInst>(ValueOp)) {
+        InstToErase.push_back(cast<Instruction>(ValueOp));
+    }
+    InstToErase.push_back(Store);
+}
diff --git a/subprojects/helper-to-tcg/src/PrepareForTcgPass/CanonicalizeIR.hpp b/subprojects/helper-to-tcg/src/PrepareForTcgPass/CanonicalizeIR.hpp
new file mode 100644
index 0000000000..089cd75a7b
--- /dev/null
+++ b/subprojects/helper-to-tcg/src/PrepareForTcgPass/CanonicalizeIR.hpp
@@ -0,0 +1,24 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#pragma once
+
+#include <llvm/IR/PassManager.h>
+
+struct VectorLayout;
+
+void canonicalizeIR(llvm::Module &M, llvm::ModuleAnalysisManager &MAM, const VectorLayout &VL);
diff --git a/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp b/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp
index 10129fa245..71749e2f6f 100644
--- a/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp
+++ b/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp
@@ -16,6 +16,7 @@
 //
 
 #include "PrepareForTcgPass.hpp"
+#include "CanonicalizeIR.hpp"
 #include "CmdLineOptions.hpp"
 #include "TransformGEPs.hpp"
 
@@ -140,5 +141,6 @@ PreservedAnalyses PrepareForTcgPass::run(Module &M,
     for (Function &F : M) {
         transformGEPs(M, F, ResultTcgGlobalMap, TypeIndexMap, DebugInfo);
     }
+    canonicalizeIR(M, MAM, VL);
     return PreservedAnalyses::none();
 }
-- 
2.52.0

[PATCH v2 28/50] helper-to-tcg: PrepareForTcgPass, identity map trivial expressions

[Anton Johansson via qemu development]

Transformation of the IR, identity mapping expressions which
would amount to nothing more than a move when emitted as TCG, but is
required in LLVM IR to not break the IR.

Trivial expressions are mapped to a `@IdentityMap` pseudo instruction
allowing them to be dealt with in a uniform manner down the line.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 subprojects/helper-to-tcg/meson.build         |  1 +
 .../src/PrepareForTcgPass/IdentityMap.cpp     | 91 +++++++++++++++++++
 .../src/PrepareForTcgPass/IdentityMap.hpp     | 39 ++++++++
 .../PrepareForTcgPass/PrepareForTcgPass.cpp   |  4 +
 4 files changed, 135 insertions(+)
 create mode 100644 subprojects/helper-to-tcg/src/PrepareForTcgPass/IdentityMap.cpp
 create mode 100644 subprojects/helper-to-tcg/src/PrepareForTcgPass/IdentityMap.hpp

diff --git a/subprojects/helper-to-tcg/meson.build b/subprojects/helper-to-tcg/meson.build
index 79ce0c6dd8..c88c759ba6 100644
--- a/subprojects/helper-to-tcg/meson.build
+++ b/subprojects/helper-to-tcg/meson.build
@@ -47,6 +47,7 @@ sources = [
     'src/PrepareForTcgPass/PrepareForTcgPass.cpp',
     'src/PrepareForTcgPass/TransformGEPs.cpp',
     'src/PrepareForTcgPass/CanonicalizeIR.cpp',
+    'src/PrepareForTcgPass/IdentityMap.cpp',
 ]
 
 clang = bindir / 'clang'
diff --git a/subprojects/helper-to-tcg/src/PrepareForTcgPass/IdentityMap.cpp b/subprojects/helper-to-tcg/src/PrepareForTcgPass/IdentityMap.cpp
new file mode 100644
index 0000000000..b9f4c8330f
--- /dev/null
+++ b/subprojects/helper-to-tcg/src/PrepareForTcgPass/IdentityMap.cpp
@@ -0,0 +1,91 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#include "IdentityMap.hpp"
+#include "PseudoInst.hpp"
+#include "TcgType.hpp"
+
+#include <llvm/ADT/SmallVector.h>
+#include <llvm/IR/IRBuilder.h>
+#include <llvm/IR/InstIterator.h>
+#include <llvm/IR/Instruction.h>
+#include <llvm/IR/Value.h>
+
+using namespace llvm;
+
+void identityMap(Module &M, Function &F) {
+    SmallVector<Instruction *, 8> InstToErase;
+
+    for (Instruction &I : instructions(F)) {
+        auto *ZExt = dyn_cast<ZExtInst>(&I);
+        if (ZExt) {
+            auto *SrcIntTy =
+                dyn_cast<IntegerType>(ZExt->getOperand(0)->getType());
+            auto *DstIntTy = dyn_cast<IntegerType>(ZExt->getType());
+            if (!SrcIntTy or !DstIntTy) {
+                continue;
+            }
+
+            auto SrcSize = ValueSize::fromLlvmType(SrcIntTy);
+            auto DstSize = ValueSize::fromLlvmType(DstIntTy);
+            if (!SrcSize or !DstSize) {
+                continue;
+            }
+
+            // TODO: Hack again to get bit width from icmp arguments, should
+            // widen in canonicalization phase.
+            if (SrcSize->LlvmBitWidth == 1) {
+                auto *ICmp = dyn_cast<ICmpInst>(ZExt->getOperand(0));
+                if (ICmp) {
+                    auto *ICmpOp = ICmp->getOperand(0);
+                    auto OpSize = ValueSize::fromLlvmType(
+                        cast<IntegerType>(ICmpOp->getType()));
+                    if (!OpSize) {
+                        continue;
+                    }
+                    SrcSize = *OpSize;
+                }
+            }
+
+            // Only identity map when TCG sizes match.
+            if (SrcSize->TcgBitWidth != DstSize->TcgBitWidth) {
+                continue;
+            }
+
+            IRBuilder<> Builder(&I);
+            ZExt->replaceAllUsesWith(createPseudoInstCall(
+                M, Builder, IdentityMap, DstIntTy, {ZExt->getOperand(0)}));
+            InstToErase.push_back(&I);
+        } else if (auto *Load = dyn_cast<LoadInst>(&I);
+                   Load and Load->getType()->isVectorTy()) {
+            Value *Ptr = Load->getPointerOperand();
+            IRBuilder<> Builder(&I);
+            Load->replaceAllUsesWith(createPseudoInstCall(
+                M, Builder, IdentityMap, Load->getType(), {Ptr}));
+            InstToErase.push_back(&I);
+        } else if (isa<FreezeInst>(&I)) {
+            IRBuilder<> Builder(&I);
+            I.replaceAllUsesWith(createPseudoInstCall(
+                M, Builder, IdentityMap, I.getType(), {I.getOperand(0)}));
+            InstToErase.push_back(&I);
+        }
+    }
+
+    for (Instruction *I : InstToErase) {
+        I->eraseFromParent();
+    }
+}
diff --git a/subprojects/helper-to-tcg/src/PrepareForTcgPass/IdentityMap.hpp b/subprojects/helper-to-tcg/src/PrepareForTcgPass/IdentityMap.hpp
new file mode 100644
index 0000000000..25de0a2c71
--- /dev/null
+++ b/subprojects/helper-to-tcg/src/PrepareForTcgPass/IdentityMap.hpp
@@ -0,0 +1,39 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#pragma once
+
+#include <llvm/IR/Function.h>
+#include <llvm/IR/Module.h>
+
+//
+// Transformation of the IR, taking what would become trivial unary operations
+// and maps them to a single @IdentityMap pseudo instruction.
+//
+// To motivate further, in order to produce nice IR on the other end, generally
+// the operands of these trivial expressions needs to be forwarded and treated
+// as the destination value (identity mapped).  However, directly removing these
+// instructions will result in broken LLVM IR (consider zext i8, i32 where both
+// the source and destination would map to TCGv_i32).
+//
+// Moreover, handling these identity mapped values in an adhoc way quickly
+// becomes cumbersome and spreads throughout the codebase.  Therefore,
+// introducing @IdentityMap allows code further down the pipeline to ignore the
+// source of the identity map.
+//
+
+void identityMap(llvm::Module &M, llvm::Function &F);
diff --git a/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp b/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp
index 71749e2f6f..4185af7956 100644
--- a/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp
+++ b/subprojects/helper-to-tcg/src/PrepareForTcgPass/PrepareForTcgPass.cpp
@@ -18,6 +18,7 @@
 #include "PrepareForTcgPass.hpp"
 #include "CanonicalizeIR.hpp"
 #include "CmdLineOptions.hpp"
+#include "IdentityMap.hpp"
 #include "TransformGEPs.hpp"
 
 #include <llvm/ADT/SCCIterator.h>
@@ -142,5 +143,8 @@ PreservedAnalyses PrepareForTcgPass::run(Module &M,
         transformGEPs(M, F, ResultTcgGlobalMap, TypeIndexMap, DebugInfo);
     }
     canonicalizeIR(M, MAM, VL);
+    for (Function &F : M) {
+        identityMap(M, F);
+    }
     return PreservedAnalyses::none();
 }
-- 
2.52.0

[PATCH v2 29/50] helper-to-tcg: Introduce TcgV structure

[Anton Johansson via qemu development]

Adds a struct representing everything a LLVM value might map to in TCG,
this includes:

  * TCGv_i* (IrValue);
  * TCGv_ptr (IrPtr);
  * TCGLabel (IrLabel);
  * 123123ull (IrImmediate);
  * intptr_t (IrPtrToOffset).

Every `TcgV` value is assigned an increasing identifier, a kind
(list above), a size, and optinally a name along with some extra
flags.  Two values are considered equal if their identifiers are equal.
For instance, this can mean one identifier might describe different LLVM
sizes at different points in a function, which is very useful when mapping
semantics from LLVM IR to TCG.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 subprojects/helper-to-tcg/include/TcgType.hpp | 276 ++++++++++++++++++
 1 file changed, 276 insertions(+)
 create mode 100644 subprojects/helper-to-tcg/include/TcgType.hpp

diff --git a/subprojects/helper-to-tcg/include/TcgType.hpp b/subprojects/helper-to-tcg/include/TcgType.hpp
new file mode 100644
index 0000000000..8cc46a6b1f
--- /dev/null
+++ b/subprojects/helper-to-tcg/include/TcgType.hpp
@@ -0,0 +1,276 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#pragma once
+
+#include "Error.hpp"
+#include "LlvmCompat.hpp"
+
+#include <llvm/ADT/ArrayRef.h>
+#include <llvm/ADT/StringRef.h>
+#include <llvm/IR/DerivedTypes.h>
+#include <llvm/Support/raw_ostream.h>
+
+#include <assert.h>
+#include <stdint.h>
+#include <string>
+#include <utility> // std::pair
+
+// Data representing allowed vector and value sizes.
+
+// Sizes corresponding to a TCGv_i* register.
+enum AllowedTcgSize : uint8_t {
+    T32 = 32,
+    T64 = 64,
+    T128 = 128,
+};
+
+// Sizes corresponding of allowed LLVM values, operations smaller than
+// `AllowedTcgSize` act on these sizes.
+enum AllowedLlvmSize : uint8_t {
+    I1 = 1,
+    I8 = 8,
+    I16 = 16,
+    I32 = 32,
+    I64 = 64,
+};
+
+// Despite limiting sizes to an enum, when assigned from runtime values they may
+// still differ.
+inline bool verifyLlvmSize(AllowedLlvmSize S) {
+    switch (S) {
+    case I1:
+    case I8:
+    case I16:
+    case I32:
+    case I64:
+        return true;
+    default:
+        return false;
+    };
+}
+
+// Size of an integer value, combining a TCG and LLVM size.
+struct ValueSize {
+    AllowedTcgSize TcgBitWidth;
+    AllowedLlvmSize LlvmBitWidth;
+
+    static llvm::Expected<ValueSize> fromBitWidth(size_t BitWidth) {
+        ValueSize Ret;
+        if (BitWidth <= 32) {
+            Ret.TcgBitWidth = T32;
+        } else if (BitWidth <= 64) {
+            Ret.TcgBitWidth = T64;
+        } else {
+            Ret.TcgBitWidth = T128;
+        }
+        Ret.LlvmBitWidth = (AllowedLlvmSize)BitWidth;
+        // There's no runtime sanity checks that an enum is not just any integer
+        // value, so verify manually.
+        if (verifyLlvmSize(Ret.LlvmBitWidth)) {
+            return Ret;
+        } else {
+            return mkError("Invalid bit width");
+        }
+    }
+
+    static llvm::Expected<ValueSize> fromLlvmType(llvm::IntegerType *Ty) {
+        return fromBitWidth(Ty->getBitWidth());
+    }
+};
+
+struct VectorSize {
+    uint8_t ElementCount;
+    AllowedLlvmSize ElementBitWidth;
+
+    inline size_t bytes() const { return (ElementCount * ElementBitWidth) / 8; }
+
+    static llvm::Expected<VectorSize> fromLlvmType(llvm::VectorType *Ty) {
+        auto *IntTy = llvm::dyn_cast<llvm::IntegerType>(Ty->getElementType());
+        if (!IntTy) {
+            return mkError(
+                "Vectors of non-integer element type not supported!\n");
+        }
+        VectorSize Ret{
+            (uint8_t)compat::getVectorElementCount(Ty),
+            (AllowedLlvmSize)IntTy->getBitWidth(),
+        };
+        // There's no runtime sanity checks that an enum is not just any integer
+        // value, so verify manually.
+        if (verifyLlvmSize(Ret.ElementBitWidth)) {
+            return Ret;
+        } else {
+            return mkError("Invalid bit width");
+        }
+    }
+};
+
+union TcgSize {
+    VectorSize Vec;
+    ValueSize Val;
+};
+
+// clang-format off
+enum TcgKind : uint8_t {
+    IrInvalid     = 0,
+    IrValue       = 1,  // TCG register value (TCGv_i*)
+    IrImmediate   = 2,  // Immediate argument to TCG operation ([u]int*_t)
+    IrPtr         = 4,  // TCG host pointer (TCGv_ptr)
+    IrPtrToOffset = 8,  // Target "gvec" vector (intptr_t)
+    IrLabel       = 16, // TCG label (TCGv_label)
+};
+// clang-format on
+
+// Describes a single value to be output in TCG, discriminated by `TcgKind`.
+//
+// This data is designed to be copied around and each value is identified via a
+// unique `Id`, as a result the "same" `TcgV` can take on e.g. different sizes
+// at different points in function, which allows one TCG variable to represent
+// multiple smaller "logical" LLVM sizes.
+//
+// TODO:
+//
+//   * There is overlap between `Kind == IrImmediate` and `(ConstantExpression
+//     and Kind == IrValue)` that is not obvious, the condition `!Name.empty()`
+//     is also rather similar to `ConstantExpression` in a lot of cases.
+//
+//   * `Name` should not be string, this is resulting in redundant copies and is
+//     also completely unnecessary given our usage pattern.  `Name` is never
+//     modified, and all strings are created by the same few passes.  A better
+//     solution would be to use a `StringSaver/BumpAllocator` in `CEmitter` and
+//     use `StringRef`s everywhere else.
+//
+struct TcgV {
+    uint16_t Id = 0;
+    TcgKind Kind = IrInvalid;
+    TcgSize Size;
+    bool ConstantExpression = false;
+    std::string Name = "";
+
+    // Static id used to identify `TcgV`s
+    inline static uint16_t IdCounter = 0;
+    static void resetId() { IdCounter = 0; }
+    static uint16_t getId() { return IdCounter++; }
+
+    // Helper functions for accessing size fields.
+    inline ValueSize intSize() const {
+        assert(Kind == IrImmediate or Kind == IrValue);
+        return Size.Val;
+    }
+
+    inline VectorSize vecSize() const {
+        assert(Kind == IrPtrToOffset);
+        return Size.Vec;
+    }
+
+    // Return as `int` to reduce casts when printing and to ease comparisons.
+    inline int tcgBitWidth() const { return intSize().TcgBitWidth; }
+    inline int llvmBitWidth() const { return intSize().LlvmBitWidth; }
+
+    static TcgV makeVector(VectorSize Size) {
+        return TcgV({}, {.Vec = Size}, IrPtrToOffset);
+    }
+
+    static TcgV makeImmediate(llvm::StringRef Name, ValueSize Size) {
+        return TcgV(Name.str(), {.Val = Size}, IrImmediate);
+    }
+
+    static TcgV makeTemp(ValueSize Size, TcgKind Kind) {
+        return TcgV({}, {.Val = Size}, Kind);
+    }
+
+    static TcgV makeConstantExpression(llvm::StringRef Expression,
+                                       ValueSize Size, TcgKind Kind) {
+        TcgV Tcg(Expression.str(), {.Val = Size}, Kind);
+        Tcg.ConstantExpression = true;
+        return Tcg;
+    }
+
+    static TcgV makeLabel() { return TcgV("", {.Val = {T32, I32}}, IrLabel); }
+
+    TcgV() = default;
+
+    TcgV(std::string Name, TcgSize Size, TcgKind Kind)
+        : Id(getId()), Kind(Kind), Size(Size), Name(Name) {}
+
+    // Equality between two values it determined only by the assigned id,
+    // consider a 16- to 8-bit truncation:
+    //
+    //   %1 = trunc i8, i16 %0,
+    //
+    // which after identity mapping becomes
+    //
+    //   %1 = call i8 @IdentityMap.i16.i8(i16 %0).
+    //
+    // Identity mapping will copy the `TcgV` assigned to `%0` to `%1`, but `%1`
+    // will retain its `LlvmSize` needed to emit correctly sized operations down
+    // the line.  Despite differing in `LlvmSize` both `%0` and `%1` will be
+    // emitted as the same TCG value with type `TCGv_i32`.
+    bool operator==(const TcgV &Other) const { return Other.Id == Id; }
+    bool operator!=(const TcgV &Other) const { return !operator==(Other); }
+
+    // Print out struct fields, useful for debugging.
+    inline void dump(llvm::raw_ostream &Out) const {
+        Out << "TcgV " << Id;
+        if (!Name.empty()) {
+            Out << " (" << Name << ")";
+        }
+        Out << ":\n";
+        Out << "  Kind: ";
+        switch (Kind) {
+        case IrInvalid:
+            Out << "IrInvalid\n";
+            break;
+        case IrValue:
+            Out << "IrValue\n";
+            break;
+        case IrImmediate:
+            Out << "IrImmediate\n";
+            break;
+        case IrPtr:
+            Out << "IrPtr\n";
+            break;
+        case IrPtrToOffset:
+            Out << "IrPtrToOffset\n";
+            break;
+        case IrLabel:
+            Out << "IrLabel\n";
+            break;
+        }
+        if (Kind == IrPtrToOffset) {
+            Out << "  VectorElementCount: " << (int)Size.Vec.ElementCount
+                << "\n";
+            Out << "  VectorElementBitWidth: " << (int)Size.Vec.ElementBitWidth
+                << "\n";
+        } else {
+            Out << "  TcgSize: " << (int)Size.Val.TcgBitWidth << "\n";
+            Out << "  LlvmSize: " << (int)Size.Val.LlvmBitWidth << "\n";
+        }
+        Out << "  ConstantExpression: " << ConstantExpression << "\n";
+    }
+};
+
+// Helper function for verifying that a set of `TcgV` are of acceptable types,
+// use with flags like
+//
+//   assertKinds({{Dst, IrPtr}, {Src, IrValue | IrImmediate}))
+//
+inline void assertKinds(llvm::ArrayRef<std::pair<const TcgV, uint8_t>> Pairs) {
+    for (auto &[Tcg, Flag] : Pairs) {
+        assert(Tcg.Kind & Flag);
+    }
+}
-- 
2.52.0

[PATCH v2 30/50] helper-to-tcg: Introduce TcgGenPass

[Anton Johansson via qemu development]

Adds a final backend pass over the LLVM module, taking
previously optimized and canonicalized LLVM IR and for
each function:

  1. Maps arguments, and propagates constant expressions;
  2. Allocates TCG values to non-constants;
  3. Emits the final TCG.

Points above will be implemented in following commits, this commits adds
boilerplate and some scaffolding for translating and emitting code.

TODO: This commit could do with some more splitting, TcgEmit.hpp and
ValueMapping.hpp and the functions therein are referenced but not added
in this commit, they come later.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 .../helper-to-tcg/include/CmdLineOptions.hpp  |  16 +
 .../helper-to-tcg/include/TcgGenPass.hpp      |  63 +++
 subprojects/helper-to-tcg/meson.build         |   1 +
 subprojects/helper-to-tcg/src/Pipeline.cpp    |  97 ++++-
 .../src/TcgGenPass/TcgGenPass.cpp             | 385 ++++++++++++++++++
 5 files changed, 561 insertions(+), 1 deletion(-)
 create mode 100644 subprojects/helper-to-tcg/include/TcgGenPass.hpp
 create mode 100644 subprojects/helper-to-tcg/src/TcgGenPass/TcgGenPass.cpp

diff --git a/subprojects/helper-to-tcg/include/CmdLineOptions.hpp b/subprojects/helper-to-tcg/include/CmdLineOptions.hpp
index 3a827f8ab7..1a4c0dd732 100644
--- a/subprojects/helper-to-tcg/include/CmdLineOptions.hpp
+++ b/subprojects/helper-to-tcg/include/CmdLineOptions.hpp
@@ -28,3 +28,19 @@ extern llvm::cl::opt<std::string> TcgGlobalMappingsName;
 extern llvm::cl::opt<std::string> UserPCRelBranchFunc;
 extern llvm::cl::opt<std::string> UserPCRelBranchFallthroughFunc;
 extern llvm::cl::opt<std::string> UserPCRelBranchConditionalFunc;
+// Options for TcgEmit
+extern llvm::cl::opt<std::string> MmuIndexFunction;
+extern llvm::cl::opt<std::string> TempVectorBlock;
+extern llvm::cl::opt<uint32_t> MaxVectorInstructions;
+extern llvm::cl::opt<uint32_t> MaxVectorTempBytes;
+// Options for MapTemporaries
+extern llvm::cl::opt<uint32_t> GuestPtrSize;
+// Options for TcgGenPass
+extern llvm::cl::opt<std::string> OutputSourceFile;
+extern llvm::cl::opt<std::string> OutputHeaderFile;
+extern llvm::cl::opt<std::string> OutputEnabledFile;
+extern llvm::cl::opt<bool> ErrorOnTranslationFailure;
+extern llvm::cl::opt<bool> StaticOutput;
+extern llvm::cl::opt<bool> AllowDeclCall;
+extern llvm::cl::opt<bool> ForwardContext;
+extern llvm::cl::opt<bool> EmitVectorPreamble;
diff --git a/subprojects/helper-to-tcg/include/TcgGenPass.hpp b/subprojects/helper-to-tcg/include/TcgGenPass.hpp
new file mode 100644
index 0000000000..8318b2c7cd
--- /dev/null
+++ b/subprojects/helper-to-tcg/include/TcgGenPass.hpp
@@ -0,0 +1,63 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#pragma once
+
+#include "DebugInfo.hpp"
+#include "FunctionAnnotation.hpp"
+#include "TcgGlobalMap.hpp"
+#include "VectorLayout.hpp"
+
+#include <llvm/IR/PassManager.h>
+
+//
+// TcgGenPass
+//
+// Backend pass responsible for emitting the final TCG code.
+//
+// This pass is further diveded into smaller passes that map arguments,
+// propagate constants, allocate registers, and emit finally TCG.
+//
+// Note, the incoming IR is broken and linearized to an array of basic blocks,
+// which acts to simplify the various backend passes.
+//
+
+class TcgGenPass : public llvm::PassInfoMixin<TcgGenPass> {
+    llvm::raw_ostream &OutSource;
+    llvm::raw_ostream &OutHeader;
+    llvm::raw_ostream &OutHelpers;
+    llvm::raw_ostream &OutEnabled;
+    llvm::StringRef HeaderPath;
+    const AnnotationMapTy &Annotations;
+    const DebugInfoMapTy &DebugInfo;
+    const TcgGlobalMap &TcgGlobals;
+    const VectorLayout &VL;
+
+  public:
+    TcgGenPass(llvm::raw_ostream &OutSource, llvm::raw_ostream &OutHeader,
+               llvm::raw_ostream &OutHelpers, llvm::raw_ostream &OutEnabled,
+               llvm::StringRef HeaderPath, const AnnotationMapTy &Annotations,
+               const DebugInfoMapTy &DebugInfo, const TcgGlobalMap &TcgGlobals,
+               const VectorLayout &VL)
+        : OutSource(OutSource), OutHeader(OutHeader), OutHelpers(OutHelpers),
+          OutEnabled(OutEnabled), HeaderPath(HeaderPath),
+          Annotations(Annotations), DebugInfo(DebugInfo),
+          TcgGlobals(TcgGlobals), VL(VL) {}
+
+    llvm::PreservedAnalyses run(llvm::Module &M,
+                                llvm::ModuleAnalysisManager &MAM);
+};
diff --git a/subprojects/helper-to-tcg/meson.build b/subprojects/helper-to-tcg/meson.build
index c88c759ba6..880561416a 100644
--- a/subprojects/helper-to-tcg/meson.build
+++ b/subprojects/helper-to-tcg/meson.build
@@ -48,6 +48,7 @@ sources = [
     'src/PrepareForTcgPass/TransformGEPs.cpp',
     'src/PrepareForTcgPass/CanonicalizeIR.cpp',
     'src/PrepareForTcgPass/IdentityMap.cpp',
+    'src/TcgGenPass/TcgGenPass.cpp',
 ]
 
 clang = bindir / 'clang'
diff --git a/subprojects/helper-to-tcg/src/Pipeline.cpp b/subprojects/helper-to-tcg/src/Pipeline.cpp
index 9b45664bdc..6555762b9d 100644
--- a/subprojects/helper-to-tcg/src/Pipeline.cpp
+++ b/subprojects/helper-to-tcg/src/Pipeline.cpp
@@ -19,6 +19,7 @@
 #include "LlvmCompat.hpp"
 #include "PrepareForOptPass.hpp"
 #include "PrepareForTcgPass.hpp"
+#include "TcgGenPass.hpp"
 #include "VectorLayout.hpp"
 
 #if LLVM_VERSION_MAJOR == 15
@@ -46,6 +47,7 @@
 #include <llvm/Support/MathExtras.h>
 #include <llvm/Support/SourceMgr.h>
 #include <llvm/Support/TargetSelect.h>
+#include <llvm/Support/ToolOutputFile.h>
 #include <llvm/Target/TargetMachine.h>
 #include <llvm/Transforms/Scalar/DCE.h>
 #include <llvm/Transforms/Scalar/SROA.h>
@@ -105,7 +107,7 @@ cl::opt<std::string> UserPCRelBranchFallthroughFunc(
              " from inside a conditional block, but that condition isn't taken."
              " and we instead need to fallthrough to the next instruction."),
     cl::init(""), cl::cat(Cat));
- 
+
 static llvm::cl::opt<bool> VecForceBigEndian(
     "vec-force-big-endian",
     cl::desc("Force emission of vector operations as if on a big-endian host."),
@@ -127,6 +129,76 @@ static llvm::cl::opt<LanePreference> VecLane0(
                          "Vector lane 0 is in the most-significant bytes")),
     cl::init(LeastSignificant), cl::cat(Cat));
 
+// Options for TcgEmit
+cl::opt<std::string> MmuIndexFunction(
+    "mmu-index-function",
+    cl::desc("Name of a (uint32_t tb_flag) -> int function returning the "
+             "mmu index from the tb_flags of the current translation block"),
+    cl::init("get_tb_mmu_index"), cl::cat(Cat));
+
+cl::opt<std::string>
+    TempVectorBlock("temp-vector-block",
+                    cl::desc("Name of uint8_t[...] field in CPUArchState used "
+                             "for allocating temporary gvec variables"),
+                    cl::init("tmp_vmem"), cl::cat(Cat));
+
+cl::opt<uint32_t> MaxVectorTempBytes(
+    "max-vector-temp-bytes",
+    cl::desc("Maximum bytes to be allocated for vector temporaries"),
+    cl::init(0), cl::cat(Cat));
+
+cl::opt<uint32_t> MaxVectorInstructions(
+    "max-vector-instructions",
+    cl::desc(
+        "Maximum number of vector instructions an emitted function may have"),
+    cl::init(0), cl::cat(Cat));
+
+// Options for MapTemporaries
+cl::opt<uint32_t>
+    GuestPtrSize("guest-ptr-size",
+                 cl::desc("Pointer size of the guest architecture"),
+                 cl::init(32), cl::cat(Cat));
+
+// Options for TcgGenPass
+cl::opt<std::string> OutputSourceFile("output-source",
+                                      cl::desc("output .c file"),
+                                      cl::init("helper-to-tcg-emitted.c"),
+                                      cl::cat(Cat));
+
+cl::opt<std::string> OutputHeaderFile("output-header",
+                                      cl::desc("output .h file"),
+                                      cl::init("helper-to-tcg-emitted.h"),
+                                      cl::cat(Cat));
+
+cl::opt<std::string> OutputHelpersFile(
+    "output-helpers", cl::desc("output support helper.h file"),
+    cl::init("helper-to-tcg-support-helpers.h"), cl::cat(Cat));
+
+cl::opt<std::string>
+    OutputEnabledFile("output-enabled",
+                      cl::desc("output list of tranlated functions"),
+                      cl::init("helper-to-tcg-enabled"), cl::cat(Cat));
+
+cl::opt<bool>
+    ErrorOnTranslationFailure("error-on-translation-failure",
+                              cl::desc("Abort translation on first failure"),
+                              cl::init(false), cl::cat(Cat));
+
+cl::opt<bool> StaticOutput("static-output",
+                           cl::desc("Statically define output functions"),
+                           cl::init(false), cl::cat(Cat));
+
+cl::opt<bool>
+    AllowDeclCall("allow-decl-call",
+                  cl::desc("Forward calls to declared functions to output"),
+                  cl::init(false), cl::cat(Cat));
+
+cl::opt<bool> ForwardContext(
+    "forward-context",
+    cl::desc(
+        "Pass a DisasContext struct to all translated functions and calls"),
+    cl::init(false), cl::cat(Cat));
+
 // Define a TargetTransformInfo (TTI) subclass, this allows for overriding
 // common per-llvm-target information expected by other LLVM passes, such
 // as the width of the largest scalar/vector registers.  Needed for consistent
@@ -300,5 +372,28 @@ int main(int argc, char **argv) {
         MPM.addPass(createModuleToFunctionPassAdaptor(std::move(FPM)));
     }
 
+    //
+    // Finally we run a backend pass that converts from LLVM IR to TCG,
+    // and emits the final code.
+    //
+
+    std::error_code EC;
+    ToolOutputFile OutSource(OutputSourceFile, EC, compat::OpenFlags);
+    ToolOutputFile OutHeader(OutputHeaderFile, EC, compat::OpenFlags);
+    ToolOutputFile OutHelpers(OutputHelpersFile, EC, compat::OpenFlags);
+    ToolOutputFile OutEnabled(OutputEnabledFile, EC, compat::OpenFlags);
+    assert(!EC);
+
+    MPM.addPass(TcgGenPass(OutSource.os(), OutHeader.os(), OutHelpers.os(),
+                           OutEnabled.os(), OutputHeaderFile, Annotations,
+                           DebugInfo, TcgGlobals, VL));
+
+    MPM.run(*M.get(), MAM);
+
+    OutSource.keep();
+    OutHeader.keep();
+    OutHelpers.keep();
+    OutEnabled.keep();
+
     return 0;
 }
diff --git a/subprojects/helper-to-tcg/src/TcgGenPass/TcgGenPass.cpp b/subprojects/helper-to-tcg/src/TcgGenPass/TcgGenPass.cpp
new file mode 100644
index 0000000000..a831a75204
--- /dev/null
+++ b/subprojects/helper-to-tcg/src/TcgGenPass/TcgGenPass.cpp
@@ -0,0 +1,385 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#include "TcgGenPass.hpp"
+#include "CmdLineOptions.hpp"
+#include "DebugInfo.hpp"
+#include "Error.hpp"
+#include "LinearizeBlocks.hpp"
+#include "TcgEmit.hpp"
+#include "TcgType.hpp"
+#include "ValueMapping.hpp"
+
+#include <llvm/ADT/PostOrderIterator.h>
+#include <llvm/ADT/SmallBitVector.h>
+#include <llvm/ADT/StringRef.h>
+#include <llvm/ADT/StringSet.h>
+#include <llvm/Analysis/CallGraph.h>
+#include <llvm/IR/BasicBlock.h>
+#include <llvm/IR/Constants.h>
+#include <llvm/IR/DerivedTypes.h>
+#include <llvm/IR/Function.h>
+#include <llvm/IR/Module.h>
+#include <llvm/Support/Debug.h>
+#include <llvm/Support/FormatVariadic.h>
+#include <llvm/Support/raw_ostream.h>
+
+#define DEBUG_TYPE "tcg-gen-pass"
+
+using namespace llvm;
+
+struct TranslatedFunction {
+    StringRef Name;
+    std::string Decl;
+    std::string Code;
+    std::string DispatchCode;
+    bool IsHelper;
+    bool HasVectorTemporaries;
+    bool NeedVectorSizeChangeOps;
+};
+
+static Expected<TranslatedFunction>
+translateFunction(Module &M, FunctionAnalysisManager &FAM, Function *F,
+                  const TcgGlobalMap &TcgGlobals,
+                  const AnnotationMapTy &Annotations,
+                  const DebugInfoMapTy &DebugInfo,
+                  const SmallPtrSet<Function *, 16> HasTranslatedFunction) {
+    assert(!F->isDeclaration());
+
+    TranslatedFunction TF = {
+        .Name = F->getName(),
+    };
+
+    {
+        // Remove prefix for helper functions to get cleaner emitted names
+        TF.IsHelper = TF.Name.consume_front("helper_");
+    }
+
+    std::string Body;
+    raw_string_ostream Out(Body);
+    CEmitter CE;
+    TcgEmitter TE(Out, CE);
+    TempAllocationData TAD;
+
+    // Following block of function calls make up the majority of the backend,
+    // the rest is mostly string emission.
+    {
+        // Linearize blocks, putting into a vector and breaking the LLVM IR by
+        // some replacing branches with pseudo instructions.
+        const LinearBlocks Blocks = linearizeBlocks(M, FAM, *F);
+        LLVM_DEBUG({
+            dbgs() << "Translating " << F->getName() << "\n";
+            for (auto &BB : Blocks) {
+                dbgs() << *BB << "\n";
+            }
+        });
+
+        // Map arguments to `TcgV`, store results in `TempAllocationData`.
+        if (auto Err = mapArguments(*F, Annotations, DebugInfo, TAD); Err) {
+            return Err;
+        }
+
+        // Make a forward pass over the IR, and propagate constant expressions
+        // forward, assigning them to values in `TempAllocationData`.
+        if (auto Err = propagateConstantExpressions(CE, *F, Blocks, Annotations,
+                                                    DebugInfo, TcgGlobals, TAD);
+            Err) {
+            return Err;
+        }
+
+        // Make a backward pass over the IR and assign temporary `TcgV`s to
+        // non-constant expressions, attempt to reuse created temporaries.
+        // Mapped temporaries are stored in `TempAllocationData`.
+        if (auto Err = allocateTemporaries(*F, Blocks, Annotations, DebugInfo,
+                                           CE, TcgGlobals, TAD);
+            Err) {
+            return Err;
+        }
+
+        // At this point all `Value`s in the function have been mapped to either
+        // a temporary or a constant expression, with the exception of labels.
+        // Make a forward pass over the IR and emit the final TCG operations.
+        if (auto Err = mapTcgOperations(Blocks, TcgGlobals, Annotations,
+                                        HasTranslatedFunction, TAD, TE, CE);
+            Err) {
+            return Err;
+        }
+    }
+
+    Out.flush();
+
+    raw_string_ostream OutFunc(TF.Code);
+    raw_string_ostream HeaderWriter(TF.Decl);
+    raw_string_ostream DispatchWriter(TF.DispatchCode);
+    std::string DispatchCall;
+    raw_string_ostream DispatchCallWriter(DispatchCall);
+    bool IsVectorInst = false;
+
+    if (StaticOutput) {
+        HeaderWriter << "static ";
+    }
+    HeaderWriter << "void " << "emit_" << TF.Name << '(';
+    SmallVector<TcgV, 4> CArgs;
+
+    if (!F->getReturnType()->isVoidTy()) {
+        assert(TAD.hasReturnValue());
+        IsVectorInst = (TAD.ReturnValue.Kind == IrPtrToOffset);
+        CArgs.push_back(TAD.ReturnValue);
+    }
+
+    for (const Argument &Arg : F->args()) {
+        TcgV T = TAD.Map[&Arg];
+        IsVectorInst |= (T.Kind == IrPtrToOffset);
+        CArgs.push_back(T);
+    }
+
+    if (ForwardContext) {
+        HeaderWriter << "DisasContext *ctx";
+        if (!CArgs.empty()) {
+            HeaderWriter << ", ";
+        }
+    }
+
+    auto CArgIt = CArgs.begin();
+    if (CArgIt != CArgs.end()) {
+        HeaderWriter << TE.getType(*CArgIt) << ' ' << getName(*CArgIt);
+        ++CArgIt;
+    }
+    while (CArgIt != CArgs.end()) {
+        HeaderWriter << ", " << TE.getType(*CArgIt) << ' ' << getName(*CArgIt);
+        ++CArgIt;
+    }
+
+    if (!IsVectorInst) {
+        DispatchCallWriter << "emit_" << TF.Name << "(";
+        auto CArgIt = CArgs.begin();
+        if (CArgIt != CArgs.end()) {
+            DispatchWriter << "static inline void gen_helper_" << TF.Name
+                           << "(";
+            for (int i = 0; i < CArgs.size(); ++i) {
+                if (i > 0) {
+                    DispatchWriter << ", ";
+                }
+                DispatchWriter << TE.getType(TE.materialize(CArgs[i])) << " "
+                               << getName(CArgs[i]);
+            }
+            DispatchWriter << ")\n{\n";
+
+            DispatchWriter << "emit_" << TF.Name << "(";
+            for (int i = 0; i < CArgs.size(); ++i) {
+                if (i > 0) {
+                    DispatchWriter << ", ";
+                }
+                if (CArgs[i].Kind == IrImmediate) {
+                    DispatchWriter << "tcgv_i" << CArgs[i].tcgBitWidth()
+                                   << "_temp(" << getName(CArgs[i]) << ")->val";
+                } else {
+                    DispatchWriter << getName(CArgs[i]);
+                }
+            }
+            DispatchWriter << ");\n}\n";
+        }
+    }
+
+    // Copy over function declaration from header to source file
+    HeaderWriter << ')';
+
+    OutFunc << "// " << *F->getReturnType() << ' ' << F->getName() << '\n';
+    OutFunc << HeaderWriter.str();
+    OutFunc << " {\n";
+    OutFunc << Body;
+    OutFunc << "}\n";
+
+    HeaderWriter << ';';
+
+    if (MaxVectorTempBytes > 0 and
+        TE.allocatedVectorMemory() > MaxVectorTempBytes) {
+        return mkError(formatv("Uses too much vector memory: {0} > {1}",
+                               TE.allocatedVectorMemory(), MaxVectorTempBytes)
+                           .str());
+    }
+
+    if (MaxVectorInstructions > 0 and
+        TE.numVectorInstructions() > MaxVectorInstructions) {
+        return mkError(formatv("Uses too many vector instructions: {0} > {1}",
+                               TE.numVectorInstructions(),
+                               MaxVectorInstructions)
+                           .str());
+    }
+
+    TF.HasVectorTemporaries = (TE.allocatedVectorMemory() > 0);
+    TF.NeedVectorSizeChangeOps = TE.NeedVectorSizeChangeOps;
+
+    HeaderWriter.flush();
+    DispatchWriter.flush();
+    DispatchCallWriter.flush();
+
+    return TF;
+}
+
+PreservedAnalyses TcgGenPass::run(Module &M, ModuleAnalysisManager &MAM) {
+    auto &CG = MAM.getResult<CallGraphAnalysis>(M);
+    auto &FAM =
+        MAM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
+
+    // Vector of translation results, order matters, lower indices are emitted
+    // higher up in the output file, and may be used by the higher indices.
+    SmallVector<TranslatedFunction, 16> TranslatedFunctions;
+    // Two sets used for quickly looking up whether or not a function has
+    // already been translated, or the translation failed.
+    SmallPtrSet<Function *, 16> FailedToTranslateFunction;
+    SmallPtrSet<Function *, 16> HasTranslatedFunction;
+    bool NeedVectorMem = false;
+    bool NeedVectorSizeChangeOps = false;
+    for (Function &F : M) {
+        if (F.isDeclaration()) {
+            continue;
+        }
+
+        // Depth first traversal of call graph.  Needed to ensure called
+        // functions are translated before the current function.
+        CallGraphNode *Node = CG[&F];
+        for (auto *N : make_range(po_begin(Node), po_end(Node))) {
+            Function *F = N->getFunction();
+
+            // If F in the call graph has already been translated and failed,
+            // abort translation of the current function.
+            if (FailedToTranslateFunction.contains(F)) {
+                break;
+            }
+
+            // Skip translation of invalid functions or functions that have
+            // already been translated.
+            if (!F or F->isDeclaration() or HasTranslatedFunction.contains(F)) {
+                continue;
+            }
+
+            // Reset variable IDs for the current function, since al TcgVs are
+            // function-local.
+            TcgV::resetId();
+
+            auto Translated =
+                translateFunction(M, FAM, F, TcgGlobals, Annotations, DebugInfo,
+                                  HasTranslatedFunction);
+            if (!Translated) {
+                FailedToTranslateFunction.insert(F);
+                LLVM_DEBUG({
+                    dbgs() << F->getName() << ": " << Translated.takeError()
+                           << "\n";
+                });
+                if (ErrorOnTranslationFailure) {
+                    return PreservedAnalyses::all();
+                }
+            } else {
+                TranslatedFunctions.push_back(*Translated);
+                HasTranslatedFunction.insert(F);
+                NeedVectorMem |= Translated->HasVectorTemporaries;
+                NeedVectorSizeChangeOps |= Translated->NeedVectorSizeChangeOps;
+            }
+        }
+    }
+
+    // Preamble
+    OutSource << "#include \"qemu/osdep.h\"\n";
+    OutSource << "#include \"qemu/log.h\"\n";
+    OutSource << "#include \"cpu.h\"\n";
+    OutSource << "#include \"translate.h\"\n";
+    OutSource << "#include \"tcg/tcg-op.h\"\n";
+    OutSource << "#include \"tcg/tcg-op-gvec.h\"\n";
+    OutSource << "#include \"tcg/tcg.h\"\n";
+    // OutSource << "#include \"exec/helper-gen.h\"\n";
+    if (ForwardContext) {
+        OutSource << "#include \"translate.h\"\n";
+    }
+    if (!StaticOutput) {
+        OutSource << "#include \"tcg/tcg-global-mappings.h\"\n";
+    }
+    if (MmuIndexFunction.size() > 0) {
+        OutSource << "#include \"exec/translation-block.h\"";
+    }
+    OutSource << '\n';
+
+    if (!StaticOutput) {
+        OutSource << "#include \""
+                  << HeaderPath.substr(HeaderPath.find_last_of('/') + 1)
+                  << "\"\n";
+        OutSource << '\n';
+
+        // Emit extern definitions for all global TCGv_* that are mapped
+        // to the CPUState.
+        for (size_t TypeIndex = 0; TypeIndex < TcgGlobals.size(); ++TypeIndex) {
+            for (auto &P : TcgGlobals[TypeIndex]) {
+                const TcgGlobal &Global = P.second;
+                auto Size = ValueSize::fromBitWidth(Global.Size);
+                assert(Size);
+                OutSource << "extern " << "TCGv_i" << (int)Size->TcgBitWidth
+                          << " " << Global.Code;
+                if (Global.NumElements > 1) {
+                    OutSource << "[" << Global.NumElements << "]";
+                }
+                OutSource << ";\n";
+            }
+        }
+    }
+
+    if (NeedVectorSizeChangeOps) {
+        emitVectorSizeChangeOps(OutSource, OutHelpers, VL);
+    }
+
+    if (NeedVectorMem) {
+        emitVectorMem(OutSource);
+    }
+
+    if (ForwardContext) {
+        OutHeader << "struct DisasContext;\n";
+        OutHeader << "typedef struct DisasContext DisasContext;\n";
+        OutSource << "#include \"translate.h\"\n";
+    }
+
+    // Emit translated functions
+    for (auto &TF : TranslatedFunctions) {
+        OutSource << TF.Code << '\n';
+        OutHeader << TF.Decl << '\n';
+        OutEnabled << TF.Name << '\n';
+    }
+
+    // Emit a dispatched to go from helper function address to our
+    // emitted code, if we succeeded.
+    OutHeader << "bool helper_to_tcg_dispatcher(void *func, TCGTemp *ret_temp, "
+                 "int nargs, TCGTemp **args);\n";
+
+    OutHeader << "#include \"tcg/helper-info.h\"\n";
+    OutHeader << "#include \"exec/helper-head.h.inc\"\n";
+    for (Function &F : M) {
+        StringRef Name = F.getName();
+        if (!Name.consume_front("helper_")) {
+            continue;
+        }
+        if (!HasTranslatedFunction.contains(&F)) {
+            SmallVector<Type *> ArgTys;
+            for (auto &A : F.args()) {
+                ArgTys.push_back(A.getType());
+            }
+            emitHelperGen(OutHeader, Name, F.getReturnType(), ArgTys);
+        }
+    }
+    for (TranslatedFunction &TF : TranslatedFunctions) {
+        OutHeader << TF.DispatchCode;
+    }
+
+    return PreservedAnalyses::all();
+}
-- 
2.52.0

[PATCH v2 31/50] helper-to-tcg: TcgGenPass, linearize basic blocks

[Anton Johansson via qemu development]

Passes over the IR that will follow all depend on a specific iteration
order over the IR that must match between passes, establish that
iteration order.  Also replace conditional branches with a pseudo
instruction closer in semantics to TCG, this breaks the IR verification,
but as this is the last transformation of the IR in this tool that's
acceptable.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 subprojects/helper-to-tcg/meson.build         |   1 +
 .../src/TcgGenPass/LinearizeBlocks.cpp        | 140 ++++++++++++++++++
 .../src/TcgGenPass/LinearizeBlocks.hpp        |  32 ++++
 3 files changed, 173 insertions(+)
 create mode 100644 subprojects/helper-to-tcg/src/TcgGenPass/LinearizeBlocks.cpp
 create mode 100644 subprojects/helper-to-tcg/src/TcgGenPass/LinearizeBlocks.hpp

diff --git a/subprojects/helper-to-tcg/meson.build b/subprojects/helper-to-tcg/meson.build
index 880561416a..58e5f67153 100644
--- a/subprojects/helper-to-tcg/meson.build
+++ b/subprojects/helper-to-tcg/meson.build
@@ -49,6 +49,7 @@ sources = [
     'src/PrepareForTcgPass/CanonicalizeIR.cpp',
     'src/PrepareForTcgPass/IdentityMap.cpp',
     'src/TcgGenPass/TcgGenPass.cpp',
+    'src/TcgGenPass/LinearizeBlocks.cpp',
 ]
 
 clang = bindir / 'clang'
diff --git a/subprojects/helper-to-tcg/src/TcgGenPass/LinearizeBlocks.cpp b/subprojects/helper-to-tcg/src/TcgGenPass/LinearizeBlocks.cpp
new file mode 100644
index 0000000000..28dff1c572
--- /dev/null
+++ b/subprojects/helper-to-tcg/src/TcgGenPass/LinearizeBlocks.cpp
@@ -0,0 +1,140 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#include "LinearizeBlocks.hpp"
+#include "PseudoInst.hpp"
+
+#include <llvm/ADT/PostOrderIterator.h>
+#include <llvm/ADT/SmallPtrSet.h>
+#include <llvm/ADT/SmallSet.h>
+#include <llvm/ADT/SmallVector.h>
+#include <llvm/Analysis/RegionInfo.h>
+#include <llvm/Analysis/VectorUtils.h>
+#include <llvm/IR/Constants.h>
+#include <llvm/IR/DerivedTypes.h>
+#include <llvm/IR/Dominators.h>
+#include <llvm/IR/Function.h>
+#include <llvm/IR/IRBuilder.h>
+#include <llvm/IR/InstIterator.h>
+#include <llvm/IR/InstrTypes.h>
+#include <llvm/IR/Instruction.h>
+#include <llvm/IR/Instructions.h>
+#include <llvm/IR/Intrinsics.h>
+#include <llvm/IR/Module.h>
+#include <llvm/IR/PassManager.h>
+#include <llvm/IR/PatternMatch.h>
+#include <llvm/Support/Casting.h>
+
+using namespace llvm;
+using namespace PatternMatch;
+
+// Needed to track and remove instructions not handled by a subsequent dead code
+// elimination, this applies to calls to pseudo instructions in particular.
+using EraseInstVec = SmallVector<Instruction *, 16>;
+
+static void convertICmpBrToPseudInst(LLVMContext &Context, RegionInfo &RI,
+                                     EraseInstVec &InstToErase, Module &M,
+                                     Instruction *I, BasicBlock *NextBB) {
+    auto *ICmp = dyn_cast<ICmpInst>(I);
+    if (!ICmp) {
+        return;
+    }
+
+    // Since we want to remove the icmp instruction we ensure that
+    // all uses are branch instructions that can be converted into
+    // @brcond.* calls.
+    for (User *U : ICmp->users()) {
+        if (!isa<BranchInst>(U)) {
+            return;
+        }
+    }
+
+    Value *Op0 = ICmp->getOperand(0);
+    Value *Op1 = ICmp->getOperand(1);
+    auto *CmpIntTy = dyn_cast<IntegerType>(Op0->getType());
+    if (!CmpIntTy) {
+        return;
+    }
+    for (User *U : ICmp->users()) {
+        auto *Br = cast<BranchInst>(U);
+
+        BasicBlock *True = Br->getSuccessor(0);
+        BasicBlock *False = Br->getSuccessor(1);
+
+        IRBuilder<> Builder(Br);
+
+        // TODO: This is strange, indeally the unreachable branch should have
+        // been optimized out, here we invert the conditional branch if we can
+        // fallthrough to the reachable branch.
+        bool TrueUnreachable =
+            True->getTerminator()->getOpcode() == Instruction::Unreachable and
+            False->getTerminator()->getOpcode() != Instruction::Unreachable;
+
+        // If the next basic block is either of our true/false
+        // branches, we can fallthrough instead of branching.
+        bool Fallthrough = (NextBB == True or NextBB == False);
+
+        // If the succeeding basic block is the true branch we
+        // invert the condition so we can fallthrough instead.
+        ICmpInst::Predicate Predicate;
+        if (NextBB == True or (TrueUnreachable and NextBB == False)) {
+            std::swap(True, False);
+            Predicate = ICmp->getInversePredicate();
+        } else {
+            Predicate = ICmp->getPredicate();
+        }
+
+        createPseudoInstCall(M, Builder, Brcond, Builder.getVoidTy(),
+                             {Builder.getInt8(Fallthrough),
+                              ConstantInt::get(CmpIntTy, Predicate), Op0, Op1,
+                              True, False});
+
+        InstToErase.push_back(Br);
+    }
+    InstToErase.push_back(ICmp);
+}
+
+LinearBlocks linearizeBlocks(Module &M, FunctionAnalysisManager &FAM,
+                             Function &F) {
+    assert(!F.isDeclaration());
+
+    LinearBlocks Blocks{};
+
+    LLVMContext &Context = F.getContext();
+    EraseInstVec InstToErase;
+    auto &RI = FAM.getResult<RegionInfoAnalysis>(F);
+
+    ReversePostOrderTraversal<Function *> RPOT(&F);
+    for (auto BBIt = RPOT.begin(); BBIt != RPOT.end(); ++BBIt) {
+        Blocks.push_back(*BBIt);
+    }
+
+    for (int i = 0; i < Blocks.size(); ++i) {
+        BasicBlock *BB = Blocks[i];
+        BasicBlock *NextBB = (i + 1 < Blocks.size()) ? Blocks[i + 1] : nullptr;
+        for (Instruction &I : *BB) {
+            convertICmpBrToPseudInst(Context, RI, InstToErase, M, &I, NextBB);
+        }
+    }
+
+    // Finally clean up instructions we need to remove manually
+    for (Instruction *I : InstToErase) {
+        I->eraseFromParent();
+    }
+
+    return Blocks;
+}
diff --git a/subprojects/helper-to-tcg/src/TcgGenPass/LinearizeBlocks.hpp b/subprojects/helper-to-tcg/src/TcgGenPass/LinearizeBlocks.hpp
new file mode 100644
index 0000000000..8d9a1afc54
--- /dev/null
+++ b/subprojects/helper-to-tcg/src/TcgGenPass/LinearizeBlocks.hpp
@@ -0,0 +1,32 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#pragma once
+
+#include <llvm/ADT/SmallVector.h>
+#include <llvm/IR/PassManager.h>
+
+namespace llvm {
+class BasicBlock;
+class Function;
+class Module;
+} // namespace llvm
+
+using LinearBlocks = llvm::SmallVector<llvm::BasicBlock *, 16>;
+LinearBlocks linearizeBlocks(llvm::Module &M,
+                             llvm::FunctionAnalysisManager &FAM,
+                             llvm::Function &F);
-- 
2.52.0

[PATCH v2 32/50] helper-to-tcg: TcgGenPass, introduce Value <-> TcgV map

[Anton Johansson via qemu development]

Adds a `TempAllocationData` structure which will be the primary struct
holding information about mappings between `Value`s and `TcgV`s.

Any identified return value is also stored along with flags for easier
use by other passes.  All functions which populate `TempAllocationData`
will be declared in this header.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 .../src/TcgGenPass/ValueMapping.hpp           | 110 ++++++++++++++++++
 1 file changed, 110 insertions(+)
 create mode 100644 subprojects/helper-to-tcg/src/TcgGenPass/ValueMapping.hpp

diff --git a/subprojects/helper-to-tcg/src/TcgGenPass/ValueMapping.hpp b/subprojects/helper-to-tcg/src/TcgGenPass/ValueMapping.hpp
new file mode 100644
index 0000000000..6aaf8acaf3
--- /dev/null
+++ b/subprojects/helper-to-tcg/src/TcgGenPass/ValueMapping.hpp
@@ -0,0 +1,110 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#pragma once
+
+#include "DebugInfo.hpp"
+#include "FunctionAnnotation.hpp"
+#include "LinearizeBlocks.hpp"
+#include "TcgEmit.hpp"
+#include "TcgGlobalMap.hpp"
+#include "TcgType.hpp"
+
+#include <llvm/ADT/SmallSet.h>
+#include <llvm/ADT/StringRef.h>
+#include <llvm/IR/BasicBlock.h>
+#include <llvm/IR/Instructions.h>
+#include <llvm/Support/Error.h>
+
+//
+// Value Mapping
+//
+// Data structures and functions needed for mapping various LLVM `Value`s to
+// `TcgV`s.
+//
+
+namespace llvm {
+class Function;
+}
+
+// Flags reprensting common special cases for function return values, used when
+// emitted TCG to produce better output.
+enum TempAllocationFlags {
+    SkipReturnMov = 1,
+    ReturnsImmediate = 2,
+    ReturnsValue = 4,
+
+    HasReturn = (ReturnsImmediate | ReturnsValue),
+};
+
+// Main data structure responsible for holding mappings between LLVM `Value`s
+// and `TcgV`s, populated by functions declared below.
+struct TempAllocationData {
+    // Mapping of LLVM Values to the corresponding TcgV
+    llvm::DenseMap<const llvm::Value *, TcgV> Map;
+
+    // Whether or not the final mov in an instruction can safely
+    // be ignored or not.
+    uint8_t flags = 0;
+    TcgV ReturnValue;
+
+    inline bool hasReturnValue() const { return flags & HasReturn; }
+
+    inline TcgV map(const llvm::Value *V, const TcgV &T) {
+        return Map.try_emplace(V, T).first->second;
+    }
+};
+
+inline const llvm::iterator_range<llvm::User::const_op_iterator>
+getOperands(const llvm::Instruction *const I) {
+    switch (I->getOpcode()) {
+    case Instruction::GetElementPtr:
+        return llvm::cast<llvm::GetElementPtrInst>(I)->operands();
+    case Instruction::Call:
+        return llvm::cast<llvm::CallInst>(I)->args();
+    default:
+        return I->operands();
+    }
+}
+
+// Defined in MapArguments.cpp
+llvm::Error mapArguments(const llvm::Function &F,
+                         const AnnotationMapTy &AnnotationMap,
+                         const DebugInfoMapTy &DebugInfo,
+                         TempAllocationData &TAD);
+
+// Defined in MapConstantExpressions.cpp
+llvm::Error propagateConstantExpressions(CEmitter &C, const llvm::Function &F,
+                                         const LinearBlocks &Blocks,
+                                         const AnnotationMapTy &AnnotationMap,
+                                         const DebugInfoMapTy &DebugInfo,
+                                         const TcgGlobalMap &TcgGlobals,
+                                         TempAllocationData &TAD);
+// Defined in MapTemporaries.cpp
+llvm::Error allocateTemporaries(const llvm::Function &F,
+                                const LinearBlocks &Blocks,
+                                const AnnotationMapTy &AnnotationMap,
+                                const DebugInfoMapTy &DebugInfo, CEmitter &C,
+                                const TcgGlobalMap &TcgGlobal,
+                                TempAllocationData &TAD);
+
+// Defined MapTcgOperations.cpp
+llvm::Error mapTcgOperations(
+    const LinearBlocks &Blocks, const TcgGlobalMap &TcgGlobals,
+    const AnnotationMapTy &AnnotationMap,
+    const llvm::SmallPtrSet<llvm::Function *, 16> &HasTranslatedFunction,
+    const TempAllocationData &TAD, TcgEmitter &Tcg, CEmitter &C);
-- 
2.52.0

[PATCH v2 33/50] helper-to-tcg: TcgGenPass, add structs for string emission

[Anton Johansson via qemu development]

Adds two new structures `TcgEmitter` and `CEmitter` with the purpose of
producing valid TCG and C expressions in string form, these will be used
in following commits.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 subprojects/helper-to-tcg/meson.build         |    1 +
 .../helper-to-tcg/src/TcgGenPass/TcgEmit.cpp  | 1203 +++++++++++++++++
 .../helper-to-tcg/src/TcgGenPass/TcgEmit.hpp  |  315 +++++
 3 files changed, 1519 insertions(+)
 create mode 100644 subprojects/helper-to-tcg/src/TcgGenPass/TcgEmit.cpp
 create mode 100644 subprojects/helper-to-tcg/src/TcgGenPass/TcgEmit.hpp

diff --git a/subprojects/helper-to-tcg/meson.build b/subprojects/helper-to-tcg/meson.build
index 58e5f67153..c03a4d5df6 100644
--- a/subprojects/helper-to-tcg/meson.build
+++ b/subprojects/helper-to-tcg/meson.build
@@ -50,6 +50,7 @@ sources = [
     'src/PrepareForTcgPass/IdentityMap.cpp',
     'src/TcgGenPass/TcgGenPass.cpp',
     'src/TcgGenPass/LinearizeBlocks.cpp',
+    'src/TcgGenPass/TcgEmit.cpp',
 ]
 
 clang = bindir / 'clang'
diff --git a/subprojects/helper-to-tcg/src/TcgGenPass/TcgEmit.cpp b/subprojects/helper-to-tcg/src/TcgGenPass/TcgEmit.cpp
new file mode 100644
index 0000000000..8a39fd2589
--- /dev/null
+++ b/subprojects/helper-to-tcg/src/TcgGenPass/TcgEmit.cpp
@@ -0,0 +1,1203 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#include "TcgEmit.hpp"
+#include "CmdLineOptions.hpp"
+#include "TcgType.hpp"
+#include "VectorLayout.hpp"
+
+#include <llvm/ADT/SmallString.h>
+
+using namespace llvm;
+
+// Function returning a TcgV representing the current MMU index.
+// Needed for memory operations.
+static const TcgV mmuindex() {
+    if (ForwardContext) {
+        return TcgV::makeImmediate("ctx->mem_idx", {T32, I32});
+    } else {
+        return TcgV::makeImmediate(
+            MmuIndexFunction + "(tcg_ctx->gen_tb->flags)", {T32, I32});
+    }
+}
+
+//
+// TcgEmitter private functions
+//
+
+void TcgEmitter::genVecUnaryCall(StringRef Name, int ElementSize,
+                                 const TcgV &Dst, const TcgV &Src,
+                                 const TcgV &Size) {
+    Out << Name << "(MO_" << ElementSize << ", " << getName(Dst) << ", "
+        << getName(Src) << ", " << getName(Size) << ", " << getName(Size)
+        << ");\n";
+
+    // Track number of vector operations
+    ++NumVectorInstructions;
+}
+
+void TcgEmitter::genVecBinaryCall(StringRef Name, int ElementSize,
+                                  const TcgV &Dst, const TcgV &Src0,
+                                  const TcgV &Src1, size_t Size) {
+    Out << Name << "(MO_" << ElementSize << ", " << getName(Dst) << ", "
+        << getName(Src0) << ", " << getName(Src1) << Size << ", " << Size
+        << ");\n";
+}
+
+void TcgEmitter::genVecCall(StringRef Name, llvm::ArrayRef<const TcgV> Args,
+                            const VectorSize OpSize, const VectorSize MaxSize) {
+    Out << "tcg_gen_gvec_" << Name << "(MO_" << (int)OpSize.ElementBitWidth
+        << ", ";
+    emitArgListTcg(Args.begin(), Args.end());
+    Out << ", " << OpSize.bytes() << ", " << MaxSize.bytes() << ");\n";
+    ++NumVectorInstructions;
+}
+
+const std::string TcgEmitter::getType(const TcgV &Value) {
+    switch (Value.Kind) {
+    case IrValue:
+        return Twine("TCGv_i").concat(Twine(Value.tcgBitWidth())).str();
+    case IrImmediate:
+        if (Value.tcgBitWidth() == 1) {
+            return "bool";
+        } else {
+            return C.intType(false, Value.llvmBitWidth());
+        }
+    case IrPtr:
+        return "TCGv_ptr";
+    case IrPtrToOffset:
+        return "intptr_t";
+    case IrLabel:
+        return "TCGLabel *";
+    default:
+        abort();
+    }
+}
+
+inline StringRef mapPredicate(const CmpInst::Predicate &Pred) {
+    switch (Pred) {
+    case CmpInst::ICMP_EQ:
+        return "TCG_COND_EQ";
+    case CmpInst::ICMP_NE:
+        return "TCG_COND_NE";
+    case CmpInst::ICMP_UGT:
+        return "TCG_COND_GTU";
+    case CmpInst::ICMP_UGE:
+        return "TCG_COND_GEU";
+    case CmpInst::ICMP_ULT:
+        return "TCG_COND_LTU";
+    case CmpInst::ICMP_ULE:
+        return "TCG_COND_LEU";
+    case CmpInst::ICMP_SGT:
+        return "TCG_COND_GT";
+    case CmpInst::ICMP_SGE:
+        return "TCG_COND_GE";
+    case CmpInst::ICMP_SLT:
+        return "TCG_COND_LT";
+    case CmpInst::ICMP_SLE:
+        return "TCG_COND_LE";
+    default:
+        abort();
+    }
+}
+
+static std::string mapBinOp(const Instruction::BinaryOps &Opcode,
+                            const TcgV &Src0, const TcgV &Src1) {
+    const bool IsImmediate =
+        (Src0.Kind == IrImmediate or Src1.Kind == IrImmediate);
+    // TODO:
+    const bool IsPtr = (Opcode == Instruction::Add and
+                        (Src0.Kind == IrPtr or Src1.Kind == IrPtr));
+    assert(IsImmediate or Src0.tcgBitWidth() == Src1.tcgBitWidth());
+    std::string Expr = "";
+    llvm::raw_string_ostream ExprStream(Expr);
+
+    // Check for valid boolean operations if operating on a boolean
+    if (Src0.llvmBitWidth() == 1) {
+        assert(Src1.llvmBitWidth() == 1);
+        switch (Opcode) {
+        case Instruction::And:
+        case Instruction::Or:
+        case Instruction::Xor:
+            break;
+        default:
+            abort();
+        }
+    }
+
+    switch (Opcode) {
+    case Instruction::Add:
+        ExprStream << "tcg_gen_add";
+        break;
+    case Instruction::Sub:
+        ExprStream << "tcg_gen_sub";
+        break;
+    case Instruction::And:
+        ExprStream << "tcg_gen_and";
+        break;
+    case Instruction::Or:
+        ExprStream << "tcg_gen_or";
+        break;
+    case Instruction::Xor:
+        ExprStream << "tcg_gen_xor";
+        break;
+    case Instruction::Mul:
+        ExprStream << "tcg_gen_mul";
+        break;
+    case Instruction::UDiv:
+        ExprStream << "tcg_gen_divu";
+        break;
+    case Instruction::SDiv:
+        ExprStream << "tcg_gen_div";
+        break;
+    case Instruction::AShr:
+        ExprStream << "tcg_gen_sar";
+        break;
+    case Instruction::LShr:
+        ExprStream << "tcg_gen_shr";
+        break;
+    case Instruction::Shl:
+        ExprStream << "tcg_gen_shl";
+        break;
+    default:
+        abort();
+    }
+
+    if (IsImmediate) {
+        ExprStream << "i";
+    }
+
+    if (IsPtr) {
+        ExprStream << "_ptr";
+    } else {
+        ExprStream << "_i" << (int)Src0.tcgBitWidth();
+    }
+
+    ExprStream.flush();
+
+    return Expr;
+}
+
+static std::string mapVecBinOp(const Instruction::BinaryOps &Opcode,
+                               const TcgV &Src0, const TcgV &Src1) {
+    const bool IsShift = Opcode == Instruction::Shl or
+                         Opcode == Instruction::LShr or
+                         Opcode == Instruction::AShr;
+
+    std::string Suffix;
+    switch (Src1.Kind) {
+    case IrPtrToOffset:
+        Suffix = (IsShift) ? "v" : "";
+        break;
+    case IrValue:
+        Suffix = "s";
+        break;
+    case IrImmediate:
+        Suffix = "i";
+        break;
+    default:
+        abort();
+    }
+
+    switch (Opcode) {
+    case Instruction::Add:
+        return "add" + Suffix;
+    case Instruction::Sub:
+        return "sub" + Suffix;
+    case Instruction::Mul:
+        return "mul" + Suffix;
+    case Instruction::And:
+        return "and" + Suffix;
+    case Instruction::Or:
+        return "or" + Suffix;
+    case Instruction::Xor:
+        return "xor" + Suffix;
+    case Instruction::Shl:
+        return "shl" + Suffix;
+    case Instruction::LShr:
+        return "shr" + Suffix;
+    case Instruction::AShr:
+        return "sar" + Suffix;
+    default:
+        abort();
+    }
+}
+
+void TcgEmitter::genSetLabel(const TcgV &L) {
+    assert(L.Kind == IrLabel);
+    Out << "gen_set_label(" << getName(L) << ");\n";
+}
+
+void TcgEmitter::defineNewTemp(const TcgV &Tcg) {
+    assert(!Tcg.ConstantExpression);
+    if (Tcg.Kind == IrPtrToOffset and !EmittedVectorMem) {
+        EmittedVectorMem = true;
+        Out << "VectorMem mem = {0};\n";
+    }
+    Out << getType(Tcg) << " " << getName(Tcg) << " = ";
+    switch (Tcg.Kind) {
+    case IrValue:
+        Out << "tcg_temp_new_i" << (int)Tcg.tcgBitWidth() << "();\n";
+        break;
+    case IrPtr:
+        Out << "tcg_temp_new_ptr();\n";
+        break;
+    case IrPtrToOffset:
+        AllocatedVectorMemory += Tcg.vecSize().bytes();
+        Out << "temp_new_gvec(&mem, " << Tcg.vecSize().bytes() << ");\n";
+        break;
+    case IrLabel:
+        Out << "gen_new_label();\n";
+        break;
+    default:
+        abort();
+    }
+}
+
+void TcgEmitter::genBr(const TcgV &L) {
+    assert(L.Kind == IrLabel);
+    Out << "tcg_gen_br(" << getName(L) << ");\n";
+}
+
+void TcgEmitter::genExts(const TcgV &Dst, const TcgV &Src) {
+    assertKinds({{Dst, IrValue}, {Src, IrValue}});
+    StackTwine<64> FuncStr = Twine("tcg_gen_ext") + Twine(Src.llvmBitWidth()) +
+                             "s_i" + Twine(Dst.tcgBitWidth());
+    emitCallTcg(FuncStr, {Dst, Src});
+}
+
+void TcgEmitter::genExtI32I64(const TcgV &Dst, const TcgV &Src) {
+    assertKinds({{Dst, IrValue}, {Src, IrValue}});
+    emitCallTcg("tcg_gen_ext_i32_i64", {Dst, Src});
+}
+
+void TcgEmitter::genExtrlI64I32(const TcgV &Dst, const TcgV &Src) {
+    assertKinds({{Dst, IrValue}, {Src, IrValue}});
+    emitCallTcg("tcg_gen_extrl_i64_i32", {Dst, Src});
+}
+
+void TcgEmitter::genExtuI32I64(const TcgV &Dst, const TcgV &Src) {
+    assertKinds({{Dst, IrValue}, {Src, IrValue}});
+    emitCallTcg("tcg_gen_extu_i32_i64", {Dst, Src});
+}
+
+void TcgEmitter::genExtrhI64I32(const TcgV &Dst, const TcgV &Src) {
+    assertKinds({{Dst, IrValue}, {Src, IrValue}});
+    emitCallTcg("tcg_gen_extrh_i64_i32", {Dst, Src});
+}
+
+void TcgEmitter::genExtract(bool Sign, const TcgV &Dst, const TcgV &Src,
+                            const TcgV &Offset, const TcgV &Length) {
+    assertKinds({{Dst, IrValue},
+                 {Src, IrValue | IrImmediate},
+                 {Offset, IrImmediate},
+                 {Length, IrImmediate}});
+    assert(Dst.tcgBitWidth() == Src.tcgBitWidth());
+    const char *SignStr = (Sign) ? "s" : "";
+    const TcgV MSrc = materialize(Src);
+    Out << "if (" << getName(Length) << " > 0) {\n";
+    Out << "tcg_gen_" << SignStr << "extract_i" << (int)Dst.tcgBitWidth()
+        << "(";
+    emitArgListTcg({Dst, MSrc, Offset, Length});
+    Out << ");\n";
+    Out << "}\n";
+}
+
+void TcgEmitter::genDeposit(const TcgV &Dst, const TcgV &Into,
+                            const TcgV &Offset, const TcgV &Length,
+                            const TcgV &From) {
+    assertKinds({{Dst, IrValue},
+                 {Into, IrValue | IrImmediate},
+                 {From, IrValue | IrImmediate},
+                 {Offset, IrImmediate},
+                 {Length, IrImmediate}});
+    assert(Dst.tcgBitWidth() == Into.tcgBitWidth());
+    assert(Dst.tcgBitWidth() == From.tcgBitWidth());
+    Out << "if (" << getName(Length) << " > 0) {\n";
+    Out << "tcg_gen_deposit_i" << (int)Dst.tcgBitWidth() << "(";
+    const TcgV MInto = materialize(Into);
+    const TcgV MFrom = materialize(From);
+    emitArgListTcg({Dst, MInto, MFrom, Offset, Length});
+    Out << ");\n";
+    Out << "}\n";
+}
+
+void TcgEmitter::genPtrToValue(const TcgV &Dst, const TcgV &Src) {
+    assertKinds({{Dst, IrValue}, {Src, IrPtr}});
+    if (Dst.tcgBitWidth() == 64) {
+        emitCallTcg("tcg_gen_ext_ptr_i64", {Dst, Src});
+    } else {
+        emitCallTcg("tcg_gen_trunc_ptr_i32", {Dst, Src});
+    }
+}
+
+void TcgEmitter::genValueToPtr(const TcgV &Dst, const TcgV &Src) {
+    assertKinds({{Dst, IrPtr}, {Src, IrValue}});
+    if (Src.tcgBitWidth() == 64) {
+        emitCallTcg("tcg_gen_trunc_i64_ptr", {Dst, Src});
+    } else {
+        emitCallTcg("tcg_gen_trunc_i32_ptr", {Dst, Src});
+    }
+}
+
+void TcgEmitter::genConcat(const TcgV &Dst, const TcgV &Src0,
+                           const TcgV &Src1) {
+    assertKinds({{Dst, IrValue}, {Src0, IrValue}, {Src1, IrValue}});
+    emitCallTcg("tcg_gen_concat_i32_i64", {Dst, Src0, Src1});
+}
+
+void TcgEmitter::genMov(const TcgV &Dst, const TcgV &Src) {
+    assertKinds({{Dst, IrValue}, {Src, IrValue | IrImmediate}});
+    assert(Dst.tcgBitWidth() == Src.tcgBitWidth());
+    const char *ImmStr = (Src.Kind == IrImmediate) ? "i" : "";
+    Out << "tcg_gen_mov" << ImmStr << "_i" << (int)Dst.tcgBitWidth() << "("
+        << getName(Dst) << ", " << getName(Src) << ");\n";
+}
+
+void TcgEmitter::genMovPtr(const TcgV &Dst, const TcgV &Src) {
+    assertKinds({{Dst, IrPtr}, {Src, IrPtr}});
+    Out << "tcg_gen_mov_ptr(" << getName(Dst) << ", " << getName(Src) << ");\n";
+}
+
+void TcgEmitter::genAddPtr(const TcgV &Dst, const TcgV &Ptr,
+                           const TcgV &Offset) {
+    assertKinds({{Dst, IrPtr}, {Ptr, IrPtr}});
+    switch (Offset.Kind) {
+    case IrPtrToOffset:
+    case IrImmediate: {
+        emitCallTcg("tcg_gen_addi_ptr", {Dst, Ptr, Offset});
+    } break;
+    case IrPtr: {
+        emitCallTcg("tcg_gen_add_ptr", {Dst, Ptr, Offset});
+    } break;
+    case IrValue: {
+        auto OffsetPtr = TcgV::makeTemp({}, IrPtr);
+        defineNewTemp(OffsetPtr);
+        genValueToPtr(OffsetPtr, Offset);
+        emitCallTcg("tcg_gen_add_ptr", {Dst, Ptr, OffsetPtr});
+    } break;
+    default:
+        abort();
+    }
+}
+
+void TcgEmitter::genBinOp(const TcgV &Dst, const Instruction::BinaryOps Opcode,
+                          const TcgV &Src0, const TcgV &Src1) {
+    auto OpStr = mapBinOp(Opcode, Src0, Src1);
+    emitCallTcg(OpStr, {Dst, Src0, Src1});
+}
+
+void TcgEmitter::genMovcond(const CmpInst::Predicate &Pred, const TcgV &Ret,
+                            const TcgV &C1, const TcgV &C2, const TcgV &V1,
+                            const TcgV &V2) {
+    const size_t OpWidth = Ret.tcgBitWidth();
+    assert(OpWidth == C1.tcgBitWidth());
+    assert(OpWidth == C2.tcgBitWidth());
+    assert(OpWidth == V1.tcgBitWidth());
+    assert(OpWidth == V2.tcgBitWidth());
+    const TcgV mC1 = materialize(C1);
+    const TcgV mC2 = materialize(C2);
+    const TcgV mV1 = materialize(V1);
+    const TcgV mV2 = materialize(V2);
+    Out << "tcg_gen_movcond_i" << OpWidth << '(' << mapPredicate(Pred) << ", ";
+    emitArgListTcg({Ret, mC1, mC2, mV1, mV2});
+    Out << ");\n";
+}
+
+void TcgEmitter::genSetcond(const CmpInst::Predicate &Pred, const TcgV &Dst,
+                            const TcgV &Src0, const TcgV &Src1) {
+    assertKinds(
+        {{Dst, IrValue}, {Src0, IrValue}, {Src1, IrValue | IrImmediate}});
+    const size_t OpWidth = Dst.tcgBitWidth();
+    assert(OpWidth == Src0.tcgBitWidth());
+    assert(OpWidth == Src1.tcgBitWidth());
+    const char *ImmStr = (Src1.Kind == IrImmediate) ? "i" : "";
+    Out << "tcg_gen_setcond" << ImmStr << "_i" << OpWidth;
+    Out << "(" << mapPredicate(Pred) << ", ";
+    emitArgListTcg({Dst, Src0, Src1});
+    Out << ");\n";
+}
+
+void TcgEmitter::genBrcond(const CmpInst::Predicate &Pred, const TcgV &Src0,
+                           const TcgV &Src1, const TcgV &Label) {
+    assertKinds({{Src0, IrValue | IrImmediate},
+                 {Src1, IrValue | IrImmediate},
+                 {Label, IrLabel}});
+    assert(Src0.tcgBitWidth() == Src1.tcgBitWidth());
+    const char *ImmStr = (Src1.Kind == IrImmediate) ? "i" : "";
+    Out << "tcg_gen_brcond" << ImmStr << "_i" << (int)Src0.tcgBitWidth();
+    Out << "(" << mapPredicate(Pred) << ", ";
+    emitArgListTcg({materialize(Src0), Src1, Label});
+    Out << ");\n";
+}
+
+std::string TcgEmitter::getMemOp(uint8_t Size, uint8_t Endianness,
+                                 uint8_t Sign) {
+    std::string MemOpStr{};
+    raw_string_ostream MemOpStream(MemOpStr);
+    MemOpStream << "MO_" << (int)8 * Size;
+    switch (Endianness) {
+    case 0:
+        break; // do nothing
+    case 1:
+        MemOpStream << " | MO_LE";
+        break;
+    case 2:
+        MemOpStream << " | MO_BE";
+        break;
+    default:
+        abort();
+    }
+    switch (Sign) {
+    case 0:
+        break;
+    case 1:
+        MemOpStream << " | MO_SIGN";
+        break;
+    default:
+        abort();
+    }
+    return MemOpStream.str();
+}
+
+void TcgEmitter::genGuestLoad(const TcgV &Dst, const TcgV &Ptr,
+                              StringRef MemOp) {
+    assertKinds({{Dst, IrValue}, {Ptr, IrValue | IrImmediate}});
+    Out << "tcg_gen_qemu_ld_i" << (int)Dst.tcgBitWidth() << "(";
+    emitArgListTcg({Dst, materialize(Ptr), mmuindex()});
+    Out << ", " << MemOp << ");\n";
+}
+
+void TcgEmitter::genGuestStore(const TcgV &Ptr, const TcgV &Src,
+                               StringRef MemOp) {
+    assertKinds({{Ptr, IrValue | IrImmediate}, {Src, IrValue | IrImmediate}});
+    Out << "tcg_gen_qemu_st_i" << (int)Src.tcgBitWidth() << "(";
+    emitArgListTcg({materialize(Src), materialize(Ptr), mmuindex()});
+    Out << ", " << MemOp << ");\n";
+}
+
+void TcgEmitter::genHostLoad(const TcgV &Dst, const TcgV &Ptr,
+                             const TcgV &Offset) {
+    assertKinds({{Dst, IrValue}, {Ptr, IrPtr}});
+    const size_t LlvmSize = Dst.llvmBitWidth();
+    const size_t TcgSize = Dst.tcgBitWidth();
+    if (LlvmSize < TcgSize) {
+        Out << "tcg_gen_ld" << LlvmSize << "u_i" << TcgSize;
+    } else {
+        Out << "tcg_gen_ld_i" << TcgSize;
+    }
+    Out << "(";
+    emitArgListTcg({Dst, Ptr, Offset});
+    Out << ");\n";
+}
+
+void TcgEmitter::genHostLoadFromVec(const TcgV &Dst, const TcgV &Offset) {
+    genHostLoad(Dst, getGlobalEnv(), Offset);
+}
+
+void TcgEmitter::genHostStore(const TcgV &Ptr, const TcgV &Src) {
+    assertKinds({{Ptr, IrPtr}, {Src, IrValue | IrImmediate}});
+    const size_t LlvmSize = Src.llvmBitWidth();
+    const size_t TcgSize = Src.tcgBitWidth();
+    if (LlvmSize < TcgSize) {
+        Out << "tcg_gen_st" << LlvmSize << "_i" << TcgSize;
+    } else {
+        Out << "tcg_gen_st_i" << TcgSize;
+    }
+    Out << "(";
+    emitArgListTcg({materialize(Src), Ptr});
+    Out << ", 0);\n";
+}
+void TcgEmitter::genFunnelShl(const TcgV &Dst, const TcgV &Src0,
+                              const TcgV &Src1, const TcgV &Shift) {
+    const size_t OpWidth = Dst.tcgBitWidth();
+    assert(OpWidth == Src0.tcgBitWidth());
+    assert(OpWidth == Src1.tcgBitWidth());
+    assert(OpWidth == Shift.tcgBitWidth());
+    if (OpWidth == 32) {
+        auto Temp = TcgV::makeTemp({T64, I64}, IrValue);
+        defineNewTemp(Temp);
+        genConcat(Temp, Src1, Src0);
+
+        if (Shift.Kind == IrImmediate) {
+            genBinOp(Temp, Instruction::Shl, Temp, Shift);
+        } else {
+            auto Ext = TcgV::makeTemp({T64, I64}, IrValue);
+            defineNewTemp(Ext);
+            genExtuI32I64(Ext, Shift);
+            genBinOp(Temp, Instruction::Shl, Temp, Ext);
+        }
+
+        genExtrhI64I32(Dst, Temp);
+    } else {
+        genCallHelper(
+            "helper_fshl_i64",
+            {Dst, materialize(Src0), materialize(Src1), materialize(Shift)});
+    }
+}
+
+void TcgEmitter::genBitreverse(const TcgV &Dst, const TcgV &Src) {
+    assertKinds({{Dst, IrValue}, {Src, IrValue}});
+    StackTwine<64> FuncName = Twine("helper_bitreverse") +
+                              Twine(Dst.tcgBitWidth()) + "_i" +
+                              Twine(Src.tcgBitWidth());
+    genCallHelper(FuncName, {Dst, Src});
+}
+
+void TcgEmitter::genCountLeadingZeros(const TcgV &Dst, const TcgV &Src) {
+    assertKinds({{Dst, IrValue}, {Src, IrValue}});
+    assert(Dst.tcgBitWidth() == Src.tcgBitWidth());
+    Out << "tcg_gen_clzi_i" << Dst.tcgBitWidth() << "(";
+    emitArgListTcg({Dst, Src});
+    Out << ", " << Src.tcgBitWidth() << ");\n";
+}
+
+void TcgEmitter::genCountTrailingZeros(const TcgV &Dst, const TcgV &Src) {
+    assertKinds({{Dst, IrValue}, {Src, IrValue}});
+    assert(Dst.tcgBitWidth() == Src.tcgBitWidth());
+    Out << "tcg_gen_ctzi_i" << Dst.tcgBitWidth() << "(";
+    emitArgListTcg({Dst, Src});
+    Out << ", " << Src.tcgBitWidth() << ");\n";
+}
+
+void TcgEmitter::genCountOnes(const TcgV &Dst, const TcgV &Src) {
+    assertKinds({{Dst, IrValue}, {Src, IrValue}});
+    assert(Dst.tcgBitWidth() == Src.tcgBitWidth());
+    Out << "tcg_gen_ctpop_i" << Dst.tcgBitWidth() << "(";
+    emitArgListTcg({Dst, Src});
+    Out << ");\n";
+}
+
+void TcgEmitter::genByteswap(const TcgV &Dst, const TcgV &Src) {
+    assertKinds({{Dst, IrValue}, {Src, IrValue}});
+    assert(Dst.tcgBitWidth() == Src.tcgBitWidth());
+    Out << "tcg_gen_bswap" << Dst.llvmBitWidth() << "_i" << Src.tcgBitWidth()
+        << "(";
+    emitArgListTcg({Dst, Src});
+    Out << ");\n";
+}
+
+void TcgEmitter::genUnsignedSatSub(const TcgV &Dst, const TcgV &Src0,
+                                   const TcgV &Src1) {
+    assertKinds({{Dst, IrValue}, {Src0, IrValue}, {Src1, IrValue}});
+    const size_t OpWidth = Dst.tcgBitWidth();
+    assert(OpWidth == Src0.tcgBitWidth());
+    assert(OpWidth == Src1.tcgBitWidth());
+    Out << "tcg_gen_ussub_i" << OpWidth << "(";
+    emitArgListTcg({Dst, Src0, Src1});
+    Out << ");\n";
+}
+
+inline TcgKind opKind(const TcgV &Dst, const TcgV &Src) {
+    return (Dst.Kind == IrPtr) ? Src.Kind : Dst.Kind;
+}
+
+void TcgEmitter::genVecBinOpStr(StringRef Op, const TcgV &Dst, const TcgV &Src0,
+                                const TcgV &Src1) {
+    assertKinds({{Dst, IrPtr | IrPtrToOffset},
+                 {Src0, IrPtrToOffset},
+                 {Src1, IrPtrToOffset | IrValue | IrImmediate}});
+    const VectorSize Size = Src0.vecSize();
+    Out << "tcg_gen_gvec_" << Op << "(MO_" << (int)Size.ElementBitWidth << ", ";
+    emitArgListTcg({Dst, Src0, Src1});
+    Out << ", " << Size.bytes() << ", " << Size.bytes() << ");\n";
+    // Track number of vector operations
+    ++NumVectorInstructions;
+}
+
+void TcgEmitter::genVecBinOp(const Instruction::BinaryOps Opcode,
+                             const TcgV &Dst, const TcgV &Src0,
+                             const TcgV &Src1) {
+    genVecBinOpStr(mapVecBinOp(Opcode, Src0, Src1), Dst, Src0, Src1);
+}
+
+void TcgEmitter::genVecSignedSatAdd(const TcgV &Dst, const TcgV &Src0,
+                                    const TcgV &Src1) {
+    genVecBinOpStr("ssadd", Dst, Src0, Src1);
+}
+
+void TcgEmitter::genVecSignedSatSub(const TcgV &Dst, const TcgV &Src0,
+                                    const TcgV &Src1) {
+    genVecBinOpStr("sssub", Dst, Src0, Src1);
+}
+
+void TcgEmitter::genVecUnsignedSatSub(const TcgV &Dst, const TcgV &Src0,
+                                      const TcgV &Src1) {
+    genVecBinOpStr("ussub", Dst, Src0, Src1);
+}
+
+void TcgEmitter::genVecOrScalarBinOp(StringRef Op, const TcgV &Dst,
+                                     const TcgV &Src0, const TcgV &Src1) {
+    assertKinds({{Dst, IrValue | IrPtrToOffset},
+                 {Src0, IrValue | IrPtrToOffset | IrImmediate},
+                 {Src1, IrValue | IrPtrToOffset | IrImmediate}});
+    switch (Dst.Kind) {
+    case IrValue: {
+        Out << "tcg_gen_" << Op << "_i" << Dst.tcgBitWidth() << "(";
+        emitArgListTcg({Dst, materialize(Src0), materialize(Src1)});
+        Out << ");\n";
+    } break;
+    case IrPtrToOffset: {
+        genVecBinOpStr(Op, Dst, Src0, Src1);
+    } break;
+    default:
+        abort();
+    }
+}
+
+void TcgEmitter::genVecSignedMax(const TcgV &Dst, const TcgV &Src0,
+                                 const TcgV &Src1) {
+    genVecOrScalarBinOp("smax", Dst, Src0, Src1);
+}
+
+void TcgEmitter::genVecUnsignedMax(const TcgV &Dst, const TcgV &Src0,
+                                   const TcgV &Src1) {
+    genVecOrScalarBinOp("umax", Dst, Src0, Src1);
+}
+
+void TcgEmitter::genVecSignedMin(const TcgV &Dst, const TcgV &Src0,
+                                 const TcgV &Src1) {
+    genVecOrScalarBinOp("smin", Dst, Src0, Src1);
+}
+
+void TcgEmitter::genVecUnsignedMin(const TcgV &Dst, const TcgV &Src0,
+                                   const TcgV &Src1) {
+    genVecOrScalarBinOp("umin", Dst, Src0, Src1);
+}
+
+void TcgEmitter::genVecMemcpy(const TcgV &Dst, const TcgV &Src,
+                              const TcgV &Size) {
+    genVecUnaryCall("tcg_gen_gvec_mov", 8, Dst, Src, Size);
+}
+
+void TcgEmitter::genVecMemset(const TcgV &Dst, const TcgV &Src,
+                              const TcgV &Size) {
+    assertKinds({{Dst, IrPtrToOffset},
+                 {Src, IrValue | IrImmediate},
+                 {Size, IrImmediate}});
+    switch (Src.Kind) {
+    case IrValue:
+        Out << "tcg_gen_gvec_dup_i" << Src.tcgBitWidth() << "(MO_"
+            << Src.llvmBitWidth() << ", ";
+        emitArgListTcg({Dst, Size, Size, Src});
+        Out << ");\n";
+        break;
+    case IrImmediate:
+        Out << "tcg_gen_gvec_dup_imm" << "(MO_" << Src.llvmBitWidth() << ", ";
+        emitArgListTcg({Dst, Size, Size, Src});
+        Out << ");\n";
+        break;
+    default:
+        abort();
+    }
+    // Track number of vector operations
+    ++NumVectorInstructions;
+}
+
+void TcgEmitter::genVecSplat(const TcgV &Dst, const TcgV &Src) {
+    const size_t Bytes = Dst.vecSize().bytes();
+    const auto Size = TcgV::makeImmediate(Twine(Bytes).str(), {T64, I64});
+    genVecMemset(Dst, Src, Size);
+}
+
+void TcgEmitter::genVecArrSplat(const TcgV &Dst, const TcgV &Src) {
+    assert(Src.ConstantExpression);
+    const VectorSize SrcSize = Src.vecSize();
+    const VectorSize DstSize = Dst.vecSize();
+    const std::string DstName = getName(Dst);
+    // NOTE: We are emitting static constant arrays of `uint64_t[]` with the
+    // purpose of initializing vectors to these constants using
+    // `tcg_gen_gvec_mov_var()`.  This should be equivalent to a vectorized load
+    // from a constant pointer into the read-only section of the binary, into a
+    // guest vector.  If this is problematic we'll have
+    // resubprojects/helper-to-tcg/passes/backend/TcgEmit.hsort to adding
+    // constant vector storage to `env` if needed.
+    const std::string ArrData = Twine(DstName).concat("_data").str();
+    Out << "static const " << C.intType(false, SrcSize.ElementBitWidth) << " "
+        << ArrData << "[] = " << getName(Src) << ";\n";
+    Out << "tcg_gen_gvec_mov_var(MO_" << (int)SrcSize.ElementBitWidth << ", "
+        << getName(getGlobalEnv()) << ", " << DstName << ", tcg_constant_ptr("
+        << ArrData << "), 0, " << SrcSize.bytes() << ", " << DstSize.bytes()
+        << ");\n";
+    // Track number of vector operations
+    ++NumVectorInstructions;
+}
+
+void TcgEmitter::genVecBitsel(const TcgV &Dst, const TcgV &Cond,
+                              const TcgV &Src0, const TcgV &Src1) {
+    assertKinds({{Dst, IrPtrToOffset},
+                 {Cond, IrPtrToOffset},
+                 {Src0, IrPtrToOffset},
+                 {Src1, IrPtrToOffset}});
+    genVecCall("bitsel", {Dst, Cond, Src0, Src1}, Src0.vecSize(),
+               Src0.vecSize());
+}
+
+void TcgEmitter::genVecCmp(const TcgV &Dst, const CmpInst::Predicate &Pred,
+                           const TcgV &Src0, const TcgV &Src1) {
+    assertKinds(
+        {{Dst, IrPtrToOffset}, {Src0, IrPtrToOffset}, {Src1, IrPtrToOffset}});
+    // NOTE: Return type of a LLVM vector compare is `<128 x i1>`, here the
+    // operand type is taken as the return type similar to how `icmp` is
+    // mapped.
+    const VectorSize Size = Src0.vecSize();
+    Out << "tcg_gen_gvec_cmp(" << mapPredicate(Pred) << ", " << "MO_"
+        << (int)Size.ElementBitWidth << ", " << getName(Dst) << ", "
+        << getName(Src0) << ", " << getName(Src1) << ", " << Size.bytes()
+        << ", " << Size.bytes() << ");\n";
+    // Track number of vector operations
+    ++NumVectorInstructions;
+}
+
+void TcgEmitter::genAbs(const TcgV &Dst, const TcgV &Src) {
+    assertKinds(
+        {{Dst, IrValue | IrPtrToOffset}, {Src, IrValue | IrPtrToOffset}});
+    switch (Dst.Kind) {
+    case IrValue: {
+        const auto FuncStr =
+            Twine("tcg_gen_abs_i").concat(Twine(Src.tcgBitWidth())).str();
+        emitCallTcg(FuncStr, {Dst, Src});
+    } break;
+    case IrPtrToOffset: {
+        genVecCall("abs", {Dst, Src}, Src.vecSize(), Dst.vecSize());
+    } break;
+    default:
+        abort();
+    }
+}
+
+void TcgEmitter::genVecNot(const TcgV &Dst, const TcgV &Src) {
+    assertKinds({{Dst, IrPtrToOffset}, {Src, IrPtrToOffset}});
+    genVecCall("not", {Dst, Src}, Dst.vecSize(), Dst.vecSize());
+}
+
+void TcgEmitter::genVecSizeChange(StringRef Name, uint8_t DstElementBits,
+                                  const TcgV &Dst, const TcgV &Src) {
+    assertKinds({{Dst, IrPtrToOffset}, {Src, IrPtrToOffset}});
+    const VectorSize Size = Src.vecSize();
+    Out << "gen_vec_" << Name << "_" << (int)Size.ElementBitWidth << "_"
+        << (int)DstElementBits << "(" << getName(Dst) << ", " << getName(Src)
+        << ", " << Size.bytes() << ");\n";
+    // Track number of vector operations
+    ++NumVectorInstructions;
+    // Indicates definitions of gen_helper_vec_[zext|trunc_sext] need to be
+    // emitted.
+    NeedVectorSizeChangeOps = true;
+}
+
+void TcgEmitter::genVecTrunc(uint8_t DstElementBits, const TcgV &Dst,
+                             const TcgV &Src) {
+    genVecSizeChange("trunc", DstElementBits, Dst, Src);
+}
+
+void TcgEmitter::genVecSext(uint8_t DstElementBits, const TcgV &Dst,
+                            const TcgV &Src) {
+    genVecSizeChange("sext", DstElementBits, Dst, Src);
+}
+
+void TcgEmitter::genVecZext(uint8_t DstElementBits, const TcgV &Dst,
+                            const TcgV &Src) {
+    genVecSizeChange("zext", DstElementBits, Dst, Src);
+}
+
+static inline size_t roundToMultiple(size_t X, size_t Multiple) {
+    return Multiple * ((X + Multiple - 1) / Multiple);
+}
+
+std::string CEmitter::intType(bool Signed, uint8_t LlvmSize) {
+    return Twine((Signed) ? "int" : "uint")
+        .concat(Twine(roundToMultiple(LlvmSize, 8)))
+        .concat("_t")
+        .str();
+}
+
+inline StringRef mapCPredicate(const CmpInst::Predicate &Pred) {
+    switch (Pred) {
+    case CmpInst::ICMP_EQ:
+        return "==";
+    case CmpInst::ICMP_NE:
+        return "!=";
+    case CmpInst::ICMP_UGT:
+        return ">";
+    case CmpInst::ICMP_UGE:
+        return ">=";
+    case CmpInst::ICMP_ULT:
+        return "<";
+    case CmpInst::ICMP_ULE:
+        return "<=";
+    case CmpInst::ICMP_SGT:
+        return ">";
+    case CmpInst::ICMP_SGE:
+        return ">=";
+    case CmpInst::ICMP_SLT:
+        return "<";
+    case CmpInst::ICMP_SLE:
+        return "<=";
+    default:
+        abort();
+    }
+}
+
+inline bool predicateNeedsSignCast(const CmpInst::Predicate &Pred) {
+    switch (Pred) {
+    case CmpInst::ICMP_EQ:
+    case CmpInst::ICMP_NE:
+    case CmpInst::ICMP_UGT:
+    case CmpInst::ICMP_UGE:
+    case CmpInst::ICMP_ULT:
+    case CmpInst::ICMP_ULE:
+        return false;
+    case CmpInst::ICMP_SGT:
+    case CmpInst::ICMP_SGE:
+    case CmpInst::ICMP_SLT:
+    case CmpInst::ICMP_SLE:
+        return true;
+    default:
+        abort();
+    }
+}
+
+enum BinOpSrcCast {
+    CastNone,
+    CastSigned,
+    CastUnsigned,
+};
+
+std::string CEmitter::mapBinOp(const Instruction::BinaryOps &Opcode,
+                               const TcgV &Src0, const TcgV &Src1) {
+    assert(Src0.Kind == IrImmediate and Src1.Kind == IrImmediate);
+    std::string Op;
+    BinOpSrcCast CastSrc0 = CastNone;
+    BinOpSrcCast CastSrc1 = CastNone;
+    switch (Opcode) {
+    case Instruction::Add:
+        Op = "+";
+        break;
+    case Instruction::And:
+        Op = "&";
+        break;
+    case Instruction::AShr:
+        CastSrc0 = CastSigned;
+        Op = ">>";
+        break;
+    case Instruction::LShr:
+        CastSrc0 = CastUnsigned;
+        Op = ">>";
+        break;
+    case Instruction::Shl:
+        Op = "<<";
+        break;
+    case Instruction::Mul:
+        Op = "*";
+        break;
+    case Instruction::UDiv:
+        CastSrc0 = CastUnsigned;
+        CastSrc1 = CastUnsigned;
+        Op = "/";
+        break;
+    case Instruction::SDiv:
+        CastSrc0 = CastSigned;
+        CastSrc1 = CastSigned;
+        Op = "/";
+        break;
+    case Instruction::Or:
+        Op = "|";
+        break;
+    case Instruction::Sub:
+        Op = "-";
+        break;
+    case Instruction::Xor:
+        Op = "^";
+        break;
+    default:
+        abort();
+    }
+
+    std::string Expr = "";
+    llvm::raw_string_ostream ExprStream(Expr);
+    ExprStream << "(";
+    if (CastSrc0 != CastNone) {
+        ExprStream << "("
+                   << intType(CastSrc0 == CastSigned, Src0.llvmBitWidth())
+                   << ") ";
+    }
+    ExprStream << getName(Src0) << " " << Op << " ";
+    if (CastSrc1 != CastNone) {
+        ExprStream << "("
+                   << intType(CastSrc1 == CastSigned, Src1.llvmBitWidth())
+                   << ") ";
+    }
+    ExprStream << getName(Src1) << ")";
+    ExprStream.flush();
+
+    return Expr;
+}
+
+// TODO:
+TcgV CEmitter::ptrAdd(const TcgV &Ptr, const TcgV &Offset) {
+    assert(Offset.Kind == IrImmediate);
+    switch (Ptr.Kind) {
+    case IrImmediate: {
+        std::string Expr = "";
+        llvm::raw_string_ostream ExprStream(Expr);
+        ExprStream << "(" << intType(false, Ptr.tcgBitWidth())
+                   << " *) ((uintptr_t) " << getName(Ptr) << " + "
+                   << getName(Offset) << ")";
+        ExprStream.flush();
+        return TcgV::makeImmediate(Expr, Ptr.intSize());
+    };
+    case IrPtrToOffset: {
+        std::string Expr = "";
+        llvm::raw_string_ostream ExprStream(Expr);
+        ExprStream << "(" << getName(Ptr) << " + " << getName(Offset) << ")";
+        ExprStream.flush();
+        TcgV Dst = Ptr;
+        Dst.Kind = IrPtrToOffset;
+        Dst.Name = Expr;
+        Dst.ConstantExpression = true;
+        return Dst;
+    };
+    default:
+        abort();
+    }
+}
+
+TcgV CEmitter::ternary(const TcgV &Cond, const TcgV &True, const TcgV &False) {
+    assert(Cond.Kind == IrImmediate);
+    std::string Expr = "";
+    llvm::raw_string_ostream ExprStream(Expr);
+    ExprStream << "(" << getName(Cond) << " ? " << getName(True) << " : "
+               << getName(False) << ")";
+    ExprStream.flush();
+    return TcgV::makeConstantExpression(Expr, True.intSize(), True.Kind);
+}
+
+TcgV CEmitter::deref(const TcgV &Ptr, ValueSize Size) {
+    assert(Ptr.Kind == IrImmediate);
+    std::string Expr = Twine("*").concat(getName(Ptr)).str();
+    return TcgV::makeImmediate(Expr, Size);
+}
+
+TcgV CEmitter::compare(const CmpInst::Predicate &Pred, const TcgV &Lhs,
+                       const TcgV &Rhs) {
+    assert(Lhs.Kind == IrImmediate and Rhs.Kind == IrImmediate);
+    const bool NeedsCast = predicateNeedsSignCast(Pred);
+    const std::string LhsCast =
+        (NeedsCast) ? "(" + intType(true, Lhs.llvmBitWidth()) + ")" : "";
+    const std::string RhsCast =
+        (NeedsCast) ? "(" + intType(true, Rhs.llvmBitWidth()) + ")" : "";
+    std::string Expr = "";
+    llvm::raw_string_ostream ExprStream(Expr);
+    ExprStream << "(" << LhsCast << getName(Lhs) << " " << mapCPredicate(Pred)
+               << " " << RhsCast << getName(Rhs) << ")";
+    ExprStream.flush();
+    return TcgV::makeImmediate(Expr, Lhs.intSize());
+}
+
+TcgV CEmitter::extend(bool Signed, const TcgV &V, ValueSize Size) {
+    assert(V.Kind == IrImmediate or
+           (V.ConstantExpression and V.Kind == IrValue));
+    std::string Expr = "";
+    llvm::raw_string_ostream ExprStream(Expr);
+    ExprStream << "((" << intType(Signed, Size.LlvmBitWidth) << ") ("
+               << intType(Signed, V.llvmBitWidth()) << ") " << getName(V)
+               << ")";
+    ExprStream.flush();
+    return TcgV::makeImmediate(Expr, Size);
+}
+
+TcgV CEmitter::binop(Instruction::BinaryOps Opcode, const TcgV &Src0,
+                     const TcgV &Src1) {
+    std::string Op = mapBinOp(Opcode, Src0, Src1);
+    ValueSize LargestSize = (Src0.llvmBitWidth() > Src1.llvmBitWidth())
+                                ? Src0.intSize()
+                                : Src1.intSize();
+    return TcgV::makeImmediate(Op, LargestSize);
+}
+
+void emitVectorMem(raw_ostream &Out) {
+    Out << "typedef struct VectorMem {\n";
+    Out << "    uint32_t allocated;\n";
+    Out << "} VectorMem;\n\n";
+
+    Out << "static intptr_t temp_new_gvec(VectorMem *mem, uint32_t size)\n";
+    Out << "{\n";
+    Out << "    uint32_t off = ROUND_UP(mem->allocated, size);\n";
+    Out << "    g_assert(off + size <= STRUCT_SIZEOF_FIELD(CPUArchState, "
+        << TempVectorBlock << "));\n";
+    Out << "    mem->allocated = off + size;\n";
+    Out << "    return offsetof(CPUArchState, " << TempVectorBlock
+        << ") + off;\n";
+    Out << "}\n";
+}
+
+static void emitVectorSizeChangeHelper(raw_ostream &OutSource,
+                                       raw_ostream &OutHeader, StringRef Name,
+                                       StringRef IntPrefix, int SrcSize,
+                                       int DstSize) {
+    const std::string NameWithTypes =
+        (Twine("vec_") + Name + "_" + Twine(SrcSize) + "_" + Twine(DstSize))
+            .str();
+
+    // Emit helper declarations.
+    OutHeader << "DEF_HELPER_FLAGS_3(" << NameWithTypes
+              << ", TCG_CALL_NO_RWG, void, ptr, ptr, i32)\n";
+
+    // Emit helper definitions.
+    OutSource << "void HELPER(" << NameWithTypes
+              << ")(void *d, void *a, uint32_t size)\n{\n";
+    OutSource << "for (intptr_t i = 0; i < (size / sizeof(" << IntPrefix
+              << SrcSize << "_t)); ++i) {\n";
+    OutSource << IntPrefix << SrcSize << "_t aa = *((" << IntPrefix << SrcSize
+              << "_t *) a + i);\n";
+    OutSource << "*((" << IntPrefix << DstSize << "_t *) d + i) = aa;\n";
+    OutSource << "}\n";
+    OutSource << "}\n\n";
+
+    // Emit gen_vec_*() function invoking helper functions.
+    OutSource << "static inline void G_GNUC_UNUSED gen_" << NameWithTypes
+              << "(intptr_t dofs, intptr_t aofs, uint32_t size)\n{\n";
+    OutSource << "TCGv_ptr d = tcg_temp_new_ptr();\n";
+    OutSource << "TCGv_ptr a = tcg_temp_new_ptr();\n";
+    OutSource << "tcg_gen_addi_ptr(d, tcg_env, dofs);\n";
+    OutSource << "tcg_gen_addi_ptr(a, tcg_env, aofs);\n";
+    OutSource << "gen_helper_" << NameWithTypes
+              << "(d, a, tcg_constant_i32(size));\n";
+    OutSource << "}\n\n";
+}
+
+static void emitVectorIndex(raw_ostream &OutSource, const VectorLayout &VL,
+                            size_t LaneSize) {
+    const size_t LanesPerColumn = 64 / LaneSize;
+    const size_t ColumnsPerBlock = VL.BlockBytes / 8;
+    OutSource << "inline size_t ind_" << LaneSize << "(size_t i) {\n";
+    OutSource << "const size_t c = i / " << LanesPerColumn << ";\n";
+    OutSource << "const size_t b = c / " << ColumnsPerBlock << ";\n";
+    OutSource << "const size_t ci = i % " << LanesPerColumn << ";\n";
+    OutSource << "const size_t bc = c % " << ColumnsPerBlock << ";\n";
+    OutSource << "return " << (8 * VL.BlockBytes) / LaneSize << " * b";
+    if (VL.HostBigEndian ^ (VL.Lane0 == MostSignificant)) {
+        OutSource << " + " << LanesPerColumn << " * (" << ColumnsPerBlock - 1
+                  << " - bc)";
+    } else {
+        OutSource << " + " << LanesPerColumn << " * bc";
+    }
+    if (VL.HostBigEndian ^ (VL.Lane0 == MostSignificant)) {
+        OutSource << " + (" << LanesPerColumn - 1 << " - ci);\n";
+    } else {
+        OutSource << " + ci;\n";
+    }
+    OutSource << "}\n\n";
+}
+
+void emitVectorSizeChangeOps(raw_ostream &OutSource, raw_ostream &OutHeader,
+                             const VectorLayout &VL) {
+    OutSource << "#define HELPER_H \"helper-to-tcg-support-helpers.h\"\n";
+    OutSource << "#include \"exec/helper-proto-common.h\"\n";
+    OutSource << "#include \"exec/helper-proto.h.inc\"\n";
+    OutSource << "#include \"exec/helper-gen-common.h\"\n";
+    OutSource << "#include \"exec/helper-gen.h.inc\"\n\n";
+
+    const uint8_t Sizes[] = {8, 16, 32, 64};
+
+    for (uint8_t Size : Sizes) {
+        emitVectorIndex(OutSource, VL, Size);
+    }
+
+    for (uint8_t SmallSize : Sizes) {
+        for (uint8_t LargeSize : Sizes) {
+            if (SmallSize >= LargeSize) {
+                continue;
+            }
+            emitVectorSizeChangeHelper(OutSource, OutHeader, "trunc", "uint",
+                                       LargeSize, SmallSize);
+            emitVectorSizeChangeHelper(OutSource, OutHeader, "zext", "uint",
+                                       SmallSize, LargeSize);
+            emitVectorSizeChangeHelper(OutSource, OutHeader, "sext", "int",
+                                       SmallSize, LargeSize);
+        }
+    }
+}
+
+void emitHelperGen(raw_ostream &Out, StringRef Name, Type *ReturnTy,
+                   ArrayRef<Type *> ArgTys) {
+    // Given LLVM Type produce the corresponding TCG type that would be expected
+    // as a helper argument.
+    auto emitType = [&](Type *Ty) {
+        switch (Ty->getTypeID()) {
+        case Type::IntegerTyID: {
+            auto IntTy = cast<IntegerType>(Ty);
+            auto Size = ValueSize::fromLlvmType(IntTy);
+            assert(Size);
+            Out << "TCGv_i" << (int)Size->TcgBitWidth;
+            return true;
+        }
+        case Type::PointerTyID: {
+            Out << "TCGv_ptr";
+            return true;
+        }
+        default:
+            return false;
+        };
+    };
+
+    // Given LLVM Type and name string produce the neccessary `TCGv` to
+    // `TCGTemp` conversion expeted by `tcg_gen_callN`.
+    auto emitTemp = [&](Type *Ty, StringRef str) {
+        switch (Ty->getTypeID()) {
+        case Type::IntegerTyID: {
+            auto IntTy = cast<IntegerType>(Ty);
+            auto Size = ValueSize::fromLlvmType(IntTy);
+            assert(Size);
+            Out << "tcgv_i" << (int)Size->TcgBitWidth << "_temp(" << str << ")";
+        } break;
+        case Type::PointerTyID: {
+            Out << "tcgv_ptr_temp(" << str << ")";
+        } break;
+        default:
+            Out << "NULL";
+            break;
+        };
+    };
+
+    Out << "extern TCGHelperInfo helper_info_" << Name << ";\n";
+    Out << "static inline void gen_helper_" << Name << "(";
+
+    bool HasRet = emitType(ReturnTy);
+    if (HasRet) {
+        Out << " ret";
+    }
+    for (int i = 0; i < ArgTys.size(); ++i) {
+        if (HasRet or i > 0) {
+            Out << ", ";
+        }
+        assert(emitType(ArgTys[i]));
+        Out << " a" << i;
+    }
+    Out << ")\n{\n";
+
+    Out << "tcg_gen_call" << ArgTys.size() << "(helper_info_" << Name << ".func"
+        << ", &helper_info_" << Name << ", ";
+
+    emitTemp(ReturnTy, "ret");
+    for (int i = 0; i < ArgTys.size(); ++i) {
+        Out << ", ";
+        emitTemp(ArgTys[i], Twine("a").concat(Twine(i)).str());
+    }
+    Out << ");\n";
+    Out << "}\n";
+}
diff --git a/subprojects/helper-to-tcg/src/TcgGenPass/TcgEmit.hpp b/subprojects/helper-to-tcg/src/TcgGenPass/TcgEmit.hpp
new file mode 100644
index 0000000000..26d0ba2afe
--- /dev/null
+++ b/subprojects/helper-to-tcg/src/TcgGenPass/TcgEmit.hpp
@@ -0,0 +1,315 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#pragma once
+
+#include "TcgType.hpp"
+
+#include <llvm/ADT/StringRef.h>
+#include <llvm/ADT/Twine.h>
+#include <llvm/IR/InstrTypes.h> // for CmpInst::Predicate
+#include <llvm/IR/Instruction.h>
+#include <llvm/IR/Value.h>
+#include <llvm/Support/raw_ostream.h>
+
+#include <string>
+
+using llvm::CmpInst;
+using llvm::Instruction;
+using llvm::raw_ostream;
+using llvm::SmallVector;
+using llvm::StringRef;
+using llvm::Twine;
+using llvm::Type;
+using llvm::Value;
+
+class CEmitter;
+
+struct VectorLayout;
+
+inline std::string getName(const TcgV &V) {
+    if (!V.Name.empty() or V.ConstantExpression) {
+        return V.Name;
+    } else {
+        switch (V.Kind) {
+        case IrImmediate:
+            return (!V.Name.empty()) ? V.Name
+                                     : Twine("imm").concat(Twine(V.Id)).str();
+        case IrValue:
+            return Twine("tmp").concat(Twine(V.Id)).str();
+        case IrPtr:
+            return Twine("ptr").concat(Twine(V.Id)).str();
+        case IrPtrToOffset:
+            return Twine("vec").concat(Twine(V.Id)).str();
+        case IrLabel:
+            return Twine("label").concat(Twine(V.Id)).str();
+        default:
+            abort();
+        };
+    }
+}
+
+// Small helper class to construct `Twine`s on the stack.
+template <size_t N> class StackTwine {
+    llvm::SmallString<N> Str;
+
+  public:
+    StackTwine() = delete;
+    StackTwine(Twine &&T) { T.toVector(Str); }
+    operator StringRef() { return Str; }
+};
+
+class TcgEmitter {
+    raw_ostream &Out;
+    CEmitter &C;
+    size_t AllocatedVectorMemory = 0;
+    size_t NumVectorInstructions = 0;
+    bool EmittedVectorMem = false;
+
+    void genVecUnaryCall(StringRef Name, int ElementSize, const TcgV &Dst,
+                         const TcgV &Src, const TcgV &Size);
+    void genVecBinaryCall(StringRef Name, int ElementSize, const TcgV &Dst,
+                          const TcgV &Src0, const TcgV &Src1, size_t Size);
+    void genVecCall(StringRef Name, llvm::ArrayRef<const TcgV> Args,
+                    const VectorSize OpSize, const VectorSize MaxSize);
+    void genVecSizeChange(StringRef Name, uint8_t DstElementBits,
+                          const TcgV &Dst, const TcgV &Src);
+
+  public:
+    bool NeedVectorSizeChangeOps = false;
+
+    TcgEmitter(raw_ostream &Out, CEmitter &C) : Out(Out), C(C) {}
+
+    inline size_t allocatedVectorMemory() const {
+        return AllocatedVectorMemory;
+    }
+    inline size_t numVectorInstructions() const {
+        return NumVectorInstructions;
+    }
+
+    const TcgV getGlobalEnv() {
+        return TcgV::makeConstantExpression("tcg_env", {}, IrPtr);
+    }
+
+    const TcgV getDisasContext() {
+        return TcgV::makeConstantExpression("ctx", {}, IrPtr);
+    }
+
+    // String representation of types
+    const std::string getType(const TcgV &Value);
+
+    inline const TcgV materialize(const TcgV &Value) {
+        if (Value.Kind != IrImmediate) {
+            return Value;
+        }
+        TcgV M = Value;
+        M.Name = Twine("tcg_constant_i")
+                     .concat(Twine((int)Value.tcgBitWidth()))
+                     .concat("(")
+                     .concat(getName(Value))
+                     .concat(")")
+                     .str();
+        M.Kind = IrValue;
+        return M;
+    }
+
+    template <typename I> void emitArgListTcg(const I Beg, const I End) {
+        auto It = Beg;
+        if (It != End) {
+            Out << getName(*It);
+            ++It;
+        }
+        while (It != End) {
+            Out << ", " << getName(*It);
+            ++It;
+        }
+    }
+
+    template <typename I>
+    void emitCall(const StringRef S, const I Beg, const I End) {
+        Out << S << '(';
+        auto It = Beg;
+        if (It != End) {
+            Out << *It;
+            ++It;
+        }
+        while (It != End) {
+            Out << ", " << *It;
+            ++It;
+        }
+        Out << ");\n";
+    }
+
+    template <typename Iterator>
+    void emitCallTcg(const StringRef S, Iterator Begin, Iterator End) {
+        assert(Begin != End);
+        Out << S << '(';
+        Out << getName(*Begin);
+        ++Begin;
+        while (Begin != End) {
+            Out << ", " << getName(*Begin);
+            ++Begin;
+        }
+        Out << ");\n";
+    }
+
+    inline void emitArgListTcg(const std::initializer_list<TcgV> Args) {
+        emitArgListTcg(Args.begin(), Args.end());
+    }
+
+    inline void emitCall(const StringRef S,
+                         const std::initializer_list<StringRef> Args) {
+        emitCall(S, Args.begin(), Args.end());
+    }
+
+    inline void emitCallTcg(StringRef S, std::initializer_list<TcgV> Args) {
+        emitCallTcg(S, Args.begin(), Args.end());
+    }
+
+    inline void genCallHelper(const StringRef Helper,
+                              const std::initializer_list<TcgV> Args) {
+        auto Func = Twine("gen_").concat(Helper).str();
+        emitCallTcg(Func, Args);
+    }
+
+    template <typename I>
+    void genCallHelper(const StringRef Helper, I Beg, I End) {
+        auto Func = Twine("gen_").concat(Helper).str();
+        emitCallTcg(Func, Beg, End);
+    }
+
+    template <typename I>
+    void genCallCFunc(const StringRef Name, TcgV Ret, I Beg, I End) {
+        if (Ret.Kind != IrInvalid) {
+            Out << getName(Ret) << " = ";
+        }
+        emitCallTcg(Name, Beg, End);
+    }
+
+    void genNewLabel();
+    void genSetLabel(const TcgV &L);
+
+    void defineNewTemp(const TcgV &Tcg);
+
+    void genBr(const TcgV &L);
+
+    void genExts(const TcgV &Dst, const TcgV &Src);
+    void genExtI32I64(const TcgV &Dst, const TcgV &Src);
+    void genExtrlI64I32(const TcgV &Dst, const TcgV &Src);
+    void genExtuI32I64(const TcgV &Dst, const TcgV &Src);
+    void genExtrhI64I32(const TcgV &Dst, const TcgV &Src);
+    void genExtract(bool Sign, const TcgV &Dst, const TcgV &Src,
+                    const TcgV &Offset, const TcgV &Length);
+    void genDeposit(const TcgV &Dst, const TcgV &Into, const TcgV &From,
+                    const TcgV &Offset, const TcgV &Length);
+
+    void genPtrToValue(const TcgV &Dst, const TcgV &Src);
+    void genValueToPtr(const TcgV &Dst, const TcgV &Src);
+
+    void genConcat(const TcgV &Dst, const TcgV &Src1, const TcgV &Src2);
+    void genMov(const TcgV &Dst, const TcgV &Src);
+    void genMovPtr(const TcgV &Dst, const TcgV &Src);
+    void genAddPtr(const TcgV &Dst, const TcgV &Ptr, const TcgV &Offset);
+    void genBinOp(const TcgV &Dst, const Instruction::BinaryOps Opcode,
+                  const TcgV &Src0, const TcgV &Src1);
+    void genMovcond(const CmpInst::Predicate &Pred, const TcgV &Ret,
+                    const TcgV &C1, const TcgV &C2, const TcgV &V1,
+                    const TcgV &V2);
+    void genSetcond(const CmpInst::Predicate &Pred, const TcgV &Dst,
+                    const TcgV &Op1, const TcgV &Op2);
+    void genBrcond(const CmpInst::Predicate &Pred, const TcgV &Arg1,
+                   const TcgV &Arg2, const TcgV &Label);
+
+    std::string getMemOp(uint8_t Size, uint8_t Endianness, uint8_t Sign);
+    void genGuestLoad(const TcgV &Dst, const TcgV &Ptr, StringRef MemOp);
+    void genGuestStore(const TcgV &Ptr, const TcgV &Src, StringRef MemOp);
+    void genHostLoad(const TcgV &Dst, const TcgV &Ptr, const TcgV &Offset);
+    void genHostLoadFromVec(const TcgV &Dst, const TcgV &Offset);
+    void genHostStore(const TcgV &Ptr, const TcgV &Src);
+
+    void genFunnelShl(const TcgV &Dst, const TcgV &Src0, const TcgV &Src1,
+                      const TcgV &Shift);
+    void genBitreverse(const TcgV &Dst, const TcgV &Src);
+    void genAbs(const TcgV &Dst, const TcgV &Src);
+    void genCountLeadingZeros(const TcgV &Dst, const TcgV &Src);
+    void genCountTrailingZeros(const TcgV &Dst, const TcgV &Src);
+    void genCountOnes(const TcgV &Dst, const TcgV &Src);
+    void genByteswap(const TcgV &Dst, const TcgV &Src);
+    void genUnsignedSatSub(const TcgV &Dst, const TcgV &Src0, const TcgV &Src1);
+
+    // Vector ops.
+    void genVecBinOpStr(StringRef Op, const TcgV &Dst, const TcgV &Src0,
+                        const TcgV &Src1);
+    void genVecBinOp(const Instruction::BinaryOps Opcode, const TcgV &Dst,
+                     const TcgV &Src0, const TcgV &Src1);
+    void genVecOrScalarBinOp(StringRef Op, const TcgV &Dst, const TcgV &Src0,
+                             const TcgV &Src1);
+    void genVecSignedSatAdd(const TcgV &Dst, const TcgV &Src0,
+                            const TcgV &Src1);
+    void genVecSignedSatSub(const TcgV &Dst, const TcgV &Src0,
+                            const TcgV &Src1);
+    void genVecUnsignedSatSub(const TcgV &Dst, const TcgV &Src0,
+                              const TcgV &Src1);
+    void genVecSignedMax(const TcgV &Dst, const TcgV &Src0, const TcgV &Src1);
+    void genVecUnsignedMax(const TcgV &Dst, const TcgV &Src0, const TcgV &Src1);
+    void genVecSignedMin(const TcgV &Dst, const TcgV &Src0, const TcgV &Src1);
+    void genVecUnsignedMin(const TcgV &Dst, const TcgV &Src0, const TcgV &Src1);
+    void genVecMemcpy(const TcgV &Dst, const TcgV &Src, const TcgV &Size);
+    void genVecMemset(const TcgV &Dst, const TcgV &Src, const TcgV &Size);
+    void genVecSplat(const TcgV &Dst, const TcgV &Src);
+    void genVecArrSplat(const TcgV &Dst, const TcgV &Src);
+    void genVecBitsel(const TcgV &Dst, const TcgV &Cond, const TcgV &Src0,
+                      const TcgV &Src1);
+    void genVecCmp(const TcgV &Dst, const CmpInst::Predicate &Pred,
+                   const TcgV &Src0, const TcgV &Src1);
+    void genVecNot(const TcgV &Dst, const TcgV &Src);
+    void genVecTrunc(uint8_t DstElementBits, const TcgV &Dst, const TcgV &Src);
+    void genVecSext(uint8_t DstElementBits, const TcgV &Dst, const TcgV &Src);
+    void genVecZext(uint8_t DstElementBits, const TcgV &Dst, const TcgV &Src);
+};
+
+class CEmitter {
+    std::string mapBinOp(const Instruction::BinaryOps &Opcode, const TcgV &Src0,
+                         const TcgV &Src1);
+
+  public:
+    CEmitter() {}
+
+    std::string intType(bool Signed, uint8_t LlvmSize);
+    TcgV ptrAdd(const TcgV &Ptr, const TcgV &Offset);
+    TcgV ternary(const TcgV &Cond, const TcgV &True, const TcgV &False);
+    TcgV deref(const TcgV &Ptr, ValueSize Size);
+    TcgV compare(const CmpInst::Predicate &Pred, const TcgV &Src0,
+                 const TcgV &Src1);
+    TcgV extend(bool Signed, const TcgV &V, ValueSize Size);
+    TcgV binop(Instruction::BinaryOps Opcode, const TcgV &Src0,
+               const TcgV &Src1);
+};
+
+// The below function deal with a lot of code generation that we expect to be
+// present in the output.
+
+// Emits `VectorMem` struct definition along with allocation funcs needed for
+// producing vector temporaries.
+void emitVectorMem(raw_ostream &Out);
+// Emit `gen_vec_zext*()` and friends for performing vector size changing
+// operations, taking target vector layout into account.
+void emitVectorSizeChangeOps(raw_ostream &OutSource, raw_ostream &OutHeader,
+                             const VectorLayout &VL);
+// Emits gen_helper_*() definition for helpers that failed to translate.
+void emitHelperGen(raw_ostream &Out, StringRef Name, Type *ReturnTy,
+                   llvm::ArrayRef<Type *> ArgTys);
-- 
2.52.0

[PATCH v2 34/50] helper-to-tcg: TcgGenPass, map arguments to TCG

[Anton Johansson via qemu development]

Mapping for arguments is slighty different than other `Value`s and depend
on function annotations, perform it as a separe pass.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 subprojects/helper-to-tcg/meson.build         |   1 +
 .../src/TcgGenPass/MapArguments.cpp           | 122 ++++++++++++++++++
 2 files changed, 123 insertions(+)
 create mode 100644 subprojects/helper-to-tcg/src/TcgGenPass/MapArguments.cpp

diff --git a/subprojects/helper-to-tcg/meson.build b/subprojects/helper-to-tcg/meson.build
index c03a4d5df6..22190025d0 100644
--- a/subprojects/helper-to-tcg/meson.build
+++ b/subprojects/helper-to-tcg/meson.build
@@ -51,6 +51,7 @@ sources = [
     'src/TcgGenPass/TcgGenPass.cpp',
     'src/TcgGenPass/LinearizeBlocks.cpp',
     'src/TcgGenPass/TcgEmit.cpp',
+    'src/TcgGenPass/MapArguments.cpp',
 ]
 
 clang = bindir / 'clang'
diff --git a/subprojects/helper-to-tcg/src/TcgGenPass/MapArguments.cpp b/subprojects/helper-to-tcg/src/TcgGenPass/MapArguments.cpp
new file mode 100644
index 0000000000..04aa8cf10c
--- /dev/null
+++ b/subprojects/helper-to-tcg/src/TcgGenPass/MapArguments.cpp
@@ -0,0 +1,122 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#include "DebugInfo.hpp"
+#include "ValueMapping.hpp"
+
+#include "Error.hpp"
+#include "FunctionAnnotation.hpp"
+#include "TcgType.hpp"
+
+#include <llvm/IR/Constants.h>
+#include <llvm/IR/DerivedTypes.h>
+#include <llvm/IR/Function.h>
+#include <llvm/IR/Module.h>
+
+#define DEBUG_TYPE "map-arguments"
+
+//
+// Map Arguments
+//
+// Assign `TcgV`s to function arguments, taking function annotations into
+// account to force types.  Mapping them early and separately both makes sure
+// that no other pass accidently assign to them, and separates out any special
+// case logic that's necessary.
+//
+
+using namespace llvm;
+
+static Expected<TcgV> mapIntegerArgument(TempAllocationData &TAD,
+                                         const DebugInfoMapTy &DebugInfo,
+                                         const Argument *Arg,
+                                         uint8_t Annotations) {
+    auto *IntTy = cast<IntegerType>(Arg->getType());
+    auto Size = ValueSize::fromLlvmType(IntTy);
+    if (!Size) {
+        return Size.takeError();
+    }
+
+    StringRef Name = getDebugVarName(DebugInfo, Arg);
+
+    if ((Annotations & (uint8_t)ArgumentAnnotation::Immediate) != 0) {
+        auto Tcg = TcgV::makeImmediate(Name, *Size);
+        return TAD.map(Arg, Tcg);
+    } else {
+        auto Tcg = TcgV::makeTemp(*Size, IrValue);
+        Tcg.Name = Name;
+        return TAD.map(Arg, Tcg);
+    }
+}
+
+static Expected<TcgV> mapPointerArgument(TempAllocationData &TAD,
+                                         const DebugInfoMapTy &DebugInfo,
+                                         const Argument *Arg,
+                                         uint8_t Annotations) {
+    // If the value has an associated name from the debug information, use it
+    StringRef Name{};
+    if (auto It = DebugInfo.find(Arg); It != DebugInfo.end()) {
+        Name = It->second.VarName;
+    }
+
+    if ((Annotations & (uint8_t)ArgumentAnnotation::PtrToOffset) != 0) {
+        auto Tcg = TcgV::makeVector({});
+        Tcg.Name = Name;
+        return TAD.map(Arg, Tcg);
+    } else {
+        auto Tcg = TcgV::makeTemp({}, IrPtr);
+        Tcg.Name = Name;
+        return TAD.map(Arg, Tcg);
+    }
+}
+
+static Expected<TcgV> mapArgument(TempAllocationData &Data,
+                                  const DebugInfoMapTy &DebugInfo,
+                                  const Argument *Arg, uint8_t Annotations) {
+    // We only map each argument once, assert it's not been mapped previously.
+    assert(!Data.Map.count(Arg));
+
+    Type *Ty = Arg->getType();
+    if (isa<IntegerType>(Ty)) {
+        return mapIntegerArgument(Data, DebugInfo, Arg, Annotations);
+    } else if (isa<PointerType>(Ty)) {
+        return mapPointerArgument(Data, DebugInfo, Arg, Annotations);
+    }
+
+    return mkError("Unable to map value ", Arg);
+}
+
+Error mapArguments(const llvm::Function &F,
+                   const AnnotationMapTy &AnnotationMap,
+                   const DebugInfoMapTy &DebugInfo, TempAllocationData &Data) {
+    // Map arguments
+    for (size_t I = 0; I < F.arg_size(); ++I) {
+        const Argument *Arg = F.getArg(I);
+        auto It = AnnotationMap.find(&F);
+        const uint8_t Annotations =
+            (It != AnnotationMap.end()) ? It->second.getArgFlag(I) : 0;
+        Expected<TcgV> TcgArg = mapArgument(Data, DebugInfo, Arg, Annotations);
+        if (!TcgArg) {
+            return TcgArg.takeError();
+        }
+        LLVM_DEBUG({
+            dbgs() << "Mapped argument " << *Arg << " to: \n";
+            TcgArg->dump(dbgs());
+        });
+    }
+
+    return Error::success();
+}
-- 
2.52.0

[PATCH v2 35/50] helper-to-tcg: TcgGenPass, propagate constant expresssions

[Anton Johansson via qemu development]

In a forward pass over the instructions, fold and propagate constant
expressions to `TcgV`s with a string representing the expression.

This needs to be performed before register allocation where `TcgV`
registers are assigned to `Value`s to get the best register reuse.
Register allocation is also performed in a backwards pass over the
instructions, so these two passes cannot be nicely combined.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 subprojects/helper-to-tcg/meson.build         |   1 +
 .../src/TcgGenPass/MapConstantExpressions.cpp | 328 ++++++++++++++++++
 2 files changed, 329 insertions(+)
 create mode 100644 subprojects/helper-to-tcg/src/TcgGenPass/MapConstantExpressions.cpp

diff --git a/subprojects/helper-to-tcg/meson.build b/subprojects/helper-to-tcg/meson.build
index 22190025d0..046cdb916b 100644
--- a/subprojects/helper-to-tcg/meson.build
+++ b/subprojects/helper-to-tcg/meson.build
@@ -52,6 +52,7 @@ sources = [
     'src/TcgGenPass/LinearizeBlocks.cpp',
     'src/TcgGenPass/TcgEmit.cpp',
     'src/TcgGenPass/MapArguments.cpp',
+    'src/TcgGenPass/MapConstantExpressions.cpp',
 ]
 
 clang = bindir / 'clang'
diff --git a/subprojects/helper-to-tcg/src/TcgGenPass/MapConstantExpressions.cpp b/subprojects/helper-to-tcg/src/TcgGenPass/MapConstantExpressions.cpp
new file mode 100644
index 0000000000..0d0810ebc8
--- /dev/null
+++ b/subprojects/helper-to-tcg/src/TcgGenPass/MapConstantExpressions.cpp
@@ -0,0 +1,328 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#include "ValueMapping.hpp"
+
+#include "DebugInfo.hpp"
+#include "Error.hpp"
+#include "PseudoInst.hpp"
+#include "TcgEmit.hpp"
+#include "TcgGlobalMap.hpp"
+#include "TcgType.hpp"
+
+#include <llvm/IR/BasicBlock.h>
+#include <llvm/IR/Constants.h>
+#include <llvm/IR/DerivedTypes.h>
+#include <llvm/IR/Function.h>
+#include <llvm/IR/Instructions.h>
+#include <llvm/IR/Module.h>
+
+using namespace llvm;
+
+//
+// Functions for mapping an LLVM Value to a TcgV
+//
+
+// Provides a C string representation of a ConstantInt
+static std::string constantIntToStr(const ConstantInt *C) {
+    SmallString<20> ResultStr;
+    auto *Int = cast<ConstantInt>(C);
+    const APInt Value = Int->getUniqueInteger();
+    const unsigned BitWidth = Int->getBitWidth();
+    if (BitWidth == 1) {
+        // Emit as bool
+        return (Value.getBoolValue()) ? "true" : "false";
+    } else if (BitWidth == 64 and !Int->isNegative() and Int->uge(0xefff)) {
+        // Emit hex-formatted integer as 64-bit constants often occur in vector
+        // expressions, and are way easier to read.
+        Value.toString(ResultStr, 16, false, true);
+        return Twine(ResultStr).str();
+    } else {
+        // Emit as signed integer
+        const char *SuffixStr = "";
+        if (Value.ugt(UINT32_MAX) or C->getBitWidth() == 64) {
+            SuffixStr = Int->isNegative() ? "ll" : "ull";
+        }
+        bool IsMax =
+            (Int->isNegative()) ? Value.isMaxSignedValue() : Value.isMaxValue();
+        bool IsMin = Int->isNegative() and Value.isMinSignedValue();
+        unsigned Bitwidth = Value.getBitWidth();
+        if (IsMax) {
+            return Twine("INT").concat(Twine(Bitwidth)).concat("_MAX").str();
+        } else if (IsMin) {
+            return Twine("INT").concat(Twine(Bitwidth)).concat("_MIN").str();
+        } else {
+            Value.toString(ResultStr, 10, Value.isNegative(), true);
+            return Twine(ResultStr).concat(SuffixStr).str();
+        }
+    }
+}
+
+static Expected<TcgV> mapIntegerConstant(TempAllocationData &TAD,
+                                         const ConstantInt *V) {
+    auto Size = ValueSize::fromLlvmType(cast<IntegerType>(V->getType()));
+    if (!Size) {
+        return Size.takeError();
+    }
+    auto Tcg = TcgV::makeTemp(*Size, IrImmediate);
+    Tcg.Name = constantIntToStr(V);
+    return TAD.map(V, Tcg);
+}
+
+static Expected<TcgV> mapVectorConstant(TempAllocationData &TAD,
+                                        const AnnotationMapTy &Annotations,
+                                        const DebugInfoMapTy &DebugInfo,
+                                        const Value *V, VectorType *VecTy) {
+    auto *Const = dyn_cast<Constant>(V);
+    if (!Const) {
+        return mkError("Non-constant vector");
+    }
+
+    auto Size = VectorSize::fromLlvmType(VecTy);
+    if (!Size) {
+        return Size.takeError();
+    }
+    auto Tcg = TcgV::makeVector(*Size);
+
+    // At this point, splatted vectors should have been converted to calls to
+    // @VecSplat, to more closely match TCG and benefit from future variable
+    // assignments.
+    assert(!Const->getSplatValue());
+
+    std::string ExprStr;
+    raw_string_ostream Expr(ExprStr);
+
+    // Map constant elements of vector where elements differ
+    //   <32 x i32> <i32 1, i32 2, ..., i32 16>
+
+    Expr << "{";
+    for (unsigned I = 0; I < Size->ElementCount; ++I) {
+        ConstantInt *C = cast<ConstantInt>(Const->getAggregateElement(I));
+        Expr << constantIntToStr(C);
+        if (I < Size->ElementCount - 1) {
+            Expr << ", ";
+        }
+    }
+    Expr << "}";
+    Tcg.Name = ExprStr;
+    Tcg.ConstantExpression = true;
+
+    return TAD.map(V, Tcg);
+}
+
+// Given a LLVM value, assigns a TcgV by type (integer, pointer, vector).  If
+// the given value has already been mapped to a TcgV, return it.
+static Expected<TcgV> mapConstant(TempAllocationData &TAD,
+                                  const AnnotationMapTy &AnnotationMap,
+                                  const DebugInfoMapTy &DebugInfo,
+                                  const Value *V) {
+    // Return previously mapped value
+    auto It = TAD.Map.find(V);
+    if (It != TAD.Map.end()) {
+        return It->second;
+    }
+
+    Type *Ty = V->getType();
+    if (auto *ConstInt = dyn_cast<ConstantInt>(V)) {
+        return mapIntegerConstant(TAD, ConstInt);
+    } else if (isa<VectorType>(Ty)) {
+        return mapVectorConstant(TAD, AnnotationMap, DebugInfo, V,
+                                 cast<VectorType>(Ty));
+    }
+
+    return mkError("Unable to map value ", V);
+}
+
+Error propagateConstantExpressions(CEmitter &C, const Function &F,
+                                   const LinearBlocks &Blocks,
+                                   const AnnotationMapTy &AnnotationMap,
+                                   const DebugInfoMapTy &DebugInfo,
+                                   const TcgGlobalMap &TcgGlobals,
+                                   TempAllocationData &TAD) {
+    SmallVector<Value *, 16> Worklist;
+    for (BasicBlock *BB : Blocks) {
+        for (Instruction &I : *BB) {
+            // Skip all instructions for which all operands have not been
+            // mapped, at this point we're only mapping constant expressions, so
+            // this is equivalent to all operands being constant expressions.
+            bool SkipInstruction = false;
+            SmallVector<TcgV, 4> Ops;
+            for (Value *V : getOperands(&I)) {
+                // Try and map `V` as a constant, or return previously mapped
+                // value. At this point any previously mapped value must be an
+                // argument to the function.
+                Expected<TcgV> T =
+                    mapConstant(TAD, AnnotationMap, DebugInfo, V);
+                if (!T or T->Kind == IrValue or T->Kind == IrPtr) {
+                    // Do break out here, we still need to map all operands,
+                    // consider
+                    //
+                    //   call @func(i32 %nonconst, i32 0).
+                    SkipInstruction = true;
+                    continue;
+                }
+                Ops.push_back(*T);
+            }
+            if (SkipInstruction) {
+                continue;
+            }
+
+            switch (I.getOpcode()) {
+            case Instruction::SExt:
+            case Instruction::ZExt: {
+                auto *IntTy = dyn_cast<IntegerType>(I.getType());
+                if (!IntTy) {
+                    continue;
+                }
+                auto Size = ValueSize::fromLlvmType(IntTy);
+                if (!Size) {
+                    return Size.takeError();
+                }
+                bool Signed = (I.getOpcode() == Instruction::SExt);
+                TAD.Map[&I] = C.extend(Signed, Ops[0], *Size);
+            } break;
+            case Instruction::Trunc: {
+                auto Trunc = cast<TruncInst>(&I);
+                if (!Trunc->getDestTy()->isIntegerTy()) {
+                    continue;
+                }
+                TAD.Map[&I] = Ops[0];
+            } break;
+            case Instruction::Add:
+            case Instruction::And:
+            case Instruction::AShr:
+            case Instruction::LShr:
+            case Instruction::Mul:
+            case Instruction::UDiv:
+            case Instruction::SDiv:
+            case Instruction::Or:
+            case Instruction::Shl:
+            case Instruction::Sub:
+            case Instruction::Xor: {
+                auto Bin = cast<BinaryOperator>(&I);
+                if (!isa<IntegerType>(Bin->getType())) {
+                    continue;
+                }
+                assert(Ops[0].Kind == Ops[1].Kind);
+                TAD.Map[&I] = C.binop(Bin->getOpcode(), Ops[0], Ops[1]);
+            } break;
+            case Instruction::ICmp: {
+                auto *ICmp = cast<ICmpInst>(&I);
+                assert(Ops[0].Kind == Ops[1].Kind);
+                TAD.Map[&I] = C.compare(ICmp->getPredicate(), Ops[0], Ops[1]);
+            } break;
+            case Instruction::Call: {
+                auto *Call = cast<CallInst>(&I);
+
+                // Rule out calls which are marked as returning integer
+                // immediates first, then handle specific pseudo instructions.
+
+                Type *RetTy = Call->getType();
+
+                if (auto IntTy = dyn_cast<IntegerType>(RetTy)) {
+                    auto It = AnnotationMap.find(Call->getCalledFunction());
+                    if (It != AnnotationMap.end() and
+                        It->second.isSet(
+                            FunctionAnnotation::ReturnsImmediate)) {
+                        auto Size = ValueSize::fromLlvmType(IntTy);
+                        if (!Size) {
+                            return Size.takeError();
+                        }
+                        auto Tcg = TcgV::makeTemp(*Size, IrImmediate);
+                        Tcg.Name = getDebugVarName(DebugInfo, Call);
+                        Tcg.Kind = IrImmediate;
+                        TAD.Map[Call] = Tcg;
+                        continue;
+                    }
+                }
+
+                switch (getPseudoInstFromCall(Call)) {
+                case IdentityMap: {
+                    TcgV Tcg;
+                    if (auto *IntTy = dyn_cast<IntegerType>(RetTy)) {
+                        auto Size = ValueSize::fromLlvmType(IntTy);
+                        if (!Size) {
+                            return Size.takeError();
+                        }
+                        Tcg = TcgV::makeTemp(*Size, IrValue);
+                    } else if (auto *VecTy = dyn_cast<VectorType>(RetTy)) {
+                        auto Size = VectorSize::fromLlvmType(VecTy);
+                        if (!Size) {
+                            return Size.takeError();
+                        }
+                        Tcg = TcgV::makeVector(*Size);
+                    } else {
+                        abort();
+                    }
+                    Tcg.ConstantExpression = true;
+                    Tcg.Name = Ops[0].Name;
+                    TAD.Map[&I] = Tcg;
+                } break;
+                case PtrAdd: {
+                    TAD.Map[&I] = C.ptrAdd(Ops[0], Ops[1]);
+                } break;
+                case Movcond: {
+                    auto LlvmPred = static_cast<ICmpInst::Predicate>(
+                        cast<ConstantInt>(Call->getOperand(0))->getZExtValue());
+                    const TcgV Cond = C.compare(LlvmPred, Ops[1], Ops[2]);
+                    TAD.Map[&I] = C.ternary(Cond, Ops[3], Ops[4]);
+                } break;
+                case AccessGlobalArray: {
+                    const uint64_t TypeIndex =
+                        cast<ConstantInt>(Call->getArgOperand(0))
+                            ->getZExtValue();
+                    const uint64_t Offset =
+                        cast<ConstantInt>(Call->getArgOperand(1))
+                            ->getZExtValue();
+                    assert(TypeIndex < TcgGlobals.size());
+                    auto It = TcgGlobals[TypeIndex].find(Offset);
+                    assert(It != TcgGlobals[TypeIndex].end());
+                    const TcgGlobal Global = It->second;
+                    if (Ops[2].Kind != IrImmediate) {
+                        return mkError(
+                            "Global array access with non-immediate index");
+                    }
+                    auto Code = Global.Code.str() + "[" + getName(Ops[2]) + "]";
+                    TAD.Map[&I] = TcgV::makeConstantExpression(
+                        Code, *ValueSize::fromBitWidth(Global.Size), IrValue);
+                } break;
+                case AccessGlobalValue: {
+                    const uint64_t TypeIndex =
+                        cast<ConstantInt>(Call->getArgOperand(0))
+                            ->getZExtValue();
+                    const uint64_t Offset =
+                        cast<ConstantInt>(Call->getArgOperand(1))
+                            ->getZExtValue();
+                    assert(TypeIndex < TcgGlobals.size());
+                    auto It = TcgGlobals[TypeIndex].find(Offset);
+                    assert(It != TcgGlobals[TypeIndex].end());
+                    const TcgGlobal Global = It->second;
+                    TAD.Map[&I] = TcgV::makeConstantExpression(
+                        Global.Code.str(),
+                        *ValueSize::fromBitWidth(Global.Size), IrValue);
+                } break;
+                default:
+                    continue;
+                }
+            } break;
+            default:
+                continue;
+            }
+        }
+    }
+    return Error::success();
+}
-- 
2.52.0

[PATCH v2 36/50] helper-to-tcg: TcgGenPass, allocate TCG registers

[Anton Johansson via qemu development]

Based on the assumption of a cycle free IR, this commit adds a simple
register allocator for emitted values in TCG.  The goal of this pass is
to reduce the number of temporaries required in the output code, which
is especially important when dealing with gvec vectors as to not require
huge amounts of temporary storage in CPUArchState.

For each LLVM value in the IR, that has not already been mapped
(arguments, constant expressions), the allocator will assign a `struct TcgV`
reprensenting a variable in the output TCG.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 subprojects/helper-to-tcg/meson.build         |   1 +
 .../src/TcgGenPass/MapTemporaries.cpp         | 619 ++++++++++++++++++
 2 files changed, 620 insertions(+)
 create mode 100644 subprojects/helper-to-tcg/src/TcgGenPass/MapTemporaries.cpp

diff --git a/subprojects/helper-to-tcg/meson.build b/subprojects/helper-to-tcg/meson.build
index 046cdb916b..7ceac955bd 100644
--- a/subprojects/helper-to-tcg/meson.build
+++ b/subprojects/helper-to-tcg/meson.build
@@ -53,6 +53,7 @@ sources = [
     'src/TcgGenPass/TcgEmit.cpp',
     'src/TcgGenPass/MapArguments.cpp',
     'src/TcgGenPass/MapConstantExpressions.cpp',
+    'src/TcgGenPass/MapTemporaries.cpp',
 ]
 
 clang = bindir / 'clang'
diff --git a/subprojects/helper-to-tcg/src/TcgGenPass/MapTemporaries.cpp b/subprojects/helper-to-tcg/src/TcgGenPass/MapTemporaries.cpp
new file mode 100644
index 0000000000..84f90eba8a
--- /dev/null
+++ b/subprojects/helper-to-tcg/src/TcgGenPass/MapTemporaries.cpp
@@ -0,0 +1,619 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#include "DebugInfo.hpp"
+#include "ValueMapping.hpp"
+
+#include "CmdLineOptions.hpp"
+#include "Error.hpp"
+#include "LlvmCompat.hpp"
+#include "PseudoInst.hpp"
+#include "TcgEmit.hpp"
+#include "TcgType.hpp"
+
+#include <llvm/IR/BasicBlock.h>
+#include <llvm/IR/Constants.h>
+#include <llvm/IR/DerivedTypes.h>
+#include <llvm/IR/Function.h>
+#include <llvm/IR/Instructions.h>
+#include <llvm/IR/IntrinsicInst.h>
+#include <llvm/IR/Intrinsics.h>
+#include <llvm/IR/Module.h>
+#include <llvm/Transforms/Utils/Cloning.h>
+
+#define DEBUG_TYPE "map-temporaries"
+
+using namespace llvm;
+
+//
+// TCG Register Allocation Pass
+//
+// Analysis over the IR that performs basic register allocation to assign
+// identifiers representing TCGv's to all non-argument and non-constant values
+// in a given function.
+//
+// Note: Input code is assumed to be loop free, which drastically simplifies
+// the register allocation. This assumption is reasonable as we expect code
+// with loops to be either unrolled or vectorized, and we currently don't emit
+// for loops in C.
+//
+
+// Type to represent a list of free TcgV's that can be reused when
+// we need a new temporary. Exists for the duration of a function,
+// and is expected to be small <= 8 free TcgV's at any time.
+//
+// This justifies the type being an array, since iteration times to
+// find a free element will be small.
+using FreeListVector = SmallVector<TcgV, 8>;
+
+// Finds the first `TcgV` in `FreeList` with a matching `TcgBitWidth` and
+// `Kind`.
+static TcgV findFreeTcgV(FreeListVector &FreeList, TcgSize Size, TcgKind Kind) {
+    for (size_t i = 0; i < FreeList.size(); ++i) {
+        if (Kind != FreeList[i].Kind) {
+            continue;
+        }
+        bool Matches = false;
+        switch (Kind) {
+        case IrPtrToOffset:
+            Matches = Size.Vec.bytes() == FreeList[i].Size.Vec.bytes();
+            break;
+        case IrValue:
+        case IrImmediate:
+            Matches = Size.Val.TcgBitWidth == FreeList[i].Size.Val.TcgBitWidth;
+            break;
+        case IrPtr:
+            Matches = true;
+            break;
+        default:
+            continue;
+        }
+        if (Matches) {
+            TcgV Tcg = FreeList[i];
+            // Swap-remove
+            FreeList[i] = FreeList.back();
+            FreeList.pop_back();
+            return Tcg;
+        }
+    }
+    return {};
+}
+
+static bool valueIsNonImmediateArg(TempAllocationData &TAD, const Value *V) {
+    if (!isa<Argument>(V)) {
+        return false;
+    }
+    // Assume arguments have already been mapped at this point.
+    auto It = TAD.Map.find(V);
+    assert(It != TAD.Map.end());
+    return It->second.Kind != IrImmediate;
+}
+
+//
+// Functions for mapping an LLVM Value to a TcgV
+//
+
+enum MapValueFlags {
+    ForceNewValue = 1,
+};
+
+// Given an integer LLVM value assign it to a TcgV, either by creating a new
+// one or finding a suitable one on the FreeList
+static Expected<TcgV> mapInteger(TempAllocationData &TAD,
+                                 const AnnotationMapTy &Annotations,
+                                 const DebugInfoMapTy &DebugInfo,
+                                 FreeListVector &FreeList, const Value *V,
+                                 uint32_t flags) {
+    auto *Ty = cast<IntegerType>(V->getType());
+    auto Size = ValueSize::fromLlvmType(Ty);
+    if (!Size) {
+        return Size.takeError();
+    }
+
+    StringRef Name = getDebugVarName(DebugInfo, V);
+
+    if ((flags & ForceNewValue) != 0) {
+        auto Tcg = TcgV::makeTemp(*Size, IrValue);
+        Tcg.Name = Name;
+        return TAD.map(V, Tcg);
+    } else {
+        // Non-constant integer
+
+        // TODO: This is rather hacky and should be widened in an earlier pass.
+        if (auto *ICmp = dyn_cast<ICmpInst>(V)) {
+            // `icmp` return `i1`s and are used as either 32-bit or 64-bit TCG
+            // value in QEMU.  Assume the size from operands, otherwise all
+            // returned values are 32 bits in size, causing a mismatch with TCG.
+            assert(Size->LlvmBitWidth == 1);
+            auto *IntTy0 =
+                dyn_cast<IntegerType>(ICmp->getOperand(0)->getType());
+            if (!IntTy0) {
+                return mkError("Icmp on non-integer type");
+            }
+            auto Size0 = ValueSize::fromLlvmType(IntTy0);
+            if (!Size0) {
+                return Size.takeError();
+            }
+            Size->TcgBitWidth = Size0->TcgBitWidth;
+        }
+
+        TcgV Tcg = findFreeTcgV(FreeList, {.Val = *Size}, IrValue);
+        if (Tcg.Kind != IrInvalid) {
+            // Found a `TcgV` of the corresponding TCG size, update LLVM size.
+            Tcg.Size.Val.LlvmBitWidth = Size->LlvmBitWidth;
+            return TAD.map(V, Tcg);
+        } else {
+            // Otherwise, create a new value
+            auto Tcg = TcgV::makeTemp(*Size, IrValue);
+            Tcg.Name = Name;
+            return TAD.map(V, Tcg);
+        }
+    }
+}
+
+// Given an vector LLVM value assign it to a `TcgV`, either by creating a new
+// one or finding a suitable one on the `FreeList`.  Special care is taken to
+// map individual elements of constant vectors.
+static Expected<TcgV> mapVector(TempAllocationData &TAD,
+                                const AnnotationMapTy &Annotations,
+                                const DebugInfoMapTy &DebugInfo,
+                                FreeListVector &FreeList, const Value *V,
+                                VectorType *VecTy, uint32_t flags) {
+    auto Size = VectorSize::fromLlvmType(VecTy);
+    if (!Size) {
+        return Size.takeError();
+    }
+
+    StringRef Name = getDebugVarName(DebugInfo, V);
+
+    // TODO: Hacky, avoids comparison results being `<N x i1>` which doesn't map
+    // nicely to TCG, obtain size from argument instead.
+    if (auto *ICmp = dyn_cast<ICmpInst>(V)) {
+        auto *VecTy = cast<VectorType>(ICmp->getOperand(0)->getType());
+        Size = VectorSize::fromLlvmType(VecTy);
+        if (!Size) {
+            return Size.takeError();
+        }
+    }
+
+    // Create or find a `TcgV`
+    TcgV Tcg = findFreeTcgV(FreeList, {.Vec = *Size}, IrPtrToOffset);
+    if (Tcg.Kind != IrInvalid) {
+        Tcg.Size.Vec = *Size;
+    } else {
+        Tcg = TcgV::makeVector(*Size);
+        Tcg.Name = Name;
+    }
+
+    return TAD.map(V, Tcg);
+}
+
+// Given an pointer LLVM value assign it to a TcgV, either by creating a new
+// one or finding a suitable one on the FreeList.
+static Expected<TcgV> mapPointer(TempAllocationData &TAD,
+                                 const AnnotationMapTy &Annotations,
+                                 const DebugInfoMapTy &DebugInfo,
+                                 FreeListVector &FreeList, const Value *V,
+                                 uint32_t flags) {
+    // If the value has an associated name from the debug information, use it
+    StringRef Name{};
+    if (auto It = DebugInfo.find(V); It != DebugInfo.end()) {
+        Name = It->second.VarName;
+    }
+
+    if (auto *Alloca = dyn_cast<AllocaInst>(V)) {
+        // `alloca`s represent stack variables in LLVM IR and return
+        // pointers, we can simply map them to `IrValue`s
+        auto *IntTy = dyn_cast<IntegerType>(Alloca->getAllocatedType());
+        if (!IntTy) {
+            return mkError("alloca with unsupported type: ", V);
+        }
+
+        auto Size = ValueSize::fromLlvmType(IntTy);
+        if (!Size) {
+            return Size.takeError();
+        }
+
+        // find or create a new `IrValue`
+        TcgV Tcg = findFreeTcgV(FreeList, {.Val = *Size}, IrValue);
+        if (Tcg.Kind != IrInvalid) {
+            return TAD.map(V, Tcg);
+        } else {
+            auto Tcg = TcgV::makeTemp(*Size, IrValue);
+            Tcg.Name = Name;
+            return TAD.map(V, Tcg);
+        }
+    } else {
+        // Otherwise, find or create a new IrPtr of the target pointer size
+        TcgV Tcg = findFreeTcgV(FreeList, {}, IrPtr);
+        if (Tcg.Kind != IrInvalid) {
+            return TAD.map(V, Tcg);
+        } else {
+            auto Tcg = TcgV::makeTemp({}, IrPtr);
+            Tcg.Name = Name;
+            return TAD.map(V, Tcg);
+        }
+    }
+
+    return mkError("Unable to map constant ", V);
+}
+
+// Given a LLVM value, assigns a TcgV by type (integer, pointer, vector).  If
+// the given value has already been mapped to a TcgV, return it.
+static Expected<TcgV> mapValue(TempAllocationData &TAD,
+                               const AnnotationMapTy &Annotations,
+                               const DebugInfoMapTy &DebugInfo,
+                               FreeListVector &FreeList, const Value *V,
+                               uint32_t flags = 0) {
+    // Return previously mapped value
+    auto It = TAD.Map.find(V);
+    if (It != TAD.Map.end()) {
+        return It->second;
+    }
+
+    if (isa<UndefValue>(V)) {
+        return mkError("Unable to map undefined value: ", V);
+    }
+
+    Type *Ty = V->getType();
+    if (isa<IntegerType>(Ty)) {
+        return mapInteger(TAD, Annotations, DebugInfo, FreeList, V, flags);
+    } else if (isa<PointerType>(Ty)) {
+        return mapPointer(TAD, Annotations, DebugInfo, FreeList, V, flags);
+    } else if (isa<VectorType>(Ty)) {
+        return mapVector(TAD, Annotations, DebugInfo, FreeList, V,
+                         cast<VectorType>(Ty), flags);
+    }
+
+    return mkError("Unable to map value ", V);
+}
+
+static bool shouldSkipInstruction(const Instruction *const I,
+                                  bool SkipReturnMov) {
+    // Skip returns if we're skipping return mov's
+    if (isa<ReturnInst>(I) and SkipReturnMov) {
+        return true;
+    }
+    // Skip assertions
+    auto Call = dyn_cast<CallInst>(I);
+    if (!Call) {
+        return false;
+    }
+    Function *F = Call->getCalledFunction();
+    if (!F) {
+        return false;
+    }
+    StringRef Name = F->getName();
+    return (Name == "__assert_fail" or Name == "g_assertion_message_expr" or
+            isa<DbgValueInst>(I) or isa<DbgLabelInst>(I) or
+            isa<DbgDeclareInst>(I));
+}
+
+static bool shouldSkipValue(const Value *const V) {
+    return (isa<GlobalValue>(V) or isa<ConstantExpr>(V) or isa<BasicBlock>(V));
+}
+
+// A mapping of the return TCG variable to the value `RetV` is valid
+// if no use of argument occurs beetween `RetV`s definition and use, this is to
+// avoid clobbers.  Iteration may start in the middle of a basic block.
+static bool isRetMapValid(TempAllocationData &TAD,
+                          LinearBlocks::const_reverse_iterator BeginBB,
+                          LinearBlocks::const_reverse_iterator EndBB,
+                          BasicBlock::const_reverse_iterator BeginInst,
+                          BasicBlock::const_reverse_iterator EndInst,
+                          const Value *RetV) {
+    auto ItBB = BeginBB;
+    auto ItInst = BeginInst;
+
+    do {
+        do {
+            const Instruction &I = *ItInst;
+            // Check if we found definition of `RetV`.
+            if (cast<Value>(&I) == RetV) {
+                return true;
+            }
+            // Check for use of non-immediate arguments.
+            for (auto &V : getOperands(&I)) {
+                if (valueIsNonImmediateArg(TAD, V)) {
+                    return false;
+                }
+            }
+        } while (++ItInst != EndInst);
+
+        if (++ItBB != EndBB) {
+            EndInst = (*ItBB)->rend();
+            ItInst = (*ItBB)->rbegin();
+        }
+    } while (ItBB != EndBB);
+
+    return false;
+}
+
+static bool valueIsReference(const Value *V) {
+    if (auto *Call = dyn_cast<CallInst>(V)) {
+        auto *F = Call->getCalledFunction();
+        if (AllowDeclCall && F->isDeclaration()) {
+            return true;
+        }
+    }
+    return false;
+}
+
+static bool instructionClobbersArguments(const Instruction *I) {
+    if (auto *Call = dyn_cast<CallInst>(I)) {
+        auto *F = Call->getCalledFunction();
+        if (AllowDeclCall && F->isDeclaration() and
+            !compat::isFunctionQemuHelper(F->getName())) {
+            return true;
+        } else if (F->isIntrinsic() and
+                   F->getIntrinsicID() == Intrinsic::usub_sat) {
+            // TODO: usub_sat maps to tcg_gen_ussub_*() which clobbers
+            // arguments, if this changes on QEMUs end get rid of this.
+            return true;
+        }
+    }
+    return false;
+}
+
+static void removeDefinedVariable(TempAllocationData &TAD,
+                                  FreeListVector &FreeList,
+                                  const Instruction *I) {
+    auto It = TAD.Map.find(cast<Value>(I));
+    if (!isa<Argument>(I) and It != TAD.Map.end() and
+        !cast<Value>(I)->getType()->isVoidTy()) {
+        TcgV &Tcg = It->second;
+        switch (Tcg.Kind) {
+        case IrValue:
+        case IrPtr:
+        case IrPtrToOffset:
+            FreeList.push_back(Tcg);
+            break;
+        case IrImmediate:
+            break;
+        default:
+            abort();
+        }
+    }
+}
+
+Error allocateTemporaries(const Function &F, const LinearBlocks &Blocks,
+                          const AnnotationMapTy &Annotations,
+                          const DebugInfoMapTy &DebugInfo, CEmitter &C,
+                          const TcgGlobalMap &TcgGlobals,
+                          TempAllocationData &TAD) {
+    FreeListVector FreeList;
+
+    LLVM_DEBUG(dbgs() << "Allocating temporaries for " << F.getName() << "\n");
+
+    // The PrepareForOptPass removes all functions with non-int/void return
+    // types, assert this assumption.
+    Type *RetTy = F.getReturnType();
+    assert(isa<IntegerType>(RetTy) or RetTy->isVoidTy());
+    // Map integer return values
+    if (auto IntTy = dyn_cast<IntegerType>(RetTy)) {
+        auto Size = ValueSize::fromLlvmType(IntTy);
+        if (!Size) {
+            return Size.takeError();
+        }
+        TAD.flags |= ReturnsValue;
+        TAD.ReturnValue = TcgV::makeTemp(*Size, IrValue);
+    }
+
+    // Skip mov's to return value if possible, results of previous
+    // instructions might have been assigned the return value.
+    //
+    // This is possible if:
+    //   1. The return value is not an argument.
+    //   2. The return value is not a constant.
+    //   3. No use of an argument has occured after the definition of the
+    //      value being returned.
+    {
+        auto Begin = Blocks.rbegin();
+        auto End = Blocks.rend();
+        const Instruction &I = *(*Begin)->rbegin();
+
+        auto Ret = dyn_cast<ReturnInst>(&I);
+        if (Ret and Ret->getNumOperands() == 1) {
+            Value *RetV = Ret->getReturnValue();
+            bool ValidRetV = !isa<Argument>(RetV) and !isa<ConstantInt>(RetV);
+            bool ValidMap = isRetMapValid(TAD, Begin, End, (*Begin)->rbegin(),
+                                          (*Begin)->rend(), RetV);
+            if (ValidRetV and ValidMap) {
+                assert(TAD.hasReturnValue());
+                TAD.Map.try_emplace(RetV, TAD.ReturnValue);
+                TAD.flags |= SkipReturnMov;
+            }
+        }
+    }
+
+    // Iterate over instructions in reverse and try to allocate TCG
+    // variables.
+    //
+    // The algorithm is very straight forward, we keep a FreeList of TCG
+    // variables we can reuse.  Variables are allocated on first use and
+    // "freed" on definition.
+    //
+    // We allow reuse of the return TCG variable in order to save one
+    // variable and skip the return mov if possible.  Since source and
+    // return variables can overlap, when take the conservative route and
+    // only allow reuse of the return variable if no arguments have been
+    // used.
+
+    bool SeenArgUse = false;
+
+    for (auto ItBB = Blocks.rbegin(), ItBBEnd = Blocks.rend(); ItBB != ItBBEnd;
+         ++ItBB) {
+        const BasicBlock *BB = *ItBB;
+        // Loop over instructions in the basic block in reverse
+        for (auto ItInst = BB->rbegin(), ItInstEnd = BB->rend();
+             ItInst != ItInstEnd; ++ItInst) {
+            const Instruction &I = *ItInst;
+            if (shouldSkipInstruction(&I, TAD.flags & SkipReturnMov)) {
+                continue;
+            }
+
+            {
+                auto It = TAD.Map.find(&I);
+                if (It != TAD.Map.end() and (It->second.Kind == IrImmediate or
+                                             It->second.ConstantExpression)) {
+                    continue;
+                }
+            }
+
+            LLVM_DEBUG(dbgs() << "  For: " << I << "\n");
+
+            // For calls to the identity mapping pseudo instruction
+            // we simply want to propagate the type allocated for the result
+            // of the call to the operand.
+            if (isa<CallInst>(&I)) {
+                auto *Call = cast<CallInst>(&I);
+                PseudoInst Inst = getPseudoInstFromCall(Call);
+                if (Inst == IdentityMap) {
+                    Value *Arg = Call->getArgOperand(0);
+                    auto It = TAD.Map.find(cast<Value>(&I));
+                    assert(It != TAD.Map.end());
+                    const TcgV Original = It->second;
+                    LLVM_DEBUG(dbgs() << "    Identity mapping");
+                    if (TAD.Map.find(Arg) != TAD.Map.end()) {
+                        LLVM_DEBUG(dbgs() << " (forward)\n");
+                        // Propagate forward
+                        Expected<TcgV> Tcg = mapValue(TAD, Annotations,
+                                                      DebugInfo, FreeList, Arg);
+                        assert(Tcg);
+
+                        TcgV Propagated = Tcg.get();
+                        // If we're identity mapped to an argument we might
+                        // lose vector information if we propagate forward
+                        // blindly, ensure vector type remains, likewise for
+                        // scalar we might bit width.
+                        if (Original.Kind == IrPtrToOffset) {
+                            Propagated.Kind = IrPtrToOffset;
+                            Propagated.Size = Original.Size;
+                        } else {
+                            Propagated.Size.Val.LlvmBitWidth =
+                                Original.Size.Val.LlvmBitWidth;
+                        }
+                        TAD.Map[cast<Value>(&I)] = Propagated;
+
+                        LLVM_DEBUG({
+                            dbgs() << "Original:\n";
+                            Original.dump(dbgs());
+                            dbgs() << "Arg:\n";
+                            Tcg->dump(dbgs());
+                            dbgs() << "Ret:\n";
+                            Propagated.dump(dbgs());
+                        });
+                    } else {
+                        LLVM_DEBUG(dbgs() << " (forward)\n");
+                        // Propagate back
+                        TcgV Propagated = It->second;
+                        // TODO:
+                        if (auto *IntTy =
+                                dyn_cast<IntegerType>(Arg->getType())) {
+                            auto NewSize = ValueSize::fromLlvmType(IntTy);
+                            assert(NewSize);
+                            Propagated.Size.Val.LlvmBitWidth =
+                                Original.Size.Val.LlvmBitWidth;
+                        }
+                        TAD.Map[Arg] = Propagated;
+
+                        LLVM_DEBUG({
+                            dbgs() << "Original:\n";
+                            Original.dump(dbgs());
+                            dbgs() << "Ret:\n";
+                            Propagated.dump(dbgs());
+                        });
+                    }
+
+                    continue;
+                }
+            }
+
+            // Check if we've encountered any non-immediate argument yet
+            for (const Use &U : getOperands(&I)) {
+                if (valueIsNonImmediateArg(TAD, U)) {
+                    SeenArgUse = true;
+                }
+            }
+
+            // Free up variables as they are defined, iteration is in post
+            // order meaning uses of vars always occur before definitions.
+            bool AllowDestReuse = !instructionClobbersArguments(&I);
+            if (AllowDestReuse) {
+                removeDefinedVariable(TAD, FreeList, &I);
+            }
+
+            // Loop over operands and assign TcgV's. On first encounter of a
+            // given operand we assign a new TcgV from the FreeList.
+            for (const Use &V : getOperands(&I)) {
+                auto It = TAD.Map.find(V);
+                if (It != TAD.Map.end() or shouldSkipValue(V)) {
+                    continue;
+                }
+
+                uint32_t Flags = 0;
+                if (valueIsReference(V)) {
+                    Flags |= ForceNewValue;
+                }
+
+                Expected<TcgV> Tcg =
+                    mapValue(TAD, Annotations, DebugInfo, FreeList, V, Flags);
+                if (!Tcg) {
+                    return Tcg.takeError();
+                }
+
+                LLVM_DEBUG({
+                    dbgs() << "    Arg: " << *V << "\n";
+                    Tcg->dump(dbgs());
+                });
+
+                // If our value V got mapped to the return value,
+                // make sure the mapping is valid
+                //
+                // A mapping to the return value is valid as long as
+                // an argument has not been used.  This is to prevent
+                // clobbering in the case that arguments and the return
+                // value overlap.
+                if (TAD.hasReturnValue() and *Tcg == TAD.ReturnValue) {
+                    bool Valid =
+                        isRetMapValid(TAD, ItBB, ItBBEnd, ItInst, ItInstEnd, V);
+                    if (!SeenArgUse and Valid) {
+                        continue;
+                    }
+
+                    // The mapping was not valid, erase it and assign a new
+                    // one, this takes the return `TcgV` of out the `freelist`
+                    // pool.
+                    TAD.Map.erase(V);
+                    Expected<TcgV> Tcg = mapValue(TAD, Annotations, DebugInfo,
+                                                  FreeList, V, Flags);
+                    if (!Tcg) {
+                        return Tcg.takeError();
+                    }
+                }
+            }
+
+            // Free up variables as they are defined, iteration is in post
+            // order meaning uses of vars always occur before definitions.
+            if (!AllowDestReuse) {
+                removeDefinedVariable(TAD, FreeList, &I);
+            }
+        }
+    }
+
+    return Error::success();
+}
-- 
2.52.0

[PATCH v2 37/50] helper-to-tcg: TcgGenPass, emit TCG strings

[Anton Johansson via qemu development]

Forward pass over the IR which uses previosly mapped LLVM values to emit
the corresponding operations in TCG.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 subprojects/helper-to-tcg/meson.build         |   1 +
 .../src/TcgGenPass/MapTcgOperations.cpp       | 988 ++++++++++++++++++
 2 files changed, 989 insertions(+)
 create mode 100644 subprojects/helper-to-tcg/src/TcgGenPass/MapTcgOperations.cpp

diff --git a/subprojects/helper-to-tcg/meson.build b/subprojects/helper-to-tcg/meson.build
index 7ceac955bd..413d7ad8fa 100644
--- a/subprojects/helper-to-tcg/meson.build
+++ b/subprojects/helper-to-tcg/meson.build
@@ -54,6 +54,7 @@ sources = [
     'src/TcgGenPass/MapArguments.cpp',
     'src/TcgGenPass/MapConstantExpressions.cpp',
     'src/TcgGenPass/MapTemporaries.cpp',
+    'src/TcgGenPass/MapTcgOperations.cpp',
 ]
 
 clang = bindir / 'clang'
diff --git a/subprojects/helper-to-tcg/src/TcgGenPass/MapTcgOperations.cpp b/subprojects/helper-to-tcg/src/TcgGenPass/MapTcgOperations.cpp
new file mode 100644
index 0000000000..57cdad2624
--- /dev/null
+++ b/subprojects/helper-to-tcg/src/TcgGenPass/MapTcgOperations.cpp
@@ -0,0 +1,988 @@
+//
+//  Copyright(c) 2026 rev.ng Labs Srl. All Rights Reserved.
+//
+//  This program is free software; you can redistribute it and/or modify
+//  it under the terms of the GNU General Public License as published by
+//  the Free Software Foundation; either version 2 of the License, or
+//  (at your option) any later version.
+//
+//  This program is distributed in the hope that it will be useful,
+//  but WITHOUT ANY WARRANTY; without even the implied warranty of
+//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+//  GNU General Public License for more details.
+//
+//  You should have received a copy of the GNU General Public License
+//  along with this program; if not, see <http://www.gnu.org/licenses/>.
+//
+
+#include "ValueMapping.hpp"
+
+#include "CmdLineOptions.hpp"
+#include "Error.hpp"
+#include "FunctionAnnotation.hpp"
+#include "LlvmCompat.hpp"
+#include "PseudoInst.hpp"
+#include "TcgEmit.hpp"
+#include "TcgType.hpp"
+#include "ValueMapping.hpp"
+
+#include <llvm/ADT/PostOrderIterator.h>
+#include <llvm/ADT/SmallBitVector.h>
+#include <llvm/ADT/StringRef.h>
+#include <llvm/ADT/StringSet.h>
+#include <llvm/Analysis/CallGraph.h>
+#include <llvm/IR/BasicBlock.h>
+#include <llvm/IR/Constants.h>
+#include <llvm/IR/DerivedTypes.h>
+#include <llvm/IR/Function.h>
+#include <llvm/IR/GlobalValue.h>
+#include <llvm/IR/GlobalVariable.h>
+#include <llvm/IR/Instructions.h>
+#include <llvm/IR/IntrinsicInst.h>
+#include <llvm/IR/Intrinsics.h>
+#include <llvm/IR/Module.h>
+#include <llvm/Support/Debug.h>
+#include <llvm/Support/FormatVariadic.h>
+#include <llvm/Support/raw_ostream.h>
+
+#define DEBUG_TYPE "map-tcg-ops"
+
+//
+// Map TCG Operations
+//
+// Forward pass over the IR, emitting expressions to TCG using the provided
+// `TempAllocationData` value mapping.
+//
+
+using namespace llvm;
+
+// Wrapper class around a TcgV to cast it to/from 32-/64-bit
+class TcgSizeAdapter {
+    TcgEmitter &TE;
+    const TcgV Orig;
+    TcgV Adapted;
+
+  public:
+    TcgSizeAdapter(TcgEmitter &TE, const TcgV Orig) : TE(TE), Orig(Orig) {}
+
+    const TcgV get(ValueSize Size) {
+        const ValueSize OrigSize = Orig.intSize();
+        if (Orig.Kind == IrImmediate or
+            (OrigSize.TcgBitWidth == Size.TcgBitWidth)) {
+            return Orig;
+        } else if (Adapted.Kind == IrInvalid) {
+            initAdapted(Size.TcgBitWidth);
+        }
+        return Adapted;
+    }
+
+  private:
+    void initAdapted(AllowedTcgSize Size) {
+        assert(Adapted.Kind == IrInvalid);
+        const ValueSize OrigSize = Orig.intSize();
+        assert(OrigSize.TcgBitWidth != Size);
+        Adapted = TcgV::makeTemp({Size, OrigSize.LlvmBitWidth}, Orig.Kind);
+        TE.defineNewTemp(Adapted);
+        if (Size == 32) {
+            TE.genExtrlI64I32(Adapted, Orig);
+        } else {
+            TE.genExtuI32I64(Adapted, Orig);
+        }
+    }
+};
+
+enum class LabelKind : uint8_t {
+    Tcg,
+    If,
+    Else,
+    Merge,
+};
+
+struct LabelInfo {
+    LabelKind Kind;
+    TcgV Label;
+};
+
+class Mapper {
+    TcgEmitter &TE;
+    const TempAllocationData &TAD;
+
+    llvm::DenseMap<const BasicBlock *, LabelInfo> Labels;
+    // Keep track of whether a TcgV has been defined already, or not
+    SmallBitVector HasBeenDefined;
+
+  public:
+    Mapper(TcgEmitter &TE, const TempAllocationData &TAD) : TE(TE), TAD(TAD) {
+        // Default to size of previously mapped TcgVs
+        HasBeenDefined.resize(TAD.Map.size());
+    }
+
+    bool hasBeenDefined(const TcgV &V) { return HasBeenDefined[V.Id]; }
+
+    void define(const TcgV &V) { HasBeenDefined.set(V.Id); }
+
+    LabelInfo mapBbAndEmit(BasicBlock *BB, LabelKind Kind = LabelKind::Tcg) {
+        auto It = Labels.find(BB);
+        if (It == Labels.end()) {
+            TcgV Label = TcgV::makeLabel();
+            if (Kind == LabelKind::Tcg) {
+                TE.defineNewTemp(Label);
+            }
+            return Labels.try_emplace(BB, LabelInfo{Kind, Label}).first->second;
+        }
+        return It->second;
+    }
+
+    TcgV mapTcgLabel(BasicBlock *BB) {
+        const LabelInfo LI = mapBbAndEmit(BB);
+        assert(LI.Kind == LabelKind::Tcg);
+        return LI.Label;
+    }
+
+    const TcgV defineValue(const Value *V, bool ForceNondef = false) {
+        // `Value` should have already been mapped by previous passes.
+        auto It = TAD.Map.find(V);
+        assert(It != TAD.Map.end());
+        const TcgV Tcg = It->second;
+
+        // Using that `TcgV` ids are sequential for a function and start
+        // from 0.
+        // TODO: Might be nice to just track a maximum from previous passes.
+        if (Tcg.Id >= HasBeenDefined.size()) {
+            HasBeenDefined.resize(Tcg.Id + 1);
+        }
+        if (!HasBeenDefined[Tcg.Id]) {
+            if (!isa<Argument>(V) and
+                (!TAD.hasReturnValue() or Tcg != TAD.ReturnValue) and
+                !Tcg.ConstantExpression and Tcg.Kind != IrImmediate) {
+                if (!ForceNondef) {
+                    HasBeenDefined.set(Tcg.Id);
+                    TE.defineNewTemp(Tcg);
+                }
+            }
+        }
+
+        return Tcg;
+    }
+};
+
+static void ensureSignBitIsSet(TcgEmitter &TE, const TcgV &Tcg) {
+    const ValueSize Size = Tcg.intSize();
+    if (Tcg.llvmBitWidth() == Tcg.tcgBitWidth() or Tcg.Kind != IrValue) {
+        return;
+    }
+    TE.genExtract(
+        true, Tcg, Tcg, TcgV::makeImmediate("0", Size),
+        TcgV::makeImmediate(Twine((int)Size.LlvmBitWidth).str(), Size));
+}
+
+static const TcgV mapCallReturnValue(Mapper &Mapper, CallInst *Call,
+                                     bool CallToDecl = false) {
+    // Only map return value if it has > 0 uses.  Destination values of call
+    // instructions are the only ones which LLVM will not remove if unused.
+    if (Call->getType()->isVoidTy() or Call->getNumUses() == 0) {
+        return {};
+    }
+    return Mapper.defineValue(Call, CallToDecl);
+}
+
+static Instruction::BinaryOps mapPseudoInstToOpcode(PseudoInst Inst) {
+    switch (Inst) {
+    case VecAddScalar:
+    case VecAddStore:
+    case VecAddScalarStore:
+        return Instruction::Add;
+    case VecSubScalar:
+    case VecSubStore:
+    case VecSubScalarStore:
+        return Instruction::Sub;
+    case VecMulScalar:
+    case VecMulStore:
+    case VecMulScalarStore:
+        return Instruction::Mul;
+    case VecXorScalar:
+    case VecXorStore:
+    case VecXorScalarStore:
+        return Instruction::Xor;
+    case VecOrScalar:
+    case VecOrStore:
+    case VecOrScalarStore:
+        return Instruction::Or;
+    case VecAndScalar:
+    case VecAndStore:
+    case VecAndScalarStore:
+        return Instruction::And;
+    case VecShlScalar:
+    case VecShlStore:
+    case VecShlScalarStore:
+        return Instruction::Shl;
+    case VecLShrScalar:
+    case VecLShrStore:
+    case VecLShrScalarStore:
+        return Instruction::LShr;
+    case VecAShrScalar:
+    case VecAShrStore:
+    case VecAShrScalarStore:
+        return Instruction::AShr;
+    default:
+        abort();
+    }
+}
+
+static bool translatePseudoInstCall(CEmitter &CE, TcgEmitter &TE,
+                                    CallInst *Call, PseudoInst PInst,
+                                    Mapper &Mapper,
+                                    const TcgGlobalMap &TcgGlobals,
+                                    const TcgV &Ret,
+                                    ArrayRef<TcgV> Args) {
+    switch (PInst) {
+    case IdentityMap: {
+        // Nothing to do
+    } break;
+    case GetPC: {
+        TE.genCallHelper("helper_getpc", {Ret});
+    } break;
+    case PtrAdd: {
+        assert(Args[0].Kind == IrPtr or Args[0].Kind == IrPtrToOffset);
+        if (Args[0].Kind == IrPtr) {
+            TE.genAddPtr(Ret, Args[0], Args[1]);
+        } else {
+            assert(Args[1].Kind == IrValue);
+            TE.genValueToPtr(Ret, Args[1]);
+            TE.genAddPtr(Ret, Ret, Args[0]);
+            TE.genAddPtr(Ret, Ret, TE.getGlobalEnv());
+        }
+    } break;
+    case Brcond: {
+        bool Fallthrough =
+            cast<ConstantInt>(Call->getOperand(0))->getZExtValue();
+        auto LlvmPred = static_cast<ICmpInst::Predicate>(
+            cast<ConstantInt>(Call->getOperand(1))->getZExtValue());
+        TE.genBrcond(LlvmPred, Args[2], Args[3], Args[4]);
+        if (!Fallthrough) {
+            TE.genBr(Args[5]);
+        }
+    } break;
+    case Movcond: {
+        auto LlvmPred = static_cast<ICmpInst::Predicate>(
+            cast<ConstantInt>(Call->getOperand(0))->getZExtValue());
+        if (CmpInst::isSigned(LlvmPred)) {
+            // Since comprasions are made on TCG registers which contains
+            // smaller logical LLVM values, make sure to correctly sign extend
+            // smaller values for comparisons.
+            ensureSignBitIsSet(TE, Args[1]);
+            ensureSignBitIsSet(TE, Args[2]);
+        }
+        TE.genMovcond(LlvmPred, Ret, Args[1], Args[2], Args[3], Args[4]);
+    } break;
+    case VecSplat: {
+        TE.genVecSplat(Ret, Args[0]);
+    } break;
+    case VecConstant: {
+        TE.genVecArrSplat(Ret, Args[0]);
+    } break;
+    case VecNot: {
+        TE.genVecNot(Ret, Args[0]);
+    } break;
+    case VecNotStore: {
+        TE.genVecNot(Args[0], Args[1]);
+    } break;
+    case VecAddScalar:
+    case VecSubScalar:
+    case VecMulScalar:
+    case VecXorScalar:
+    case VecOrScalar:
+    case VecAndScalar:
+    case VecShlScalar:
+    case VecLShrScalar:
+    case VecAShrScalar: {
+        auto Opcode = mapPseudoInstToOpcode(PInst);
+        TE.genVecBinOp(Opcode, Ret, Args[0], Args[1]);
+    } break;
+    case VecAddStore:
+    case VecSubStore:
+    case VecMulStore:
+    case VecXorStore:
+    case VecOrStore:
+    case VecAndStore:
+    case VecShlStore:
+    case VecLShrStore:
+    case VecAShrStore:
+    case VecAddScalarStore:
+    case VecSubScalarStore:
+    case VecMulScalarStore:
+    case VecXorScalarStore:
+    case VecOrScalarStore:
+    case VecAndScalarStore:
+    case VecShlScalarStore:
+    case VecLShrScalarStore:
+    case VecAShrScalarStore: {
+        auto Opcode = mapPseudoInstToOpcode(PInst);
+        TE.genVecBinOp(Opcode, Args[0], Args[1], Args[2]);
+    } break;
+    case VecSignedSatAddStore: {
+        TE.genVecSignedSatAdd(Args[0], Args[1], Args[2]);
+    } break;
+    case VecSignedSatSubStore: {
+        TE.genVecSignedSatSub(Args[0], Args[1], Args[2]);
+    } break;
+    case VecSelectStore: {
+        TE.genVecBitsel(Args[0], Args[1], Args[2], Args[3]);
+    } break;
+    case VecAbsStore: {
+        TE.genAbs(Args[0], Args[1]);
+    } break;
+    case VecSignedMaxStore: {
+        TE.genVecSignedMax(Args[0], Args[1], Args[2]);
+    } break;
+    case VecUnsignedMaxStore: {
+        TE.genVecUnsignedMax(Args[0], Args[1], Args[2]);
+    } break;
+    case VecSignedMinStore: {
+        TE.genVecSignedMin(Args[0], Args[1], Args[2]);
+    } break;
+    case VecUnsignedMinStore: {
+        TE.genVecUnsignedMin(Args[0], Args[1], Args[2]);
+    } break;
+    case VecTruncStore: {
+        uint8_t DstElementBits =
+            cast<ConstantInt>(Call->getOperand(0))->getZExtValue();
+        TE.genVecTrunc(DstElementBits, Args[1], Args[2]);
+    } break;
+    case VecCompare: {
+        auto LlvmPred = static_cast<ICmpInst::Predicate>(
+            cast<ConstantInt>(Call->getOperand(0))->getZExtValue());
+        TE.genVecCmp(Ret, LlvmPred, Args[1], Args[2]);
+    } break;
+    case VecWideCondBitsel: {
+        TE.genVecBitsel(Ret, Args[0], Args[1], Args[2]);
+        break;
+    } break;
+    case VecWideCondBitselStore: {
+        TE.genVecBitsel(Args[0], Args[1], Args[2], Args[3]);
+        break;
+    } break;
+    case GuestLoad: {
+        uint8_t Sign = cast<ConstantInt>(Call->getOperand(1))->getZExtValue();
+        uint8_t Size = cast<ConstantInt>(Call->getOperand(2))->getZExtValue();
+        uint8_t Endianness =
+            cast<ConstantInt>(Call->getOperand(3))->getZExtValue();
+        TE.genGuestLoad(Ret, TE.materialize(Args[0]),
+                        TE.getMemOp(Size, Endianness, Sign));
+    } break;
+    case GuestStore: {
+        uint8_t Size = cast<ConstantInt>(Call->getOperand(2))->getZExtValue();
+        uint8_t Endianness =
+            cast<ConstantInt>(Call->getOperand(3))->getZExtValue();
+        TE.genGuestStore(Args[0], Args[1], TE.getMemOp(Size, Endianness, 0));
+    } break;
+    case Exception: {
+        // Map and adapt arguments to the call
+        SmallVector<TcgV, 8> IArgs;
+        for (auto Arg : Args) {
+            IArgs.push_back(TE.materialize(Arg));
+        }
+        TE.genCallHelper("helper_raise_exception", IArgs.begin(), IArgs.end());
+    } break;
+    default:
+        // unmapped pseudo inst
+        return false;
+    }
+    return true;
+}
+
+static bool translateIntrinsicCall(TcgEmitter &TE, CallInst *Call, Function *F,
+                                   const TcgV &Ret, ArrayRef<TcgV> Args,
+                                   Mapper &Mapper) {
+    switch (F->getIntrinsicID()) {
+    case Intrinsic::abs: {
+        TE.genAbs(Ret, Args[0]);
+    } break;
+    case Intrinsic::smax: {
+        TE.genVecSignedMax(Ret, Args[0], Args[1]);
+    } break;
+    case Intrinsic::smin: {
+        TE.genVecSignedMin(Ret, Args[0], Args[1]);
+    } break;
+    case Intrinsic::umax: {
+        TE.genVecUnsignedMax(Ret, Args[0], Args[1]);
+    } break;
+    case Intrinsic::umin: {
+        TE.genVecUnsignedMin(Ret, Args[0], Args[1]);
+    } break;
+    case Intrinsic::sadd_sat: {
+        TE.genVecSignedSatAdd(Ret, Args[0], Args[1]);
+    } break;
+    case Intrinsic::ssub_sat: {
+        TE.genVecSignedSatSub(Ret, Args[0], Args[1]);
+    } break;
+    case Intrinsic::usub_sat: {
+        if (Args[0].Kind == IrPtrToOffset) {
+            TE.genVecUnsignedSatSub(Ret, Args[0], Args[1]);
+        } else {
+            TE.genUnsignedSatSub(Ret, Args[0], Args[1]);
+        }
+    } break;
+    case Intrinsic::ctlz: {
+        if (Args[0].Kind == IrPtrToOffset) {
+            // no gvec equivalent to clzi
+            return false;
+        }
+        TE.genCountLeadingZeros(Ret, Args[0]);
+    } break;
+    case Intrinsic::cttz: {
+        if (Args[0].Kind == IrPtrToOffset) {
+            // no gvec equivalent to ctti
+            return false;
+        }
+        TE.genCountTrailingZeros(Ret, Args[0]);
+    } break;
+    case Intrinsic::ctpop: {
+        if (Args[0].Kind == IrPtrToOffset) {
+            // no gvec equivalent to ctpop
+            return false;
+        }
+        TE.genCountOnes(Ret, Args[0]);
+    } break;
+    case Intrinsic::bswap: {
+        TE.genByteswap(Ret, Args[0]);
+    } break;
+    case Intrinsic::fshl: {
+        TE.genFunnelShl(Ret, Args[0], Args[1], Args[2]);
+    } break;
+    case Intrinsic::bitreverse: {
+        TE.genBitreverse(Ret, Args[0]);
+    } break;
+    case Intrinsic::memcpy: {
+        TE.genVecMemcpy(Args[0], Args[1], Args[2]);
+    } break;
+    case Intrinsic::memset: {
+        TE.genVecMemset(Args[0], Args[1], Args[2]);
+    } break;
+    default:
+        // Unhandled LLVM intrinsic
+        return false;
+    }
+    return true;
+}
+
+static Error
+translateCall(const TcgGlobalMap &TcgGlobals,
+              const AnnotationMapTy &AnnotationMap,
+              const SmallPtrSet<Function *, 16> &HasTranslatedFunction,
+              TcgEmitter &TE, CEmitter &CE, Mapper &Mapper, CallInst *Call) {
+    Function *F = Call->getCalledFunction();
+    if (!F) {
+        return mkError("Indirect function calls not handled: ", Call);
+    }
+
+    assert(F->hasName());
+    StringRef Name{F->getName()};
+
+    // Filter out calls we don't care about. Note we don't have to manually deal
+    // with debug instructions after LLVM 18.
+    if (Name == "__assert_fail" or Name == "g_assertion_message_expr" or
+        Call->isDebugOrPseudoInst() or
+        (F->isIntrinsic() and
+         (F->getIntrinsicID() == Intrinsic::lifetime_start or
+          F->getIntrinsicID() == Intrinsic::lifetime_end))) {
+        return Error::success();
+    }
+
+    const TcgV Ret = mapCallReturnValue(Mapper, Call);
+    SmallVector<TcgV, 6> Args;
+    for (unsigned i = 0; i < Call->arg_size(); ++i) {
+        if (auto BB = dyn_cast<BasicBlock>(Call->getArgOperand(i))) {
+            Args.push_back(Mapper.mapTcgLabel(BB));
+        } else {
+            Args.push_back(Mapper.defineValue(Call->getArgOperand(i)));
+        }
+    }
+
+    // Function names sometimes contain embedded type information to
+    // handle polymorphic arguments, for instance
+    //
+    //   llvm.memcpy.p0i8.p0i8.i64
+    //
+    // specifying the source and desination pointer types as i8* and
+    // the size argument as an i64.
+    //
+    // Find the index for the first '.' before the types are
+    // specified
+    //
+    //   llvm.memcpy.p0i8.p0i8.i64
+    //              ^- index of this '.'
+    size_t IndexBeforeTypes = StringRef::npos;
+    for (size_t i = Name.size() - 1; i > 0; --i) {
+        const char c = Name[i];
+        bool ValidType = (c >= '0' and c <= '9') or c == 'i' or c == 'p' or
+                         c == 'a' or c == 'v' or c == 'x';
+        if (c == '.') {
+            IndexBeforeTypes = i;
+        } else if (!ValidType) {
+            break;
+        }
+    }
+    const StringRef StrippedName = Name.substr(0, IndexBeforeTypes);
+
+    // TODO:
+    //
+    // Calls to [s]extract*() need some cleanup, it's not super obious what
+    // happens, but if the length and offset arguments are immediates we can
+    // instead map the call directly to a TCG equivalent, otherwise we continue
+    // down the chain of `if`s until we end up calling emitted code.
+    //
+
+    if (F->isIntrinsic()) {
+        if (!translateIntrinsicCall(TE, Call, F, Ret, Args, Mapper)) {
+            return mkError("Unable to map intrinsic: ", Call);
+        }
+    } else if (PseudoInst PInst = getPseudoInstFromCall(Call);
+               PInst != InvalidPseudoInst) {
+        if (!translatePseudoInstCall(CE, TE, Call, PInst, Mapper, TcgGlobals,
+                                     Ret, Args)) {
+            return mkError("Unable to map pseudo inst: ", Call);
+        }
+    } else if (StrippedName == "extract32" and Args[1].Kind == IrImmediate and
+               Args[2].Kind == IrImmediate) {
+        TE.genExtract(false, Ret, Args[0], Args[1], Args[2]);
+    } else if (StrippedName == "extract64" and Args[1].Kind == IrImmediate and
+               Args[2].Kind == IrImmediate) {
+        TE.genExtract(false, Ret, Args[0], Args[1], Args[2]);
+    } else if (StrippedName == "sextract32" and Args[1].Kind == IrImmediate and
+               Args[2].Kind == IrImmediate) {
+        TE.genExtract(true, Ret, Args[0], Args[1], Args[2]);
+    } else if (StrippedName == "sextract64" and Args[1].Kind == IrImmediate) {
+        TE.genExtract(true, Ret, Args[0],
+                      TcgV::makeImmediate("0", Ret.intSize()), Args[1]);
+    } else if (StrippedName == "deposit32" and Args[2].Kind == IrImmediate and
+               Args[3].Kind == IrImmediate) {
+        TE.genDeposit(Ret, Args[0], Args[1], Args[2], Args[3]);
+    } else if (StrippedName == "deposit64" and Args[2].Kind == IrImmediate and
+               Args[3].Kind == IrImmediate) {
+        TE.genDeposit(Ret, Args[0], Args[1], Args[2], Args[3]);
+    } else if (compat::isFunctionQemuHelper(Name)) {
+        // Map and adapt arguments to the call
+        SmallVector<TcgV, 8> IArgs;
+        for (auto Arg : Args) {
+            IArgs.push_back(TE.materialize(Arg));
+        }
+        TE.genCallHelper(Name, IArgs.begin(), IArgs.end());
+    } else {
+
+        if (!AllowDeclCall and F->isDeclaration()) {
+            return mkError("call to declaration: ", Call);
+        }
+
+        if (!F->isDeclaration() and
+            HasTranslatedFunction.find(F) == HasTranslatedFunction.end()) {
+            return mkError("call to function which failed to translate: ",
+                           Call);
+        }
+
+        StringRef Name = F->getName();
+        Name.consume_front("helper_");
+
+        Annotations Ann{};
+        if (auto It = AnnotationMap.find(F); It != AnnotationMap.end()) {
+            // TODO: This is an unnecessary copy, we currently
+            // use that a default constructed `Annotations` will
+            // return 0 for all argument flags without
+            // allocating any extra space.
+            Ann = It->second;
+        }
+
+        SmallVector<TcgV, 6> TcgArgs;
+        if (ForwardContext) {
+            TcgArgs.push_back(TE.getDisasContext());
+        }
+        if (!F->isDeclaration() and Ret.Kind != IrInvalid) {
+            TcgArgs.push_back(Ret);
+        }
+        for (size_t I = 0; I < Args.size(); ++I) {
+            if (Ann.isSet(I, ArgumentAnnotation::Immediate)) {
+                TcgArgs.push_back(Args[I]);
+            } else {
+                TcgArgs.push_back(TE.materialize(Args[I]));
+            }
+        }
+
+        if (!F->isDeclaration()) {
+            StackTwine<64> Str = Twine("emit_") + Name;
+            TE.genCallCFunc(Str, {}, TcgArgs.begin(), TcgArgs.end());
+        } else {
+            TE.genCallCFunc(Name, Ret, TcgArgs.begin(), TcgArgs.end());
+        }
+    }
+
+    return Error::success();
+}
+
+Error mapTcgOperations(const LinearBlocks &Blocks,
+                       const TcgGlobalMap &TcgGlobals,
+                       const AnnotationMapTy &AnnotationMap,
+                       const SmallPtrSet<Function *, 16> &HasTranslatedFunction,
+                       const TempAllocationData &TAD, TcgEmitter &TE,
+                       CEmitter &CE) {
+    Mapper Mapper(TE, TAD);
+    for (BasicBlock *BB : Blocks) {
+        // Set label if not first basic block
+        if (BB != Blocks[0]) {
+            const TcgV Label = Mapper.mapTcgLabel(BB);
+            TE.genSetLabel(Label);
+        }
+
+        // Emit TCG generators for the current BB
+        for (Instruction &I : *BB) {
+            if (TAD.Map.lookup(&I).Kind == IrImmediate or
+                TAD.Map.lookup(&I).ConstantExpression) {
+                continue;
+            }
+
+            switch (I.getOpcode()) {
+            case Instruction::Alloca: {
+                auto Alloca = cast<AllocaInst>(&I);
+                Mapper.defineValue(Alloca);
+            } break;
+            case Instruction::Br: {
+                auto Branch = cast<BranchInst>(&I);
+                if (Branch->isConditional()) {
+                    assert(Branch->getNumSuccessors() == 2);
+                    const TcgV Condition =
+                        Mapper.defineValue(Branch->getCondition());
+                    const TcgV CCondition = TE.materialize(Condition);
+                    const TcgV True =
+                        Mapper.mapTcgLabel(Branch->getSuccessor(0));
+                    const TcgV False =
+                        Mapper.mapTcgLabel(Branch->getSuccessor(1));
+
+                    // Jump if condition is != 0
+                    auto Zero = TcgV::makeImmediate("0", CCondition.intSize());
+                    TE.genBrcond(CmpInst::Predicate::ICMP_NE, CCondition, Zero,
+                                 True);
+                    TE.genBr(False);
+                } else {
+                    const TcgV Label =
+                        Mapper.mapTcgLabel(Branch->getSuccessor(0));
+                    TE.genBr(Label);
+                }
+            } break;
+            case Instruction::SExt: {
+                auto SExt = cast<SExtInst>(&I);
+
+                const TcgV Src = Mapper.defineValue(SExt->getOperand(0));
+                const TcgV Dst = Mapper.defineValue(&I);
+                if (Src.Kind == IrPtrToOffset) {
+                    TE.genVecSext(Dst.vecSize().ElementBitWidth, Dst, Src);
+                } else {
+                    const ValueSize SrcSize = Src.intSize();
+                    const ValueSize DstSize = Dst.intSize();
+                    if (DstSize.LlvmBitWidth < 32) {
+                        return mkError("sext to unsupported size: ", &I);
+                    }
+                    if (SrcSize.LlvmBitWidth > 1 and
+                        SrcSize.LlvmBitWidth < 32) {
+                        auto ASrc = TcgSizeAdapter(TE, Src);
+                        TE.genExts(Dst, ASrc.get(DstSize));
+                    } else if (SrcSize.LlvmBitWidth == 1 and
+                               DstSize.TcgBitWidth == 32) {
+                        TE.genMov(Dst, Src);
+                    } else {
+                        TE.genExtI32I64(Dst, Src);
+                    }
+                }
+            } break;
+            case Instruction::ZExt: {
+                auto ZExt = cast<ZExtInst>(&I);
+
+                const TcgV Src = Mapper.defineValue(ZExt->getOperand(0));
+                const TcgV Dst = Mapper.defineValue(&I);
+                if (Dst.Kind == IrValue) {
+                    const ValueSize SrcSize = Src.intSize();
+                    const ValueSize DstSize = Dst.intSize();
+                    if (SrcSize.TcgBitWidth == DstSize.TcgBitWidth) {
+                        TE.genMov(Dst, Src);
+                    } else if (SrcSize.TcgBitWidth > DstSize.TcgBitWidth and
+                               SrcSize.LlvmBitWidth == 1) {
+                        // Paradoxically we may need to emit an extract
+                        // instruction for when a zero extension is requested.
+                        // This is to account for the fact that "booleans" in
+                        // tcg can be both 64- and 32-bit. So for instance zext
+                        // i1 -> i32, here i1 may actually be 64-bit.
+                        TE.genExtrlI64I32(Dst, Src);
+                    } else {
+                        TE.genExtuI32I64(Dst, Src);
+                    }
+                } else if (Dst.Kind == IrPtrToOffset) {
+                    TE.genVecZext(Dst.vecSize().ElementBitWidth, Dst, Src);
+                } else {
+                    return mkError("Invalid TcgSize!");
+                }
+            } break;
+            case Instruction::Trunc: {
+                auto Trunc = cast<TruncInst>(&I);
+
+                const TcgV Src = Mapper.defineValue(Trunc->getOperand(0));
+                const TcgV Dst = Mapper.defineValue(&I);
+                if (Dst.Kind == IrValue) {
+                    const ValueSize SrcSize = Src.intSize();
+                    const ValueSize DstSize = Dst.intSize();
+                    if (SrcSize.TcgBitWidth == 64) {
+                        if (DstSize.LlvmBitWidth == 32) {
+                            // 64 -> 32
+                            TE.genExtrlI64I32(Dst, Src);
+                        } else {
+                            // 64 -> 16,8,1
+                            // TODO:Simplify
+                            auto Offset = TcgV::makeImmediate("0", DstSize);
+                            auto Size = TcgV::makeImmediate(
+                                Twine((int)Dst.llvmBitWidth()).str(), DstSize);
+                            auto Temp = TcgV::makeTemp({T64, I64}, IrValue);
+                            TE.defineNewTemp(Temp);
+                            TE.genExtract(false, Temp, Src, Offset, Size);
+                            TE.genExtrlI64I32(Dst, Temp);
+                        }
+                    } else {
+                        // 32 -> 16,8,1
+                        // 16 -> 8,1
+                        //  8 -> 1
+                        auto Offset = TcgV::makeImmediate("0", DstSize);
+                        auto Size = TcgV::makeImmediate(
+                            Twine((int)Dst.llvmBitWidth()).str(), DstSize);
+                        TE.genExtract(false, Dst, Src, Offset, Size);
+                    }
+                } else if (Dst.Kind == IrPtrToOffset) {
+                    TE.genVecTrunc(Dst.vecSize().ElementBitWidth, Dst, Src);
+                } else {
+                    abort();
+                }
+            } break;
+            case Instruction::Add:
+            case Instruction::And:
+            case Instruction::AShr:
+            case Instruction::LShr:
+            case Instruction::Mul:
+            case Instruction::UDiv:
+            case Instruction::SDiv:
+            case Instruction::Or:
+            case Instruction::Shl:
+            case Instruction::Sub:
+            case Instruction::Xor: {
+                auto Bin = cast<BinaryOperator>(&I);
+                // Check we are working on integers
+                TcgV Op1 = Mapper.defineValue(Bin->getOperand(0));
+                TcgV Op2 = Mapper.defineValue(Bin->getOperand(1));
+                const TcgV Res = Mapper.defineValue(Bin);
+
+                // Swap operands if the first op. is an immediate
+                // and the operator is commutative
+                if (Op1.Kind == IrImmediate and Op2.Kind != IrImmediate and
+                    Bin->isCommutative()) {
+                    std::swap(Op1, Op2);
+                }
+
+                if (Res.Kind == IrValue) {
+                    // Adapt sizes to account for boolean values where
+                    // `LlvmSize` is 1, and `TcgSize` is either 32 or 64.
+                    //
+                    // Also materialize both arguments to skip trivial
+                    // sanity checks found in `tcg_gen_[op]i*()` since e.g.
+                    // LLVM wouldn't leave an add instruction with a 0
+                    // operand.  This is also important to handle situations
+                    // such as
+                    //
+                    //   TCGv_i32 tmp = tcg_temp_new_i32();
+                    //   tcg_gen_shli_i32(tmp, src, arg);
+                    //   tcg_gen_movcond_i32(TCG_COND_GTU, res
+                    //                       tcg_constant_i32(arg),
+                    //                       tcg_constant_i32(31),
+                    //                       tcg_constant_i32(0),
+                    //                       tmp);
+                    //
+                    // where LLVM might produce an "unsafe" operation and
+                    // only guard it afterwards with a conditional move.
+                    //
+                    // TODO: Emitting
+                    //
+                    //    if (arg > 31) {
+                    //        tcg_gen_movi(res, 0);
+                    //    } else {
+                    //        tcg_gen_[op]i*(res, src, arg);
+                    //    }
+                    //
+                    // is feasible with some help from `canoncializeIR()`.
+                    TcgSizeAdapter AOp1(TE, TE.materialize(Op1));
+                    TcgSizeAdapter AOp2(TE, TE.materialize(Op2));
+
+                    TE.genBinOp(Res, Bin->getOpcode(), AOp1.get(Res.intSize()),
+                                AOp2.get(Res.intSize()));
+                } else if (Res.Kind == IrPtrToOffset) {
+                    TE.genVecBinOp(Bin->getOpcode(), Res, Op1, Op2);
+                }
+            } break;
+            case Instruction::Call: {
+                auto Call = cast<CallInst>(&I);
+                auto Err =
+                    translateCall(TcgGlobals, AnnotationMap,
+                                  HasTranslatedFunction, TE, CE, Mapper, Call);
+                if (Err) {
+                    return Err;
+                }
+            } break;
+            case Instruction::ICmp: {
+                auto *ICmp = cast<ICmpInst>(&I);
+                const TcgV Op1 = Mapper.defineValue(I.getOperand(0));
+                const TcgV Op2 = Mapper.defineValue(I.getOperand(1));
+                const TcgV Res = Mapper.defineValue(ICmp);
+
+                ICmpInst::Predicate LlvmPred = ICmp->getPredicate();
+
+                if (Op1.Kind == IrPtrToOffset) {
+                    TE.genVecCmp(Res, LlvmPred, Op1, Op2);
+                } else {
+                    auto IOp1 = TE.materialize(Op1);
+                    if (ICmp->isSigned()) {
+                        ensureSignBitIsSet(TE, IOp1);
+                        ensureSignBitIsSet(TE, Op2);
+                    }
+                    TE.genSetcond(LlvmPred, Res, IOp1, Op2);
+                }
+            } break;
+            case Instruction::Select: {
+                auto Select = cast<SelectInst>(&I);
+                const TcgV Res = Mapper.defineValue(Select);
+
+                if (Res.Kind == IrPtr) {
+                    return mkError(
+                        "Select statements for pointer types not supported: ",
+                        Select);
+                }
+                const TcgV Cond = Mapper.defineValue(Select->getCondition());
+                const TcgV True = Mapper.defineValue(Select->getTrueValue());
+                const TcgV False = Mapper.defineValue(Select->getFalseValue());
+
+                if (Res.Kind == IrPtrToOffset) {
+                    TE.genVecBitsel(Res, Cond, True, False);
+                } else if (Cond.Kind == IrImmediate) {
+                    assert(Res.Kind != IrImmediate);
+                    const TcgV MTrue = TE.materialize(True);
+                    const TcgV MFalse = TE.materialize(False);
+                    TE.genMov(Res, CE.ternary(Cond, MTrue, MFalse));
+                } else {
+                    const TcgV Zero = TcgV::makeImmediate("0", Res.intSize());
+                    TcgSizeAdapter ACond(TE, Cond);
+                    TcgSizeAdapter ATrue(TE, True);
+                    TcgSizeAdapter AFalse(TE, False);
+                    if (True.Kind == IrImmediate or False.Kind == IrImmediate) {
+                        auto CTrue = TE.materialize(ATrue.get(Res.intSize()));
+                        auto CFalse = TE.materialize(AFalse.get(Res.intSize()));
+                        auto CCond = ACond.get(CTrue.intSize());
+
+                        TE.genMovcond(CmpInst::Predicate::ICMP_NE, Res, CCond,
+                                      Zero, CTrue, CFalse);
+                    } else {
+                        TE.genMovcond(CmpInst::Predicate::ICMP_NE, Res,
+                                      ACond.get(Res.intSize()), Zero,
+                                      ATrue.get(Res.intSize()),
+                                      AFalse.get(Res.intSize()));
+                    }
+                }
+            } break;
+            case Instruction::Ret: {
+                auto Ret = cast<ReturnInst>(&I);
+                if (Ret->getNumOperands() == 0) {
+                    break;
+                }
+                assert(TAD.hasReturnValue());
+                const TcgV Tcg = TAD.Map.lookup(Ret->getReturnValue());
+                // Even if `SkipReturnMov` is set we need to emit a mov for
+                // constant expressions and immediates.
+                if (Tcg.Kind == IrImmediate or Tcg.ConstantExpression or
+                    (TAD.flags & SkipReturnMov) == 0) {
+                    TE.genMov(TAD.ReturnValue, Tcg);
+                }
+            } break;
+            case Instruction::Load: {
+                auto *Load = cast<LoadInst>(&I);
+                auto *LlvmPtr = Load->getPointerOperand();
+                const TcgV Ptr = Mapper.defineValue(LlvmPtr);
+                const TcgV Res = Mapper.defineValue(Load);
+
+                switch (Ptr.Kind) {
+                case IrPtr: {
+                    auto Zero = TcgV::makeImmediate("0", Res.intSize());
+                    TE.genHostLoad(Res, Ptr, Zero);
+                } break;
+                case IrImmediate: {
+                    // Add pointer dereference to immediate address
+                    TE.genMov(Res, CE.deref(Ptr, Res.intSize()));
+                } break;
+                case IrValue: {
+                    TE.genMov(Res, Ptr);
+                } break;
+                case IrPtrToOffset: {
+                    TE.genHostLoadFromVec(Res, Ptr);
+                } break;
+                default:
+                    return mkError("Load from unsupported TcgV type");
+                };
+            } break;
+            case Instruction::Store: {
+                auto *Store = cast<StoreInst>(&I);
+                auto *LlvmPtr = Store->getPointerOperand();
+                const TcgV Val = Mapper.defineValue(Store->getValueOperand());
+                const TcgV Ptr = Mapper.defineValue(LlvmPtr);
+                if (Ptr.Kind == IrValue) {
+                    // TODO: Is this path still needed?
+                    switch (Val.Kind) {
+                    case IrImmediate:
+                    case IrValue: {
+                        TE.genMov(Ptr, Val);
+                    } break;
+                    default:
+                        return mkError("Store from unsupported TcgV type");
+                    };
+                } else if (Ptr.Kind == IrPtr) {
+                    TE.genHostStore(Ptr, TE.materialize(Val));
+                } else if (Ptr.Kind == IrPtrToOffset) {
+                    // Stores to IrPtrToOffset are ignored, they are an artifact
+                    // of IrPtrToOffset arguments being pointers. Stores to
+                    // results are instead taken care of by whatever instruction
+                    // generated the result.
+                    // TODO: This is no longer true, double check
+                } else {
+                    return mkError("Store to unsupported TcgV kind: ", Store);
+                }
+            } break;
+            case Instruction::Unreachable: {
+                // TODO: Need to make sure unreachables are optimized out
+                // earlier.
+            } break;
+            case Instruction::Switch: {
+                auto Switch = cast<SwitchInst>(&I);
+                // Operands to switch instructions alternate between
+                // case values and the corresponding label:
+                //   Operands: { Cond, DefaultLabel, Case0, Label0, Case1,
+                //   Label1, ... }
+                const TcgV Val = Mapper.defineValue(Switch->getOperand(0));
+                const TcgV Default =
+                    Mapper.mapTcgLabel(cast<BasicBlock>(Switch->getOperand(1)));
+                for (uint32_t i = 2; i < Switch->getNumOperands(); i += 2) {
+                    const TcgV BranchVal =
+                        Mapper.defineValue(Switch->getOperand(i));
+                    const TcgV Branch = Mapper.mapTcgLabel(
+                        cast<BasicBlock>(Switch->getOperand(i + 1)));
+                    TE.genBrcond(CmpInst::Predicate::ICMP_EQ, Val, BranchVal,
+                                 Branch);
+                }
+                TE.genBr(Default);
+            } break;
+            default: {
+                return mkError("Instruction not yet implemented: ", &I);
+            }
+            }
+        }
+    }
+
+    return Error::success();
+}
-- 
2.52.0

[PATCH v2 38/50] helper-to-tcg: Add README

[Anton Johansson via qemu development]

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 subprojects/helper-to-tcg/README.md | 297 ++++++++++++++++++++++++++++
 1 file changed, 297 insertions(+)
 create mode 100644 subprojects/helper-to-tcg/README.md

diff --git a/subprojects/helper-to-tcg/README.md b/subprojects/helper-to-tcg/README.md
new file mode 100644
index 0000000000..49654eb79f
--- /dev/null
+++ b/subprojects/helper-to-tcg/README.md
@@ -0,0 +1,297 @@
+# helper-to-tcg
+
+`helper-to-tcg` is a standalone LLVM IR to TCG translator, with the goal of simplifying the implementation of complicated instructions in TCG. Instruction semantics can be specified either directly in LLVM IR or any language that can be compiled to it (C, C++, ...). However, the tool is tailored towards QEMU helper functions written in C.
+
+Internally, `helper-to-tcg` consists of a mix of custom and built-in transformation and analysis passes that are applied to the input LLVM IR sequentially. The pipeline of passes is laid out as follows
+```
+           +---------------+    +-----+    +---------------+    +------------+
+LLVM IR -> | PrepareForOpt | -> | -Oz | -> | PrepareForTcg | -> | TcgGenPass | -> TCG
+           +---------------+    +-----+    +---------------+    +------------+
+```
+where the custom passes performs:
+* `PrepareForOpt` - Early information gathering and culling pass. Removes unneeded functions, maps function annotations, gathers debuginfo data, demangles function names, etc.;
+* `PrepareForTcg` - Post-optimization pass that tries to get the IR as close to TCG as possible, goal is to take complexity away from the backend;
+* `TcgGenPass` - Backend pass that propagates constants, allocates TCG variables to LLVM values, and emits final TCG C code.
+
+As for LLVM optimization, `-Oz` strikes a good balance between unrolling and vectorization, from testing. More aggressive optimization levels would often unroll loops over compacting it with loop vectorization.
+
+## Project Structure
+
+* `get-llvm-ir.py` - Helper script to convert a QEMU `.c` file to LLVM IR by getting compile flags from `compile_commands.json`;
+* `pipeline` - Implementation of a LLVM pipeline orchestrating passes and handling input;
+* `src` - Implementation of custom LLVM passes: `PrepareForOptPass`,`PrepareForTcgPass`,`TcgGenPass`, and common features;
+* `include` - Shared headers between `src`;
+* `tests` - Simple end-to-end tests of C functions we expect to be able to translate, tests fail if any function fails to translate, output is verified against expected output for support major LLVM versions.
+
+## Example Translations
+
+`helper-to-tcg` is able to deal with a wide variety of helper functions, the following code snippet contains two examples from the Hexagon architecture implementing the semantics of a predicated and instruction (`A2_pandt`) and a vectorized signed saturated 2-element scalar product (`V6_vdmpyhvsat`).
+
+```c
+int32_t HELPER(A2_pandt)(CPUHexagonState *env, int32_t RdV,
+                         int32_t PuV, int32_t RsV, int32_t RtV)
+{
+    if(fLSBOLD(PuV)) {
+        RdV=RsV&RtV;
+    } else {
+        CANCEL;
+    }
+    return RdV;
+}
+
+void HELPER(V6_vdmpyhvsat)(CPUHexagonState *env,
+                           void * restrict VdV_void,
+                           void * restrict VuV_void,
+                           void * restrict VvV_void)
+{
+    fVFOREACH(32, i) {
+        size8s_t accum = fMPY16SS(fGETHALF(0,VuV.w[i]),fGETHALF(0, VvV.w[i]));
+        accum += fMPY16SS(fGETHALF(1,VuV.w[i]),fGETHALF(1, VvV.w[i]));
+        VdV.w[i] = fVSATW(accum);
+    }
+}
+```
+For the above snippet, `helper-to-tcg` produces the following TCG
+```c
+void emit_A2_pandt(TCGv_i32 temp0, TCGv_env env, TCGv_i32 temp4,
+                   TCGv_i32 temp8, TCGv_i32 temp7, TCGv_i32 temp6) {
+    TCGv_i32 temp2 = tcg_temp_new_i32();
+    tcg_gen_andi_i32(temp2, temp8, 1);
+    TCGv_i32 temp5 = tcg_temp_new_i32();
+    tcg_gen_and_i32(temp5, temp6, temp7);
+    tcg_gen_movcond_i32(TCG_COND_EQ, temp0, temp2, tcg_constant_i32(0), temp4, temp5);
+}
+
+void emit_V6_vdmpyhvsat(TCGv_env env, intptr_t vec3,
+                        intptr_t vec7, intptr_t vec6) {
+     VectorMem mem = {0};
+     intptr_t vec0 = temp_new_gvec(&mem, 128);
+     tcg_gen_gvec_shli(MO_32, vec0, vec7, 16, 128, 128);
+     intptr_t vec5 = temp_new_gvec(&mem, 128);
+     tcg_gen_gvec_sari(MO_32, vec5, vec0, 16, 128, 128);
+     intptr_t vec1 = temp_new_gvec(&mem, 128);
+     tcg_gen_gvec_shli(MO_32, vec1, vec6, 16, 128, 128);
+     tcg_gen_gvec_sari(MO_32, vec1, vec1, 16, 128, 128);
+     tcg_gen_gvec_mul(MO_32, vec1, vec1, vec5, 128, 128);
+     intptr_t vec2 = temp_new_gvec(&mem, 128);
+     tcg_gen_gvec_sari(MO_32, vec2, vec7, 16, 128, 128);
+     tcg_gen_gvec_sari(MO_32, vec0, vec6, 16, 128, 128);
+     tcg_gen_gvec_mul(MO_32, vec2, vec0, vec2, 128, 128);
+     tcg_gen_gvec_ssadd(MO_32, vec3, vec1, vec2, 128, 128);
+}
+```
+
+In the first case, the predicated and instruction was made branchless by using a conditional move, and in the latter case the inner loop of the vectorized scalar product could be converted to a few vectorized shifts and multiplications, folllowed by a vectorized signed saturated addition.
+
+## Usage
+
+Building `helper-to-tcg` produces a binary implementing the pipeline outlined above, going from LLVM IR to TCG.
+
+### Specifying Functions to Translate
+
+Unless `--translate-all-helpers` is specified, the default behaviour of `helper-to-tcg` is to only translate functions annotated via a special `"helper-to-tcg"` annotation. Functions called by annotated functions will also be translated, see the following example:
+
+```c
+// Function will be translated, annotation provided
+__attribute__((annotate ("helper-to-tcg")))
+int f(int a, int b) {
+    return 2 * g(a, b);
+}
+
+// Function will be translated, called by annotated `f()` function
+int g(int a, int b) {
+    ...
+}
+
+// Function will not be translated
+int h(int a, int b) {
+    ...
+}
+```
+
+### Immediate and Vector Arguments
+
+Function annotations are in some cases used to provide extra information to `helper-to-tcg` not otherwise present in the IR. For example, whether an integer argument should actually be treated as an immediate rather than a register, or if a pointer argument should be treated as a `gvec` vector (offset into `CPUArchState`). For instance:
+```c
+__attribute__((annotate ("helper-to-tcg")))
+__attribute__((annotate ("immediate: 1")))
+int f(int a, int i) {
+    ...
+}
+
+__attribute__((annotate ("helper-to-tcg")))
+__attribute__((annotate ("ptr-to-offset: 0, 1")))
+void g(void * restrict a, void * restrict b) {
+    ...
+}
+```
+where `"immediate: 1"` tells `helper-to-tcg` that the argument with index `1` should be treated as an immediate (multiple arguments are specified through a comma separated list). Similarly `"ptr-to-offset: 0, 1"` indicates that arguments width index 0 and 1 should be treated as offsets from `CPUArchState` (given as `intptr_t`), rather than actual pointer arguments. For the above code, `helper-to-tcg` emits
+```c
+void emit_f(TCGv_i32 res, TCGv_i32 a, int i) {
+    ...
+}
+
+void emit_g(intptr_t a, intptr_t b) {
+    ...
+}
+```
+
+### Loads and Stores
+
+Translating loads and stores is slightly trickier, as some QEMU specific assumptions are made. Loads and stores in the input are assumed to go through the `cpu_[st|ld]*()` functions defined in `exec/cpu_ldst.h` that a helper function would use. 
+
+If using standalone input functions (not QEMU helper functions), loads and stores are still represented by `cpu_[st|ld]*()` which needs to be declared, consider:
+```c
+/* Opaque CPU state type, will be mapped to tcg_env */
+struct CPUArchState;
+typedef struct CPUArchState CPUArchState;
+
+/* Prototype of QEMU helper guest load/store functions, see exec/cpu_ldst.h */
+uint32_t cpu_ldub_data(CPUArchState *, uint32_t ptr);
+void cpu_stb_data(CPUArchState *, uint32_t ptr, uint32_t data);
+
+uint32_t helper_ld8(CPUArchState *env, uint32_t addr) {
+    return cpu_ldub_data(env, addr);
+}
+
+void helper_st8(CPUArchState *env, uint32_t addr, uint32_t data) {
+    return cpu_stb_data(env, addr, data);
+}
+```
+implementing an 8-bit load and store instruction, these will be translated to the following TCG.
+```c
+void emit_ld8(TCGv_i32 temp0, TCGv_env env, TCGv_i32 temp1) {
+    tcg_gen_qemu_ld_i32(temp0, temp1, tb_mmu_index(tcg_ctx->gen_tb->flags), MO_UB);
+}
+
+void emit_st8(TCGv_env env, TCGv_i32 temp0, TCGv_i32 temp1) {
+    tcg_gen_qemu_st_i32(temp1, temp0, tb_mmu_index(tcg_ctx->gen_tb->flags), MO_UB);
+}
+```
+Note, the emitted code assumes the definition of a `tb_mmu_index()` function to retrieve the current CPU MMU index, the name of this function can be configured via the `--mmu-index-function` flag.
+
+### Mapping CPU State
+
+In QEMU, commonly accessed fields in the `CPUArchState` are often mapped to global `TCGv*` variables representing that piece of CPU state in TCG. When translating helper functions (or other C functions), a method of specifying which fields in the CPU state should be mapped to which globals is needed. To this end, a declarative approach is taken, where mappings between CPU state and globals can be consumed by both `helper-to-tcg` and runtime QEMU for instantiating the `TCGv` globals themselves.
+
+Users must define this mapping via a global `cpu_tcg_mapping []` array, as can be seen in the following example where `mapped_field` of `CPUArchState` is mapped to the global `tcg_field`. For more complicated examples see the tests in `tests/cpustate.c`.
+```c
+#include <stdint.h>
+#include "tcg/tcg-global-mappings.h"
+
+/* Define a CPU state with some different fields */
+
+typedef struct CPUArchState {
+    uint32_t mapped_field;
+    uint32_t unmapped_field;
+} CPUArchState;
+
+/* Dummy struct, in QEMU this would correspond to TCGv_i32 in tcg.h */
+typedef struct TCGv_i32 {} TCGv_i32;
+
+/* Global TCGv representing CPU state */
+TCGv_i32 tcg_field;
+
+/*
+ * Finally provide a mapping of CPUArchState to TCG globals we care about, here
+ * we map mapped_field to tcg_field
+ */
+cpu_tcg_mapping mappings[] = {
+    CPU_TCG_MAP(CPUArchState, tcg_field, mapped_field),
+};
+
+uint32_t helper_mapped(CPUArchState *env) {
+    return env->mapped_field;
+}
+
+uint32_t helper_unmapped(CPUArchState *env) {
+    return env->unmapped_field;
+}
+```
+Note, the name of the `cpu_tcg_mapping[]` is provided via the `--tcg-global-mappings` flag. For the above example, `helper-to-tcg` emits
+```c
+extern TCGv_i32 tcg_field;
+
+void emit_mapped(TCGv_i32 temp0, TCGv_env env) {
+    tcg_gen_mov_i32(temp0, tcg_field);
+}
+
+void emit_unmapped(TCGv_i32 temp0, TCGv_env env) {
+    TCGv_ptr ptr1 = tcg_temp_new_ptr();
+    tcg_gen_addi_ptr(ptr1, env, 128ull);
+    tcg_gen_ld_i32(temp0, ptr1, 0);
+}
+```
+where accesses in the input C code are correctly mapped to the corresponding TCG globals. The unmapped `CPUArchState` access turns into pointer math and a load, whereas the mapped access turns into a `mov` from a global.
+
+### Automatic Calling of Generated Code
+
+Finally, calling the generated code is as simple as including the output of `helper-to-tcg` into the project and manually calling `emit_*(...)`. However, when dealing with an existing frontend that has a lot of helper functions already in use, we simplify this process somewhat for non-vector instructions. For translated helper functions `helper-to-tcg` emits its own `gen_helper_*()` function definitions
+```c
+/* helper-to-tcg-emitted.c */
+void emit_add(TCGv_ptr env, TCGv_i32 d, TCGv_i32 a, uint32_t imm)
+{
+    tcg_gen_addi_i32(d, a, imm);
+}
+
+/* helper-to-tcg-emitted.h */
+static inline void gen_helper_add_imm(TCGv_ptr env, TCGv_i32 d, TCGv_i32 a, TCGv_i32 imm)
+{
+    emit_add(env, d, a, tcgv_i32_temp(imm)->val);
+}
+```
+then from code that needs to call `gen_helper_add_imm` include
+```c
+/* translate.c */
+
+/*
+ * Include generated gen_helper_*() definitions if using helper-to-tcg, but
+ * only when not generating input IR since the header won't exist at that point.
+ */
+#if defined(TARGET_HELPER_TO_TCG) && !defined(HELPER_TO_TCG_IR_GEN)
+#include "helper-to-tcg-emitted.h"
+#else
+#include "exec/helper-gen.h"
+#endif
+```
+
+### Simple Command Usage
+
+Assume a `helpers.c` file with functions to translate, then to obtain LLVM IR
+```bash
+$ clang helpers.c -O0 -Xclang -disable-O0-optnone -S -emit-llvm
+```
+which produces `helpers.ll` to be fed into `helper-to-tcg`
+```bash
+$ ./helper-to-tcg helpers.ll --translate-all-helpers
+```
+where `--translate-all-helpers` means "translate all functions starting with helper_*". Finally, the above command produces `helper-to-tcg-emitted.[c|h]` with emitted TCG code.
+
+By default meson uses `llvm-config` to find LLVM, usually this corresponds to the latest version installed on the system. Since `helper-to-tcg` only supports LLVM 15-21, the path to `llvm-config` can be manually overridden using `meson configure -Dllvm_config_path=...` when needed. This is also useful for testing multiple LLVM versions. If `helper-to-tcg` is used as a part of `QEMU`, this is specified using `../configure -Dhelper_to_tcg:llvm_config_path=...`.
+
+### Testing
+
+When building `helper-to-tcg` as a part of a QEMU project the `end-to-end` tests may be ran via either of the commands
+```bash
+$ make check
+$ make check-helper-to-tcg
+$ meson test --suite 'helper-to-tcg:end-to-end'
+```
+where the last one is useful when building as a standalone project.
+
+On the QEMU end, a docker container has been added for testing with the supported LLVM versions, this may be triggered via
+```bash
+$ make docker-test-helper-to-tcg@debian-llvm
+```
+and will build `helper-to-tcg` along with the current target, run all end-to-end and `check-tcg` tests for each version of LLVM that is targeted.
+
+Debug logging can be turned on via
+```bash
+$ ./helper-to-tcg --debug
+```
+or to restrict logging to given pass
+
+```bash
+$ ./helper-to-tcg --debug-only=${pass}
+```
+where `${pass}` is one of `map-tcg-ops, map-temporaries, map-arguments, tcg-gen-pass, pipeline, transform-geps, prepare-for-opt`.
-- 
2.52.0

[PATCH v2 40/50] test: helper-to-tcg docker tests

[Anton Johansson via qemu development]

Adds a docker container with LLVM versions 15-21 and a docker test for
building helper-to-tcg and running end-to-end tests along with running
check-tcg, for testing targets such as Hexagon.

Signed-off-by: Anton Johansson <anjo@rev.ng>
--
NOTE: Last I checked I had some troubles with C++17 support within the
container, was a while ago though.
---
 tests/docker/dockerfiles/debian-llvm.docker | 38 +++++++++++++++++++
 tests/docker/test-helper-to-tcg             | 41 +++++++++++++++++++++
 2 files changed, 79 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-llvm.docker
 create mode 100755 tests/docker/test-helper-to-tcg

diff --git a/tests/docker/dockerfiles/debian-llvm.docker b/tests/docker/dockerfiles/debian-llvm.docker
new file mode 100644
index 0000000000..7bfc084a34
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-llvm.docker
@@ -0,0 +1,38 @@
+FROM docker.io/library/debian:11-slim
+
+RUN apt update && \
+    DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata && \
+    DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt install -y --no-install-recommends \
+        bison \
+        ca-certificates \
+        flex \
+        gawk \
+        libmpc-dev \
+        libmpfr-dev \
+        libglib2.0-dev \
+        libpixman-1-dev \
+        make \
+        ninja-build \
+        rsync \
+        pkgconf \
+        wget \
+        lsb-release \
+        software-properties-common \
+        gnupg \
+        meson \
+        python3-pip \
+        python3-setuptools \
+        python3-venv \
+        python3-wheel
+
+RUN /usr/bin/pip3 install tomli
+
+RUN wget https://apt.llvm.org/llvm.sh && chmod +x llvm.sh
+RUN ./llvm.sh 15 all
+RUN ./llvm.sh 16 all
+RUN ./llvm.sh 17 all
+RUN ./llvm.sh 18 all
+RUN ./llvm.sh 19 all
+RUN ./llvm.sh 20 all
+RUN ./llvm.sh 21 all
diff --git a/tests/docker/test-helper-to-tcg b/tests/docker/test-helper-to-tcg
new file mode 100755
index 0000000000..0c27040d60
--- /dev/null
+++ b/tests/docker/test-helper-to-tcg
@@ -0,0 +1,41 @@
+#!/bin/bash -e
+
+TEST_COMMAND=""
+TARGET_LIST=""
+
+cd "$BUILD_DIR"
+
+# Checks helper-to-tcg builds and passes unit tests
+# for targetted LLVM versions.  Run meson from QEMU source so helper-to-tcg tests can find
+# `tcg-global-mappings.h`.
+for version in {15..21}; do
+    llvm_config=llvm-config-${version}
+    build_dir="${BUILD_DIR}/build-helper-to-tcg-${version}"
+    cxx=$(${llvm_config} --bindir)/clang++
+    [ ! -d ${build_dir} ] && mkdir ${build_dir}
+    CXX=${cxx} meson setup ${build_dir} ${QEMU_SRC}/subprojects/helper-to-tcg -Dllvm_config_path=${llvm_config}
+    meson compile -C ${build_dir}
+    meson test -C ${build_dir} --suite 'helper-to-tcg:helper-to-tcg'
+done
+
+# Runs check-tcg for all LLVM versions
+llvm_config_main=llvm-config-15
+bin_main=$(${llvm_config_main} --bindir)
+cc_main=${bin_main}/clang
+cxx_main=${bin_main}/clang++
+for version in {15..21}; do
+    llvm_config=llvm-config-${version}
+    build_dir=build-qemu-${version}
+    bin=$(${llvm_config} --bindir)
+    cxx=${bin}/clang++
+    cc=${bin}/clang
+    [ ! -d ${build_dir} ] && mkdir ${build_dir}
+
+    pushd ${build_dir}
+
+    TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \
+    build_qemu "--cc=${cc_main} --cxx=${cxx_main} -Dhelper-to-tcg:llvm_config_path=${llvm_config} --enable-debug-tcg"
+    check_tcg
+
+    popd
+done
-- 
2.52.0

[PATCH v2 41/50] target/hexagon: Add get_tb_mmu_index()

[Anton Johansson via qemu development]

Adds a functions to return the current mmu index given tb_flags of the
current translation block.  Required by helper-to-tcg in order to
retrieve the mmu index for memory operations without changing the
signature of helper functions.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 target/hexagon/translate.c | 12 ++++++++++++
 target/hexagon/translate.h |  4 ++++
 2 files changed, 16 insertions(+)

diff --git a/target/hexagon/translate.c b/target/hexagon/translate.c
index 199b4f8c2e..bfd84bba4e 100644
--- a/target/hexagon/translate.c
+++ b/target/hexagon/translate.c
@@ -128,6 +128,18 @@ intptr_t ctx_tmp_vreg_off(DisasContext *ctx, int regnum,
     return offset;
 }
 
+#if defined(TARGET_HELPER_TO_TCG)
+/*
+ * Returns the current mmu index given tb_flags of the current translation
+ * block.  Required by helper-to-tcg in order to retrieve the mmu index for
+ * memory operations without changing the signature of helper functions.
+ */
+int get_tb_mmu_index(uint32_t flags)
+{
+    return FIELD_EX32(flags, TB_FLAGS, MMU_INDEX);
+}
+#endif
+
 static void gen_exception(int excp, uint32_t PC)
 {
     gen_helper_raise_exception(tcg_env, tcg_constant_i32(excp),
diff --git a/target/hexagon/translate.h b/target/hexagon/translate.h
index 2fca157553..6cf3b2fe3b 100644
--- a/target/hexagon/translate.h
+++ b/target/hexagon/translate.h
@@ -345,4 +345,8 @@ FIELD(PROBE_PKT_SCALAR_HVX_STORES, MMU_IDX,        5, 2)
 
 void gen_framecheck(DisasContext *ctx, TCGv_i32 addr, TCGv_i32 ea);
 
+#if defined(TARGET_HELPER_TO_TCG)
+int get_tb_mmu_index(uint32_t flags);
+#endif
+
 #endif
-- 
2.52.0

[PATCH v2 42/50] target/hexagon: Increase VECTOR_TEMPS_MAX

[Anton Johansson via qemu development]

Doubles the number of vector temporaries from 4 to 8, useful when
emitting vector instructions with helper-to-tcg to handle more
complicated helpers.  `tmp_VRegs` must be later specified later to
`helper-to-tcg` through `--temp-vector-block=tmp_VRegs`.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 target/hexagon/cpu.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/target/hexagon/cpu.h b/target/hexagon/cpu.h
index 7694fd91fa..02ac011632 100644
--- a/target/hexagon/cpu.h
+++ b/target/hexagon/cpu.h
@@ -117,7 +117,7 @@ typedef struct {
 #define SET_EXCEPTION           (env->status |= EXEC_STATUS_EXCEPTION)
 
 /* Maximum number of vector temps in a packet */
-#define VECTOR_TEMPS_MAX            4
+#define VECTOR_TEMPS_MAX            8
 
 typedef struct CPUArchState {
     target_ulong gpr[TOTAL_PER_THREAD_REGS];
@@ -160,6 +160,10 @@ typedef struct CPUArchState {
     MMVector future_VRegs[VECTOR_TEMPS_MAX] QEMU_ALIGNED(16);
     MMVector tmp_VRegs[VECTOR_TEMPS_MAX] QEMU_ALIGNED(16);
 
+#ifdef TARGET_HELPER_TO_TCG
+    uint8_t tmp_vmem[4096] QEMU_ALIGNED(16);
+#endif
+
     MMQReg QRegs[NUM_QREGS] QEMU_ALIGNED(16);
     MMQReg future_QRegs[NUM_QREGS] QEMU_ALIGNED(16);
 
-- 
2.52.0

[PATCH v2 43/50] target/hexagon: Provide env to tcg global mapping

[Anton Johansson via qemu development]

Replaces previous calls to tcg_global_mem_new*() with a declarative
global array of cpu_mapping structs.  This array can be used to
initialize all TCG globals with one function call from the target, and
may additionally be used from LLVM based tools to map between offsets
into a struct and a mapped TCGv global.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 target/hexagon/translate.c | 119 ++++++++++++++++++-------------------
 1 file changed, 57 insertions(+), 62 deletions(-)

diff --git a/target/hexagon/translate.c b/target/hexagon/translate.c
index bfd84bba4e..870475010b 100644
--- a/target/hexagon/translate.c
+++ b/target/hexagon/translate.c
@@ -33,6 +33,7 @@
 #include "genptr.h"
 #include "printinsn.h"
 #include "exec/target_page.h"
+#include "tcg/tcg-global-mappings.h"
 
 #define HELPER_H "helper.h"
 #include "exec/helper-info.c.inc"
@@ -1328,77 +1329,68 @@ static char store_val64_names[STORES_MAX][NAME_LEN];
 static char vstore_addr_names[VSTORES_MAX][NAME_LEN];
 static char vstore_size_names[VSTORES_MAX][NAME_LEN];
 static char vstore_pending_names[VSTORES_MAX][NAME_LEN];
-
-void hexagon_translate_init(void)
-{
-    int i;
-
-    opcode_init();
+static const char *store_addr_names_ptr[STORES_MAX];
+static const char *store_width_names_ptr[STORES_MAX];
+static const char *store_val32_names_ptr[STORES_MAX];
+static const char *store_val64_names_ptr[STORES_MAX];
+
+cpu_tcg_mapping tcg_global_mappings[] = {
+    /* General purpose and predicate registers */
+    CPU_TCG_MAP_ARRAY(CPUHexagonState, hex_gpr,  gpr,  hexagon_regnames),
+    CPU_TCG_MAP_ARRAY(CPUHexagonState, hex_pred, pred, hexagon_prednames),
+
+    /* Misc */
+    CPU_TCG_MAP(CPUHexagonState, hex_new_value_usr,  new_value_usr),
+    CPU_TCG_MAP(CPUHexagonState, hex_next_PC,        next_PC),
+    CPU_TCG_MAP(CPUHexagonState, hex_slot_cancelled, slot_cancelled),
+    CPU_TCG_MAP(CPUHexagonState, hex_llsc_addr,      llsc_addr),
+    CPU_TCG_MAP(CPUHexagonState, hex_llsc_val,       llsc_val),
+    CPU_TCG_MAP(CPUHexagonState, hex_llsc_val_i64,   llsc_val_i64),
 
 #ifndef CONFIG_USER_ONLY
-    for (i = 0; i < NUM_GREGS; i++) {
-            hex_greg[i] = tcg_global_mem_new_i32(tcg_env,
-                offsetof(CPUHexagonState, greg[i]),
-                hexagon_gregnames[i]);
-    }
-    for (i = 0; i < NUM_SREGS; i++) {
-        if (i < HEX_SREG_GLB_START) {
-            hex_t_sreg[i] = tcg_global_mem_new_i32(tcg_env,
-                offsetof(CPUHexagonState, t_sreg[i]),
-                hexagon_sregnames[i]);
-        }
-    }
+    CPU_TCG_MAP(CPUHexagonState, hex_cause_code, cause_code),
+    CPU_TCG_MAP(CPUHexagonState, hex_cycle_count, t_cycle_count),
+    CPU_TCG_MAP_ARRAY(CPUHexagonState, hex_greg, greg, hexagon_gregnames),
+    CPU_TCG_MAP_ARRAY(CPUHexagonState, hex_t_sreg, t_sreg, hexagon_sregnames),
 #endif
-    for (i = 0; i < TOTAL_PER_THREAD_REGS; i++) {
-        hex_gpr[i] = tcg_global_mem_new(tcg_env,
-            offsetof(CPUHexagonState, gpr[i]),
-            hexagon_regnames[i]);
-    }
-    hex_new_value_usr = tcg_global_mem_new(tcg_env,
-        offsetof(CPUHexagonState, new_value_usr), "new_value_usr");
-    hex_next_PC = tcg_global_mem_new(tcg_env,
-        offsetof(CPUHexagonState, next_PC), "next_PC");
 
-    for (i = 0; i < NUM_PREGS; i++) {
-        hex_pred[i] = tcg_global_mem_new(tcg_env,
-            offsetof(CPUHexagonState, pred[i]),
-            hexagon_prednames[i]);
-    }
-    hex_slot_cancelled = tcg_global_mem_new(tcg_env,
-        offsetof(CPUHexagonState, slot_cancelled), "slot_cancelled");
-    hex_llsc_addr = tcg_global_mem_new(tcg_env,
-        offsetof(CPUHexagonState, llsc_addr), "llsc_addr");
-    hex_llsc_val = tcg_global_mem_new(tcg_env,
-        offsetof(CPUHexagonState, llsc_val), "llsc_val");
-    hex_llsc_val_i64 = tcg_global_mem_new_i64(tcg_env,
-        offsetof(CPUHexagonState, llsc_val_i64), "llsc_val_i64");
-#ifndef CONFIG_USER_ONLY
-    hex_cause_code = tcg_global_mem_new_i32(tcg_env,
-        offsetof(CPUHexagonState, cause_code), "cause_code");
-    hex_cycle_count = tcg_global_mem_new_i64(tcg_env,
-        offsetof(CPUHexagonState, t_cycle_count), "t_cycle_count");
-#endif
-    for (i = 0; i < STORES_MAX; i++) {
-        snprintf(store_addr_names[i], NAME_LEN, "store_addr_%d", i);
-        hex_store_addr[i] = tcg_global_mem_new(tcg_env,
-            offsetof(CPUHexagonState, mem_log_stores[i].va),
-            store_addr_names[i]);
+    /* Logging stores */
+    CPU_TCG_MAP_ARRAY_OF_STRUCTS(CPUHexagonState, hex_store_addr,
+                                 mem_log_stores, va,     store_addr_names_ptr),
+    CPU_TCG_MAP_ARRAY_OF_STRUCTS(CPUHexagonState, hex_store_width,
+                                 mem_log_stores, width,  store_width_names_ptr),
+    CPU_TCG_MAP_ARRAY_OF_STRUCTS(CPUHexagonState, hex_store_val32,
+                                 mem_log_stores, data32, store_val32_names_ptr),
+    CPU_TCG_MAP_ARRAY_OF_STRUCTS(CPUHexagonState, hex_store_val64,
+                                 mem_log_stores, data64, store_val64_names_ptr),
+};
 
-        snprintf(store_width_names[i], NAME_LEN, "store_width_%d", i);
-        hex_store_width[i] = tcg_global_mem_new_i32(tcg_env,
-            offsetof(CPUHexagonState, mem_log_stores[i].width),
-            store_width_names[i]);
+size_t tcg_global_mapping_count = ARRAY_SIZE(tcg_global_mappings);
 
+static void init_cpu_reg_names(void) {
+    /*
+     * Create register names and store them in `*_names`,
+     * then copy to and array of pointers in `*_names_ptr`
+     * which is easier to pass around.
+     */
+    for (int i = 0; i < STORES_MAX; ++i) {
+        snprintf(store_addr_names[i],  NAME_LEN, "store_addr_%d",  i);
+        snprintf(store_width_names[i], NAME_LEN, "store_width_%d", i);
         snprintf(store_val32_names[i], NAME_LEN, "store_val32_%d", i);
-        hex_store_val32[i] = tcg_global_mem_new(tcg_env,
-            offsetof(CPUHexagonState, mem_log_stores[i].data32),
-            store_val32_names[i]);
-
         snprintf(store_val64_names[i], NAME_LEN, "store_val64_%d", i);
-        hex_store_val64[i] = tcg_global_mem_new_i64(tcg_env,
-            offsetof(CPUHexagonState, mem_log_stores[i].data64),
-            store_val64_names[i]);
+        store_addr_names_ptr[i]  = store_addr_names[i];
+        store_width_names_ptr[i] = store_width_names[i];
+        store_val32_names_ptr[i] = store_val32_names[i];
+        store_val64_names_ptr[i] = store_val64_names[i];
     }
+}
+
+void hexagon_translate_init(void)
+{
+    int i;
+
+    opcode_init();
+
     for (i = 0; i < VSTORES_MAX; i++) {
         snprintf(vstore_addr_names[i], NAME_LEN, "vstore_addr_%d", i);
         hex_vstore_addr[i] = tcg_global_mem_new(tcg_env,
@@ -1415,4 +1407,7 @@ void hexagon_translate_init(void)
             offsetof(CPUHexagonState, vstore_pending[i]),
             vstore_pending_names[i]);
     }
+
+    init_cpu_reg_names();
+    init_cpu_tcg_mappings(tcg_global_mappings, tcg_global_mapping_count);
 }
-- 
2.52.0

[PATCH v2 44/50] target/hexagon: Keep gen_slotval/check_noshuf for helper-to-tcg

[Anton Johansson via qemu development]

Makes sure gen_slotval() and check_noshuf() remains defined when
helper-to-tcg and idef-parser are both used.  gen_slotval() is needed
for creating a TCGv of the slot value fed to helpers (generated
helper-to-tcg code), and check_noshuf() is needed for helper definitions
used as input to helper-to-tcg.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 target/hexagon/genptr.c    | 2 +-
 target/hexagon/op_helper.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/target/hexagon/genptr.c b/target/hexagon/genptr.c
index 3f31037709..5acf237bf0 100644
--- a/target/hexagon/genptr.c
+++ b/target/hexagon/genptr.c
@@ -533,7 +533,7 @@ static inline void gen_store_conditional8(DisasContext *ctx,
     tcg_gen_movi_tl(hex_llsc_addr, ~0);
 }
 
-#ifndef CONFIG_HEXAGON_IDEF_PARSER
+#if !defined(CONFIG_HEXAGON_IDEF_PARSER) || defined(TARGET_HELPER_TO_TCG)
 static TCGv gen_slotval(DisasContext *ctx)
 {
     int slotval =
diff --git a/target/hexagon/op_helper.c b/target/hexagon/op_helper.c
index 3ce223caba..1a067cd1ce 100644
--- a/target/hexagon/op_helper.c
+++ b/target/hexagon/op_helper.c
@@ -488,7 +488,7 @@ void HELPER(probe_pkt_scalar_hvx_stores)(CPUHexagonState *env, int mask)
     }
 }
 
-#ifndef CONFIG_HEXAGON_IDEF_PARSER
+#if !defined(CONFIG_HEXAGON_IDEF_PARSER) || defined(CONFIG_HELPER_TO_TCG)
 /*
  * mem_noshuf
  * Section 5.5 of the Hexagon V67 Programmer's Reference Manual
-- 
2.52.0

[PATCH v2 45/50] target/hexagon: Emit annotations for helpers

[Anton Johansson via qemu development]

Adds the following LLVM_ANNOTATE attributes to helper functions generated
by Hexagon:

  1. "helper-to-tcg", to specify that a given helper functions should be
     translated, and;

  2. "immediate: ..." to make sure immediate arguments to helper
     functions remain immediates in the emitted TCG code (e.g. slot).

  3. "ptr-to-offset: ..." to make sure pointer arguments are treated as
     immediates representing an offset into the CPU state, needed to
     work with gvec.

Two functions are also added to hex_common.py, to firstly parse the
generated file containing all successfully translated helper functions,
and secondly to expose the indices of immediate and pointer (vector)
arguments to helper functions.  The latter is needed to generate the
input of helper-to-tcg.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 target/hexagon/gen_helper_funcs.py  | 17 ++++++++++-
 target/hexagon/gen_helper_protos.py |  2 +-
 target/hexagon/gen_tcg_funcs.py     |  2 +-
 target/hexagon/hex_common.py        | 45 +++++++++++++++++++++++------
 4 files changed, 54 insertions(+), 12 deletions(-)

diff --git a/target/hexagon/gen_helper_funcs.py b/target/hexagon/gen_helper_funcs.py
index 1629ebc0e1..7a3de21019 100755
--- a/target/hexagon/gen_helper_funcs.py
+++ b/target/hexagon/gen_helper_funcs.py
@@ -41,9 +41,23 @@ def gen_helper_function(f, tag, tagregs, tagimms):
     ret_type = hex_common.helper_ret_type(tag, regs).func_arg
 
     declared = []
-    for arg in hex_common.helper_args(tag, regs, imms):
+    helper_args, imm_inds, hvx_inds = hex_common.helper_args(tag, regs, imms)
+    for arg in helper_args:
         declared.append(arg.func_arg)
 
+    ## Specify that helpers should be translated by helper-to-tcg
+    f.write(f'QEMU_ANNOTATE("helper-to-tcg")\n')
+    ## Specify which arguments to the helper function should be treated as
+    ## immediate arguments
+    if len(imm_inds) > 0:
+        imm_inds_str = ','.join(str(i) for i in imm_inds)
+        f.write(f'QEMU_ANNOTATE("immediate: {imm_inds_str}")\n')
+    ## Specify which arguments to the helper function should be treated as
+    ## gvec vectors
+    if len(hvx_inds) > 0:
+        hvx_inds_str = ','.join(str(i) for i in hvx_inds)
+        f.write(f'QEMU_ANNOTATE("ptr-to-offset: {hvx_inds_str}")\n')
+
     arguments = ", ".join(declared)
     f.write(f"{ret_type} HELPER({tag})({arguments})\n")
     f.write("{\n")
@@ -51,6 +65,7 @@ def gen_helper_function(f, tag, tagregs, tagimms):
         f.write(hex_common.code_fmt(f"""\
             uint32_t EA;
         """))
+
     ## Declare the return variable
     if not hex_common.is_predicated(tag):
         for regtype, regid in regs:
diff --git a/target/hexagon/gen_helper_protos.py b/target/hexagon/gen_helper_protos.py
index 59c8bdd05c..769611e56c 100755
--- a/target/hexagon/gen_helper_protos.py
+++ b/target/hexagon/gen_helper_protos.py
@@ -36,7 +36,7 @@ def gen_helper_prototype(f, tag, tagregs, tagimms):
     ret_type = hex_common.helper_ret_type(tag, regs).proto_arg
     declared.append(ret_type)
 
-    for arg in hex_common.helper_args(tag, regs, imms):
+    for arg in hex_common.helper_args(tag, regs, imms)[0]:
         declared.append(arg.proto_arg)
 
     arguments = ", ".join(declared)
diff --git a/target/hexagon/gen_tcg_funcs.py b/target/hexagon/gen_tcg_funcs.py
index 6d5d99cee3..42b0064131 100755
--- a/target/hexagon/gen_tcg_funcs.py
+++ b/target/hexagon/gen_tcg_funcs.py
@@ -100,7 +100,7 @@ def gen_tcg_func(f, tag, regs, imms):
         if ret_type != "void":
             declared.append(ret_type)
 
-        for arg in hex_common.helper_args(tag, regs, imms):
+        for arg in hex_common.helper_args(tag, regs, imms)[0]:
             declared.append(arg.call_arg)
 
         arguments = ", ".join(declared)
diff --git a/target/hexagon/hex_common.py b/target/hexagon/hex_common.py
index d91a653c3d..3809749dfa 100755
--- a/target/hexagon/hex_common.py
+++ b/target/hexagon/hex_common.py
@@ -32,6 +32,7 @@
 tags = []  # list of all tags
 overrides = {}  # tags with helper overrides
 idef_parser_enabled = {}  # tags enabled for idef-parser
+helper_to_tcg_enabled = {}  # tags enabled for helper-to-tcg
 
 
 def is_sysemu_tag(tag):
@@ -307,6 +308,10 @@ def is_idef_parser_enabled(tag):
     return tag in idef_parser_enabled
 
 
+def is_helper_to_tcg_enabled(tag):
+    return tag in helper_to_tcg_enabled
+
+
 def is_hvx_insn(tag):
     return "A_CVI" in attribdict[tag]
 
@@ -349,6 +354,13 @@ def read_idef_parser_enabled_file(name):
         idef_parser_enabled = set(lines)
 
 
+def read_helper_to_tcg_enabled_file(name):
+    global helper_to_tcg_enabled
+    with open(name, "r") as helper_to_tcg_enabled_file:
+        lines = helper_to_tcg_enabled_file.read().strip().split("\n")
+        helper_to_tcg_enabled = set(lines)
+
+
 def is_predicated(tag):
     return "A_CONDEXEC" in attribdict[tag]
 
@@ -433,7 +445,7 @@ def hvx_off(self):
     def helper_proto_type(self):
         return "ptr"
     def helper_arg_type(self):
-        return "void *"
+        return "void * restrict"
     def helper_arg_name(self):
         return f"{self.reg_tcg()}_void"
 
@@ -767,7 +779,7 @@ def decl_tcg(self, f, tag, regno):
             const intptr_t {self.hvx_off()} =
                 {vreg_offset_func(tag)}(ctx, {self.reg_num}, 1, true);
         """))
-        if not skip_qemu_helper(tag):
+        if not skip_qemu_helper(tag) and not is_helper_to_tcg_enabled(tag):
             f.write(code_fmt(f"""\
                 TCGv_ptr {self.reg_tcg()} = tcg_temp_new_ptr();
                 tcg_gen_addi_ptr({self.reg_tcg()}, tcg_env, {self.hvx_off()});
@@ -792,7 +804,7 @@ def decl_tcg(self, f, tag, regno):
         f.write(code_fmt(f"""\
             const intptr_t {self.hvx_off()} = vreg_src_off(ctx, {self.reg_num});
         """))
-        if not skip_qemu_helper(tag):
+        if not skip_qemu_helper(tag) and not is_helper_to_tcg_enabled(tag):
             f.write(code_fmt(f"""\
                 TCGv_ptr {self.reg_tcg()} = tcg_temp_new_ptr();
                 tcg_gen_addi_ptr({self.reg_tcg()}, tcg_env, {self.hvx_off()});
@@ -833,7 +845,7 @@ def decl_tcg(self, f, tag, regno):
                              vreg_src_off(ctx, {self.reg_num}),
                              sizeof(MMVector), sizeof(MMVector));
         """))
-        if not skip_qemu_helper(tag):
+        if not skip_qemu_helper(tag) and not is_helper_to_tcg_enabled(tag):
             f.write(code_fmt(f"""\
                 TCGv_ptr {self.reg_tcg()} = tcg_temp_new_ptr();
                 tcg_gen_addi_ptr({self.reg_tcg()}, tcg_env, {self.hvx_off()});
@@ -862,7 +874,7 @@ def decl_tcg(self, f, tag, regno):
         f.write(code_fmt(f"""\
             const intptr_t {self.hvx_off()} = offsetof(CPUHexagonState, vtmp);
         """))
-        if not skip_qemu_helper(tag):
+        if not skip_qemu_helper(tag) and not is_helper_to_tcg_enabled(tag):
             f.write(code_fmt(f"""\
                 TCGv_ptr {self.reg_tcg()} = tcg_temp_new_ptr();
                 tcg_gen_addi_ptr({self.reg_tcg()}, tcg_env, {self.hvx_off()});
@@ -898,7 +910,7 @@ def decl_tcg(self, f, tag, regno):
             const intptr_t {self.hvx_off()} =
                 {vreg_offset_func(tag)}(ctx, {self.reg_num}, 2, true);
         """))
-        if not skip_qemu_helper(tag):
+        if not skip_qemu_helper(tag) and not is_helper_to_tcg_enabled(tag):
             f.write(code_fmt(f"""\
                 TCGv_ptr {self.reg_tcg()} = tcg_temp_new_ptr();
                 tcg_gen_addi_ptr({self.reg_tcg()}, tcg_env, {self.hvx_off()});
@@ -930,7 +942,7 @@ def decl_tcg(self, f, tag, regno):
                              vreg_src_off(ctx, {self.reg_num} ^ 1),
                              sizeof(MMVector), sizeof(MMVector));
         """))
-        if not skip_qemu_helper(tag):
+        if not skip_qemu_helper(tag) and not is_helper_to_tcg_enabled(tag):
             f.write(code_fmt(f"""\
                 TCGv_ptr {self.reg_tcg()} = tcg_temp_new_ptr();
                 tcg_gen_addi_ptr({self.reg_tcg()}, tcg_env, {self.hvx_off()});
@@ -957,7 +969,7 @@ def decl_tcg(self, f, tag, regno):
                              vreg_src_off(ctx, {self.reg_num} ^ 1),
                              sizeof(MMVector), sizeof(MMVector));
         """))
-        if not skip_qemu_helper(tag):
+        if not skip_qemu_helper(tag) and not is_helper_to_tcg_enabled(tag):
             f.write(code_fmt(f"""\
                 TCGv_ptr {self.reg_tcg()} = tcg_temp_new_ptr();
                 tcg_gen_addi_ptr({self.reg_tcg()}, tcg_env, {self.hvx_off()});
@@ -1280,8 +1292,13 @@ def helper_ret_type(tag, regs):
         raise Exception("numscalarresults > 1")
     return return_type
 
+
 def helper_args(tag, regs, imms):
     args = []
+    # Used to ensure immediates are passed translated as immediates by
+    # helper-to-tcg.
+    imm_indices = []
+    hvx_indices = []
 
     ## First argument is the CPU state
     if need_env(tag):
@@ -1302,16 +1319,20 @@ def helper_args(tag, regs, imms):
     for regtype, regid in regs:
         reg = get_register(tag, regtype, regid)
         if reg.is_written() and reg.is_hvx_reg():
+            hvx_indices.append(len(args))
             args.append(reg.helper_arg())
 
     ## Pass the source registers
     for regtype, regid in regs:
         reg = get_register(tag, regtype, regid)
         if reg.is_read() and not (reg.is_hvx_reg() and reg.is_readwrite()):
+            if reg.is_hvx_reg():
+                hvx_indices.append(len(args))
             args.append(reg.helper_arg())
 
     ## Pass the immediates
     for immlett, bits, immshift in imms:
+        imm_indices.append(len(args))
         args.append(HelperArg(
             "s32",
             f"tcg_constant_tl({imm_name(immlett)})",
@@ -1320,24 +1341,28 @@ def helper_args(tag, regs, imms):
 
     ## Other stuff the helper might need
     if need_pkt_has_multi_cof(tag):
+        imm_indices.append(len(args))
         args.append(HelperArg(
             "i32",
             "tcg_constant_tl(ctx->pkt.pkt_has_multi_cof)",
             "uint32_t pkt_has_multi_cof"
         ))
     if need_pkt_need_commit(tag):
+        imm_indices.append(len(args))
         args.append(HelperArg(
             "i32",
             "tcg_constant_tl(ctx->need_commit)",
             "uint32_t pkt_need_commit"
         ))
     if need_PC(tag):
+        imm_indices.append(len(args))
         args.append(HelperArg(
             "i32",
             "tcg_constant_tl(ctx->pkt.pc)",
             "target_ulong PC"
         ))
     if need_next_PC(tag):
+        imm_indices.append(len(args))
         args.append(HelperArg(
             "i32",
             "tcg_constant_tl(ctx->next_PC)",
@@ -1356,18 +1381,20 @@ def helper_args(tag, regs, imms):
             "uint32_t SP"
         ))
     if need_slot(tag):
+        imm_indices.append(len(args))
         args.append(HelperArg(
             "i32",
             "gen_slotval(ctx)",
             "uint32_t slotval"
         ))
     if need_part1(tag):
+        imm_indices.append(len(args))
         args.append(HelperArg(
             "i32",
             "tcg_constant_tl(insn->part1)"
             "uint32_t part1"
         ))
-    return args
+    return args, imm_indices, hvx_indices
 
 
 def parse_common_args(desc):
-- 
2.52.0

[PATCH v2 46/50] target/hexagon: Split probe_and_commit helper

[Anton Johansson via qemu development]

Split function into a slowpath such that the faster function can be
translated by helper-to-tcg.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 target/hexagon/helper.h    |  1 +
 target/hexagon/op_helper.c | 17 ++++++++++++++---
 2 files changed, 15 insertions(+), 3 deletions(-)

diff --git a/target/hexagon/helper.h b/target/hexagon/helper.h
index 033e5619d6..74d5b7ffe8 100644
--- a/target/hexagon/helper.h
+++ b/target/hexagon/helper.h
@@ -107,6 +107,7 @@ DEF_HELPER_4(probe_noshuf_load, void, env, i32, int, int)
 DEF_HELPER_2(probe_pkt_scalar_store_s0, void, env, int)
 DEF_HELPER_2(probe_hvx_stores, void, env, int)
 DEF_HELPER_2(probe_pkt_scalar_hvx_stores, void, env, int)
+DEF_HELPER_4(probe_and_commit, void, env, i32, s32, i64)
 
 #if !defined(CONFIG_USER_ONLY)
 DEF_HELPER_3(raise_stack_overflow, void, env, i32, i32)
diff --git a/target/hexagon/op_helper.c b/target/hexagon/op_helper.c
index 1a067cd1ce..c175cecb71 100644
--- a/target/hexagon/op_helper.c
+++ b/target/hexagon/op_helper.c
@@ -488,7 +488,8 @@ void HELPER(probe_pkt_scalar_hvx_stores)(CPUHexagonState *env, int mask)
     }
 }
 
-#if !defined(CONFIG_HEXAGON_IDEF_PARSER) || defined(CONFIG_HELPER_TO_TCG)
+#if !defined(CONFIG_HEXAGON_IDEF_PARSER) || defined(TARGET_HELPER_TO_TCG)
+
 /*
  * mem_noshuf
  * Section 5.5 of the Hexagon V67 Programmer's Reference Manual
@@ -500,8 +501,18 @@ static void check_noshuf(CPUHexagonState *env, bool pkt_has_scalar_store_s1,
                          uint32_t slot, target_ulong vaddr, int size,
                          uintptr_t ra)
 {
-    if (slot == 0 && pkt_has_scalar_store_s1 &&
-        ((env->slot_cancelled & (1 << 1)) == 0)) {
+    if (slot == 0 && pkt_has_scalar_store_s1) {
+        helper_probe_and_commit(env, vaddr, size, ra);
+    }
+}
+
+/*
+ *
+ */
+void HELPER(probe_and_commit)(CPUHexagonState *env, target_ulong vaddr,
+                              int size, uintptr_t ra)
+{
+    if ((env->slot_cancelled & (1 << 1)) == 0) {
         probe_read(env, vaddr, size, MMU_USER_IDX, ra);
         commit_store(env, 1, ra);
     }
-- 
2.52.0

[PATCH v2 47/50] target/hexagon: Use helper-to-tcg helper calls

[Anton Johansson via qemu development]

Replaces the default "helper-gen.h" header with those provided by
helper-to-tcg.  `gen_helper_*()` definitions are generated for all
helpers in the input module, regardless of whether they successfully
translated or not.  The generated `gen_helper_*()` calls into either
`emit_*()` or `tcg_gen_callN` depending on whether translation of that
helper was successful or not.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 target/hexagon/genptr.c    | 4 ++++
 target/hexagon/helper.h    | 8 ++++++++
 target/hexagon/translate.c | 8 ++++++++
 3 files changed, 20 insertions(+)

diff --git a/target/hexagon/genptr.c b/target/hexagon/genptr.c
index 5acf237bf0..d99580e815 100644
--- a/target/hexagon/genptr.c
+++ b/target/hexagon/genptr.c
@@ -20,7 +20,11 @@
 #include "internal.h"
 #include "tcg/tcg-op.h"
 #include "tcg/tcg-op-gvec.h"
+#if defined(TARGET_HELPER_TO_TCG)
+#include "helper-to-tcg-emitted.h"
+#else
 #include "exec/helper-gen.h"
+#endif
 #include "insn.h"
 #include "opcodes.h"
 #include "sys_macros.h"
diff --git a/target/hexagon/helper.h b/target/hexagon/helper.h
index 74d5b7ffe8..1796ef7144 100644
--- a/target/hexagon/helper.h
+++ b/target/hexagon/helper.h
@@ -133,3 +133,11 @@ DEF_HELPER_1(resched, void, env)
 DEF_HELPER_3(modify_ssr, void, env, i32, i32)
 DEF_HELPER_1(pending_interrupt, void, env)
 #endif
+
+/*
+ * Include generated helper-to-tcg support helpers, but not during IR
+ * generation since they won't have been emitted at that point.
+ */
+#if defined(TARGET_HELPER_TO_TCG) && !defined(HELPER_TO_TCG_IR_GEN)
+# include "helper-to-tcg-support-helpers.h"
+#endif
diff --git a/target/hexagon/translate.c b/target/hexagon/translate.c
index 870475010b..ab6e100a38 100644
--- a/target/hexagon/translate.c
+++ b/target/hexagon/translate.c
@@ -20,7 +20,15 @@
 #include "cpu.h"
 #include "tcg/tcg-op.h"
 #include "tcg/tcg-op-gvec.h"
+/*
+ * Include generated gen_helper_*() definitions if using helper-to-tcg, but
+ * only when not generating input IR since the header won't exist at that point.
+ */
+#if defined(TARGET_HELPER_TO_TCG) && !defined(HELPER_TO_TCG_IR_GEN)
+#include "helper-to-tcg-emitted.h"
+#else
 #include "exec/helper-gen.h"
+#endif
 #include "exec/helper-proto.h"
 #include "exec/translation-block.h"
 #include "accel/tcg/cpu-ldst.h"
-- 
2.52.0

[PATCH v2 48/50] target/hexagon: Manually call generated HVX instructions

[Anton Johansson via qemu development]

For HVX instructions that were successfully translated by helper-to-tcg,
emit calls to emit_*() "manually" from generate_*().  Recall that scalar
instructions translated by helper-to-tcg are automatically called by a
hook in tcg_gen_callN.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 target/hexagon/gen_tcg_funcs.py | 14 ++++++++
 target/hexagon/hex_common.py    | 58 +++++++++++++++++++++++++++++++++
 2 files changed, 72 insertions(+)

diff --git a/target/hexagon/gen_tcg_funcs.py b/target/hexagon/gen_tcg_funcs.py
index 42b0064131..ee3b183e7b 100755
--- a/target/hexagon/gen_tcg_funcs.py
+++ b/target/hexagon/gen_tcg_funcs.py
@@ -86,7 +86,21 @@ def gen_tcg_func(f, tag, regs, imms):
 
         arguments = ", ".join(["ctx", "ctx->insn", "&ctx->pkt"] + declared)
         f.write(f"    emit_{tag}({arguments});\n")
+    elif hex_common.is_helper_to_tcg_enabled(tag) and tag.startswith("V6_"):
+        ## For vector functions translated by helper-to-tcg we need to
+        ## manually call the emitted code.  All other instructions translated
+        ## are automatically called by the helper-functions dispatcher in
+        ## tcg_gen_callN.
+        declared = []
+        ## Handle registers
+        ret_type = hex_common.helper_ret_type(tag, regs).call_arg
+        if ret_type != "void":
+            declared.append(ret_type)
+        for arg in hex_common.helper_to_tcg_hvx_call_args(tag, regs, imms):
+            declared.append(arg)
 
+        arguments = ", ".join(declared)
+        f.write(f"    emit_{tag}({arguments});\n")
     elif hex_common.skip_qemu_helper(tag):
         if "A_FPOP" in hex_common.attribdict[tag]:
             f.write("    TCGv pkt_need_commit = ")
diff --git a/target/hexagon/hex_common.py b/target/hexagon/hex_common.py
index 3809749dfa..59fd563923 100755
--- a/target/hexagon/hex_common.py
+++ b/target/hexagon/hex_common.py
@@ -1293,6 +1293,60 @@ def helper_ret_type(tag, regs):
     return return_type
 
 
+def helper_to_tcg_hvx_call_args(tag, regs, imms):
+    args = []
+    # Used to ensure immediates are passed translated as immediates by
+    # helper-to-tcg.
+    imm_indices = []
+
+    ## First argument is the CPU state
+    if need_env(tag):
+        args.append("tcg_env")
+
+    ## For predicated instructions, we pass in the destination register
+    if is_predicated(tag):
+        for regtype, regid in regs:
+            reg = get_register(tag, regtype, regid)
+            if reg.is_writeonly() and not reg.is_hvx_reg():
+                args.append(reg.helper_arg().call_arg)
+
+    ## Pass the HVX destination registers
+    for regtype, regid in regs:
+        reg = get_register(tag, regtype, regid)
+        if reg.is_written() and reg.is_hvx_reg():
+            args.append(reg.hvx_off())
+
+    ## Pass the source registers
+    for regtype, regid in regs:
+        reg = get_register(tag, regtype, regid)
+        if reg.is_read() and not (reg.is_hvx_reg() and reg.is_readwrite()):
+            if reg.is_hvx_reg():
+                args.append(reg.hvx_off())
+            else:
+                args.append(reg.helper_arg().call_arg)
+
+    ## Pass the immediates
+    for immlett, bits, immshift in imms:
+        imm_indices.append(len(args))
+        args.append(f"{imm_name(immlett)}")
+
+    ## Other stuff the helper might need
+    if need_pkt_has_multi_cof(tag):
+        args.append("ctx->pkt->pkt_has_multi_cof")
+    if need_pkt_need_commit(tag):
+        args.append("ctx->need_commit")
+    if need_PC(tag):
+        args.append("ctx->pkt->pc")
+    if need_next_PC(tag):
+        args.append("ctx->next_PC")
+    if need_slot(tag):
+        args.append("gen_slotval(ctx)")
+    if need_part1(tag):
+        args.append("insn->part1")
+
+    return args
+
+
 def helper_args(tag, regs, imms):
     args = []
     # Used to ensure immediates are passed translated as immediates by
@@ -1406,6 +1460,8 @@ def parse_common_args(desc):
     parser.add_argument("out", help="output file")
     parser.add_argument("--idef-parser",
                         help="file of instructions translated by idef-parser")
+    parser.add_argument("--helper-to-tcg",
+                        help="file of instructions translated by helper-to-tcg")
     args = parser.parse_args()
     read_semantics_file(args.semantics)
     read_overrides_file(args.overrides)
@@ -1413,6 +1469,8 @@ def parse_common_args(desc):
     read_overrides_file(args.overrides_sys)
     if args.idef_parser:
         read_idef_parser_enabled_file(args.idef_parser)
+    if args.helper_to_tcg:
+        read_helper_to_tcg_enabled_file(args.helper_to_tcg)
     calculate_attribs()
     init_registers()
     return args
-- 
2.52.0

[PATCH v2 49/50] target/hexagon: Use idef-parser as a fallback

[Anton Johansson via qemu development]

Only generate input functions to idef-parser for instructions which
failed to be translated by helper-to-tcg, and which have no overrides.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 target/hexagon/gen_idef_parser_funcs.py | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/target/hexagon/gen_idef_parser_funcs.py b/target/hexagon/gen_idef_parser_funcs.py
index 32bce9b002..6bcc6e4bd0 100644
--- a/target/hexagon/gen_idef_parser_funcs.py
+++ b/target/hexagon/gen_idef_parser_funcs.py
@@ -49,10 +49,13 @@ def main():
     )
     parser.add_argument("semantics", help="semantics file")
     parser.add_argument("out", help="output file")
+    parser.add_argument("--helper-to-tcg", help="file of instructions translated by helper-to-tcg")
     args = parser.parse_args()
     hex_common.read_semantics_file(args.semantics)
     hex_common.calculate_attribs()
     hex_common.init_registers()
+    if args.helper_to_tcg:
+        hex_common.read_helper_to_tcg_enabled_file(args.helper_to_tcg)
     tagregs = hex_common.get_tagregs()
     tagimms = hex_common.get_tagimms()
 
@@ -62,6 +65,12 @@ def main():
         for tag in hex_common.tags:
             if hex_common.tag_ignore(tag):
                 continue
+            ## Skip instructions with overrides
+            if hex_common.skip_qemu_helper(tag):
+                continue
+            ## Skip instructions translated by helper-to-tcg
+            if hex_common.is_helper_to_tcg_enabled(tag):
+                continue
             ## Skip the priv instructions
             if "A_PRIV" in hex_common.attribdict[tag]:
                 continue
-- 
2.52.0

[PATCH v2 50/50] target/hexagon: Use helper-to-tcg

[Anton Johansson via qemu development]

Modifies meson.build to use helper-to-tcg for automatic translation of
helper functions.  Any helper functions with the "helper-to-tcg"
attribute will be automatically translated to TCG.

Order of code generation is changed, and helper functions are always
generated first, for all instructions.  Helper functions are needed as
input helper-to-tcg.  Next, input to idef-parser is generated for all
instructions that were not successfully translated by helper-to-tcg.

As such, a majority of instructions will be translated by helper-to-tcg,
and the remaining instructions fed through idef-parser can be reduced
moving forward.

Signed-off-by: Anton Johansson <anjo@rev.ng>
---
 configs/targets/hexagon-linux-user.mak |   1 +
 configs/targets/hexagon-softmmu.mak    |   1 +
 target/hexagon/meson.build             | 148 ++++++++++++++++++-------
 3 files changed, 110 insertions(+), 40 deletions(-)

diff --git a/configs/targets/hexagon-linux-user.mak b/configs/targets/hexagon-linux-user.mak
index 51fde5d60e..d441112f2e 100644
--- a/configs/targets/hexagon-linux-user.mak
+++ b/configs/targets/hexagon-linux-user.mak
@@ -4,3 +4,4 @@ TARGET_SYSTBL=syscall.tbl
 TARGET_SYSTBL_ABI=common,32,hexagon,time32,stat64,rlimit,renameat
 TARGET_LONG_BITS=32
 TARGET_NOT_USING_LEGACY_NATIVE_ENDIAN_API=y
+TARGET_HELPER_TO_TCG=y
diff --git a/configs/targets/hexagon-softmmu.mak b/configs/targets/hexagon-softmmu.mak
index a77c100f0c..bef72e8b2b 100644
--- a/configs/targets/hexagon-softmmu.mak
+++ b/configs/targets/hexagon-softmmu.mak
@@ -6,3 +6,4 @@ TARGET_LONG_BITS=32
 TARGET_NOT_USING_LEGACY_LDST_PHYS_API=y
 TARGET_NOT_USING_LEGACY_NATIVE_ENDIAN_API=y
 TARGET_NEED_FDT=y
+TARGET_HELPER_TO_TCG=y
diff --git a/target/hexagon/meson.build b/target/hexagon/meson.build
index 59cb09c107..fdacd61ccf 100644
--- a/target/hexagon/meson.build
+++ b/target/hexagon/meson.build
@@ -262,23 +262,125 @@ hexagon_softmmu_ss.add(files(
     'machine.c',
 ))
 
+helper_dep = [semantics_generated]
+helper_in = [semantics_generated, gen_tcg_h, gen_tcg_hvx_h, gen_tcg_sys_h]
+
+#
+# Step 5
+# We use Python scripts to generate the following files
+#     helper_protos_generated.h.inc
+#     helper_funcs_generated.c.inc
+#     analyze_funcs_generated.c.inc
 #
-# Step 4.5
+helper_protos_generated = custom_target(
+    'helper_protos_generated.h.inc',
+    output: 'helper_protos_generated.h.inc',
+    depends: helper_dep,
+    depend_files: [hex_common_py, gen_tcg_h, gen_tcg_hvx_h, gen_tcg_sys_h],
+    command: [python, files('gen_helper_protos.py'), helper_in, '@OUTPUT@'],
+)
+hexagon_ss.add(helper_protos_generated)
+
+helper_funcs_generated = custom_target(
+    'helper_funcs_generated.c.inc',
+    output: 'helper_funcs_generated.c.inc',
+    depends: helper_dep,
+    depend_files: [hex_common_py, gen_tcg_h, gen_tcg_hvx_h, gen_tcg_sys_h],
+    command: [python, files('gen_helper_funcs.py'), helper_in, '@OUTPUT@'],
+)
+hexagon_ss.add(helper_funcs_generated)
+
+analyze_funcs_generated = custom_target(
+    'analyze_funcs_generated.c.inc',
+    output: 'analyze_funcs_generated.c.inc',
+    depends: helper_dep,
+    depend_files: [hex_common_py, gen_tcg_h, gen_tcg_hvx_h, gen_tcg_sys_h],
+    command: [python, files('gen_analyze_funcs.py'), helper_in, '@OUTPUT@'],
+)
+hexagon_ss.add(analyze_funcs_generated)
+
+#
+# Step 6
+# If enabled, run helper-to-tcg to attempt to translate any remaining
+# helper functions, producing:
+#     helper-to-tcg-emitted.c
+#     helper-to-tcg-emitted.h
+#     helper-to-tcg-enabled
+#     helper-to-tcg-log
+#
+
+idef_command_extra = []
+idef_dep_extra = []
+if helper_to_tcg.found()
+    helper_to_tcg_get_llvm_ir_cmd = helper_to_tcg.get_variable('get_llvm_ir_cmd')
+    helper_to_tcg_pipeline = helper_to_tcg.get_variable('pipeline')
+
+    helper_to_tcg_input_files = [
+        meson.current_source_dir() / 'op_helper.c',
+        meson.current_source_dir() / 'translate.c',
+        meson.current_source_dir() / 'reg_fields.c',
+        meson.current_source_dir() / 'arch.c',
+    ]
+
+    ll = custom_target('to-ll',
+        input: helper_to_tcg_input_files,
+        output:'helper-to-tcg-input.ll',
+        depends: [helper_funcs_generated, helper_protos_generated, libqemuutil],
+        command: helper_to_tcg_get_llvm_ir_cmd + ['-o', '@OUTPUT@', '@INPUT@', '--target-path', 'target/hexagon']
+    )
+
+    helper_to_tcg_target = custom_target('helper-to-tcg-hexagon',
+        output: ['helper-to-tcg-emitted.c',
+                 'helper-to-tcg-emitted.h',
+                 'helper-to-tcg-enabled'],
+        input: [ll],
+        depends: [helper_to_tcg_pipeline, analyze_funcs_generated, helper_funcs_generated, helper_protos_generated],
+        command: [helper_to_tcg_pipeline,
+                  '--temp-vector-block=tmp_VRegs',
+                  '--max-vector-temp-bytes=1024',
+                  '--max-vector-instructions=16',
+                  '--mmu-index-function=get_tb_mmu_index',
+                  '--tcg-global-mappings=tcg_global_mappings',
+                  '--output-source=@OUTPUT0@',
+                  '--output-header=@OUTPUT1@',
+                  '--output-enabled=@OUTPUT2@',
+                  '@INPUT@']
+    )
+
+    hexagon_ss.add(helper_to_tcg_target)
+
+    # List of instructions for which TCG generation was successful
+    generated_tcg_list = helper_to_tcg_target[2].full_path()
+
+    # Setup dependencies for idef-parser
+    idef_dep_extra += helper_to_tcg_target
+    idef_command_extra += ['--helper-to-tcg', generated_tcg_list]
+
+    # Setup input and dependencies for the final step, this depends on whether
+    helper_dep += [helper_to_tcg_target]
+    helper_in += ['--helper-to-tcg', generated_tcg_list]
+endif
+
+
+
+#
+# Step 6
 # We use flex/bison based idef-parser to generate TCG code for a lot
 # of instructions. idef-parser outputs
 #     idef-generated-emitter.c
 #     idef-generated-emitter.h.inc
 #     idef-generated-enabled-instructions
 #
+
 idef_parser_enabled = get_option('hexagon_idef_parser')
 if idef_parser_enabled and ('hexagon-linux-user' in target_dirs or
                             'hexagon-softmmu' in target_dirs)
     idef_parser_input_generated = custom_target(
         'idef_parser_input.h.inc',
         output: 'idef_parser_input.h.inc',
-        depends: [semantics_generated],
+        depends: [semantics_generated] + idef_dep_extra,
         depend_files: [hex_common_py],
-        command: [python, files('gen_idef_parser_funcs.py'), semantics_generated, '@OUTPUT@'],
+        command: [python, files('gen_idef_parser_funcs.py'), semantics_generated, '@OUTPUT@'] + idef_command_extra
     )
 
     compiler = meson.get_compiler('c').cmd_array()
@@ -347,40 +449,15 @@ if idef_parser_enabled and ('hexagon-linux-user' in target_dirs or
 
     # Setup input and dependencies for the next step, this depends on whether or
     # not idef-parser is enabled
-    helper_dep = [semantics_generated, idef_generated_tcg_c, idef_generated_tcg]
-    helper_in = [semantics_generated, gen_tcg_h, gen_tcg_hvx_h, gen_tcg_sys_h, '--idef-parser', idef_generated_list]
-else
-    # Setup input and dependencies for the next step, this depends on whether or
-    # not idef-parser is enabled
-    helper_dep = [semantics_generated]
-    helper_in = [semantics_generated, gen_tcg_h, gen_tcg_hvx_h, gen_tcg_sys_h]
+    helper_dep += [idef_generated_tcg_c, idef_generated_tcg]
+    helper_in += ['--idef-parser', idef_generated_list]
 endif
 
 #
-# Step 5
+# Step 7
 # We use Python scripts to generate the following files
-#     helper_protos_generated.h.inc
-#     helper_funcs_generated.c.inc
 #     tcg_funcs_generated.c.inc
 #
-helper_protos_generated = custom_target(
-    'helper_protos_generated.h.inc',
-    output: 'helper_protos_generated.h.inc',
-    depends: helper_dep,
-    depend_files: [hex_common_py, gen_tcg_h, gen_tcg_hvx_h, gen_tcg_sys_h],
-    command: [python, files('gen_helper_protos.py'), helper_in, '@OUTPUT@'],
-)
-hexagon_ss.add(helper_protos_generated)
-
-helper_funcs_generated = custom_target(
-    'helper_funcs_generated.c.inc',
-    output: 'helper_funcs_generated.c.inc',
-    depends: helper_dep,
-    depend_files: [hex_common_py, gen_tcg_h, gen_tcg_hvx_h, gen_tcg_sys_h],
-    command: [python, files('gen_helper_funcs.py'), helper_in, '@OUTPUT@'],
-)
-hexagon_ss.add(helper_funcs_generated)
-
 tcg_funcs_generated = custom_target(
     'tcg_funcs_generated.c.inc',
     output: 'tcg_funcs_generated.c.inc',
@@ -390,14 +467,5 @@ tcg_funcs_generated = custom_target(
 )
 hexagon_ss.add(tcg_funcs_generated)
 
-analyze_funcs_generated = custom_target(
-    'analyze_funcs_generated.c.inc',
-    output: 'analyze_funcs_generated.c.inc',
-    depends: helper_dep,
-    depend_files: [hex_common_py, gen_tcg_h, gen_tcg_hvx_h, gen_tcg_sys_h],
-    command: [python, files('gen_analyze_funcs.py'), helper_in, '@OUTPUT@'],
-)
-hexagon_ss.add(analyze_funcs_generated)
-
 target_arch += {'hexagon': hexagon_ss}
 target_system_arch += {'hexagon': hexagon_softmmu_ss}
-- 
2.52.0

Re: [PATCH v2 41/50] target/hexagon: Add get_tb_mmu_index()

[Philippe Mathieu-Daudé]

Hi Anton,

On 30/7/26 05:10, Anton Johansson via qemu development wrote:
> Adds a functions to return the current mmu index given tb_flags of the
> current translation block.  Required by helper-to-tcg in order to
> retrieve the mmu index for memory operations without changing the
> signature of helper functions.
> 
> Signed-off-by: Anton Johansson <anjo@rev.ng>
> ---
>   target/hexagon/translate.c | 12 ++++++++++++
>   target/hexagon/translate.h |  4 ++++
>   2 files changed, 16 insertions(+)
> 
> diff --git a/target/hexagon/translate.c b/target/hexagon/translate.c
> index 199b4f8c2e..bfd84bba4e 100644
> --- a/target/hexagon/translate.c
> +++ b/target/hexagon/translate.c
> @@ -128,6 +128,18 @@ intptr_t ctx_tmp_vreg_off(DisasContext *ctx, int regnum,
>       return offset;
>   }
>   
> +#if defined(TARGET_HELPER_TO_TCG)
> +/*
> + * Returns the current mmu index given tb_flags of the current translation
> + * block.  Required by helper-to-tcg in order to retrieve the mmu index for
> + * memory operations without changing the signature of helper functions.
> + */

No need to guard for TARGET_HELPER_TO_TCG IMO, simply add
and use in hexagon_tr_init_disas_context().

Dropping the "Required by helper-to-tcg ..." comment, the
TARGET_HELPER_TO_TCG guard and using in init_disas_context:

Reviewed-by: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>

> +int get_tb_mmu_index(uint32_t flags)
> +{
> +    return FIELD_EX32(flags, TB_FLAGS, MMU_INDEX);
> +}
> +#endif
> +
>   static void gen_exception(int excp, uint32_t PC)
>   {
>       gen_helper_raise_exception(tcg_env, tcg_constant_i32(excp),
> diff --git a/target/hexagon/translate.h b/target/hexagon/translate.h
> index 2fca157553..6cf3b2fe3b 100644
> --- a/target/hexagon/translate.h
> +++ b/target/hexagon/translate.h
> @@ -345,4 +345,8 @@ FIELD(PROBE_PKT_SCALAR_HVX_STORES, MMU_IDX,        5, 2)
>   
>   void gen_framecheck(DisasContext *ctx, TCGv_i32 addr, TCGv_i32 ea);
>   
> +#if defined(TARGET_HELPER_TO_TCG)
> +int get_tb_mmu_index(uint32_t flags);
> +#endif

Re: [PATCH v2 46/50] target/hexagon: Split probe_and_commit helper

[Philippe Mathieu-Daudé]

On 30/7/26 05:10, Anton Johansson via qemu development wrote:
> Split function into a slowpath such that the faster function can be
> translated by helper-to-tcg.
> 
> Signed-off-by: Anton Johansson <anjo@rev.ng>
> ---
>   target/hexagon/helper.h    |  1 +
>   target/hexagon/op_helper.c | 17 ++++++++++++++---
>   2 files changed, 15 insertions(+), 3 deletions(-)
> 
> diff --git a/target/hexagon/helper.h b/target/hexagon/helper.h
> index 033e5619d6..74d5b7ffe8 100644
> --- a/target/hexagon/helper.h
> +++ b/target/hexagon/helper.h
> @@ -107,6 +107,7 @@ DEF_HELPER_4(probe_noshuf_load, void, env, i32, int, int)
>   DEF_HELPER_2(probe_pkt_scalar_store_s0, void, env, int)
>   DEF_HELPER_2(probe_hvx_stores, void, env, int)
>   DEF_HELPER_2(probe_pkt_scalar_hvx_stores, void, env, int)
> +DEF_HELPER_4(probe_and_commit, void, env, i32, s32, i64)
>   
>   #if !defined(CONFIG_USER_ONLY)
>   DEF_HELPER_3(raise_stack_overflow, void, env, i32, i32)
> diff --git a/target/hexagon/op_helper.c b/target/hexagon/op_helper.c
> index 1a067cd1ce..c175cecb71 100644
> --- a/target/hexagon/op_helper.c
> +++ b/target/hexagon/op_helper.c
> @@ -488,7 +488,8 @@ void HELPER(probe_pkt_scalar_hvx_stores)(CPUHexagonState *env, int mask)
>       }
>   }
>   
> -#if !defined(CONFIG_HEXAGON_IDEF_PARSER) || defined(CONFIG_HELPER_TO_TCG)
> +#if !defined(CONFIG_HEXAGON_IDEF_PARSER) || defined(TARGET_HELPER_TO_TCG)
> +
>   /*
>    * mem_noshuf
>    * Section 5.5 of the Hexagon V67 Programmer's Reference Manual
> @@ -500,8 +501,18 @@ static void check_noshuf(CPUHexagonState *env, bool pkt_has_scalar_store_s1,
>                            uint32_t slot, target_ulong vaddr, int size,
>                            uintptr_t ra)
>   {
> -    if (slot == 0 && pkt_has_scalar_store_s1 &&
> -        ((env->slot_cancelled & (1 << 1)) == 0)) {
> +    if (slot == 0 && pkt_has_scalar_store_s1) {
> +        helper_probe_and_commit(env, vaddr, size, ra);
> +    }
> +}
> +
> +/*
> + *
> + */

Either fill the comment block, or drop it :) Otherwise:
Reviewed-by: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>

> +void HELPER(probe_and_commit)(CPUHexagonState *env, target_ulong vaddr,
> +                              int size, uintptr_t ra)
> +{
> +    if ((env->slot_cancelled & (1 << 1)) == 0) {
>           probe_read(env, vaddr, size, MMU_USER_IDX, ra);
>           commit_store(env, 1, ra);
>       }

Re: [PATCH v2 05/50] tcg: Expose tcg_gen_ussub_sat()

[Philippe Mathieu-Daudé]

On 30/7/26 05:09, Anton Johansson via qemu development wrote:
> Maps nicely to LLVMs usub.sat instrinsic.
> 
> Signed-off-by: Anton Johansson <anjo@rev.ng>
> ---
>   include/tcg/tcg-op-common.h | 2 ++
>   tcg/tcg-op-gvec.c           | 4 ++--
>   2 files changed, 4 insertions(+), 2 deletions(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>

Re: [PATCH v2 01/50] accel/tcg: Add bitreverse and funnel-shift runtime helper functions

[Philippe Mathieu-Daudé]

On 30/7/26 05:09, Anton Johansson via qemu development wrote:
> Adds necessary helper functions for mapping LLVM IR onto TCG.
> Specifically, helpers corresponding to the bitreverse and funnel-shift
> intrinsics in LLVM.
> 
> Note: these may be converted to more efficient implementations in the
> future, but for the time being it allows helper-to-tcg to support a
> wider subset of LLVM IR.
> 
> Signed-off-by: Anton Johansson <anjo@rev.ng>
> --
> TODO: `tcg_gen_*()` variants will be added in the next version, it
> slipped through (Richard).
> ---
>   accel/tcg/tcg-runtime.c | 28 ++++++++++++++++++++++++++++
>   accel/tcg/tcg-runtime.h |  6 ++++++
>   2 files changed, 34 insertions(+)


> +uint32_t HELPER(bitreverse8_i32)(uint32_t x)
> +{
> +  return revbit8((uint8_t) x);
> +}
> +
> +uint32_t HELPER(bitreverse16_i32)(uint32_t x)
> +{
> +  return revbit16((uint16_t) x);
> +}
> +
> +uint32_t HELPER(bitreverse32_i32)(uint32_t x)
> +{
> +  return revbit32(x);
> +}

Please split in 2 patches, adding bitreverse family in
one, and fshl_i64 in another.

For bitreverse:
Reviewed-by: Philippe Mathieu-Daudé <philmd@oss.qualcomm.com>

>   /* 64-bit helpers */
>   
> +uint64_t HELPER(fshl_i64)(uint64_t a, uint64_t b, uint64_t c)

Please use clearer hi/lo/sh/val/ret instead of a/b/c/d/shift:

   HELPER(fshl_i64)(uint64_t hi, uint64_t lo, uint64_t sh)

> +{
> +    Int128 d = int128_make128(b, a);
> +    Int128 shift = int128_lshift(d, c);
> +    return int128_gethi(shift);
> +}

Re: [PATCH v2 01/50] accel/tcg: Add bitreverse and funnel-shift runtime helper functions

[Richard Henderson]

On 7/29/26 20:09, Anton Johansson via qemu development wrote:
> +uint64_t HELPER(fshl_i64)(uint64_t a, uint64_t b, uint64_t c)
> +{
> +    Int128 d = int128_make128(b, a);
> +    Int128 shift = int128_lshift(d, c);
> +    return int128_gethi(shift);
> +}

The existing similar opcode is extract2, performing a constant right double-word shift. 
I'm open to adding a variable version, and adding left-shift, but we shouldn't have two 
names for the same operation.

Plus Phil's comment about not doing two things in the same patch.

r~

Re: [PATCH v2 02/50] accel/tcg: Add getpc helper

[Richard Henderson]

On 7/29/26 20:09, Anton Johansson via qemu development wrote:
> Introduces a helper function to return the current pc (returnaddress of
> the helper), this allows helper-to-tcg to correctly translate nested
> helper functions that checks for faulting memory operations.
> 
> This is useful as an optimization where a helper functions has a
> commonly taken fast path that doesn't fault e.g.
> 
>    void HELPER(outer)(...)
>    {
>        ... // non faulting operations
>        if (some_uncommon_condition) {
>            helper_inner(..., GETPC())
>        }
>    }
> 
> the outer helper along with the condition can then be emitted as TCG,
> and helper_inner() will be emitted as a gen_helper_inner(, ra), where ra
> is the result of gen_helper_getpc().
> 
> NOTE: This is not ideal since we're introducing extra helper calls,
>        and the solution doesn't deal with the "inner" function not being
>        a helper.  A better solution and what we'll probably do in the
>        next version of the patchset is to instead emit a helper
>        definition for the "inner" function that uses GETPC().

This is both confusing and wrong.

If you're transforming outer to hot tcg + cold helper, the cold helper would *still* use 
GETPC(), which would still produce an address within the jit code block for the guest 
instruction.


r~