[PATCH RFC v1 16/20] KVM: x86: Decode REX2 prefix in the emulator

Chang S. Bae
2025-11-10T18:25:04Z

Extend the instruction emulator to recognize and interpret the REX2
prefix byte. Also, detect and flag invalid prefix sequences after a REX2
prefix.

In the existing prefix-decoding loop,
* The loop exits when the first non-prefix byte is encountered.
* Any non-REX prefix clears previously recorded REX information.

For REX2, however, once a REX2 prefix is encountered, most subsequent
prefixes are invalid. So, each subsequent prefix needs to be validated
before continuing the loop.

Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
---
RFC note:
The REX2 decoding itself is straightforward. The additional logic is
mainly to detect and handle invalid prefix sequences. If this seems
excessive, there is a chance to cut off this check since VMX would raise
'#UD' on such cases anyway.
---
arch/x86/kvm/emulate.c | 24 +++++++++++++++++++++++-
1 file changed, 23 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c
index 9bd61ea496e5..f9381a4055d6 100644
--- a/arch/x86/kvm/emulate.c
+++ b/arch/x86/kvm/emulate.c
@@ -4844,7 +4844,7 @@ int x86_decode_insn(struct x86_emulate_ctxt *ctxt, void *insn, int insn_len, int
ctxt->op_bytes = def_op_bytes;
ctxt->ad_bytes = def_ad_bytes;

- /* Legacy prefixes. */
+ /* Legacy and REX/REX2 prefixes. */
for (;;) {
switch (ctxt->b = insn_fetch(u8, ctxt)) {
case 0x66: /* operand-size override */
@@ -4887,9 +4887,20 @@ int x86_decode_insn(struct x86_emulate_ctxt *ctxt, void *insn, int insn_len, int
case 0x40 ... 0x4f: /* REX */
if (mode != X86EMUL_MODE_PROT64)
goto done_prefixes;
+ if (ctxt->rex_prefix == REX2_PREFIX)
+ break;
ctxt->rex_prefix = REX_PREFIX;
ctxt->rex.raw = 0x0f & ctxt->b;
continue;
+ case 0xd5: /* REX2 */
+ if (mode != X86EMUL_MODE_PROT64)
+ goto done_prefixes;
+ if (ctxt->rex_prefix == REX2_PREFIX &&
+ ctxt->rex.bits.m0 == 0)
+ break;
+ ctxt->rex_prefix = REX2_PREFIX;
+ ctxt->rex.raw = insn_fetch(u8, ctxt);
+ continue;
case 0xf0: /* LOCK */
ctxt->lock_prefix = 1;
break;
@@ -4901,6 +4912,17 @@ int x86_decode_insn(struct x86_emulate_ctxt *ctxt, void *insn, int insn_len, int
goto done_prefixes;
}

+ if (ctxt->rex_prefix == REX2_PREFIX) {
+ /*
+ * A legacy or REX prefix following a REX2 prefix
+ * forms an invalid byte sequences. Likewise,
+ * a second REX2 prefix following a REX2 prefix
+ * with M0=0 is invalid.
+ */
+ ctxt->rex_prefix = REX2_INVALID;
+ goto done_prefixes;
+ }
+
/* Any legacy prefix after a REX prefix nullifies its effect. */
ctxt->rex_prefix = REX_NONE;
ctxt->rex.raw = 0;
--
2.51.0