Re: [PATCH RFC v1 16/20] KVM: x86: Decode REX2 prefix in the emulator

Paolo Bonzini
2025-11-11T17:56:03Z

On 11/10/25 19:01, Chang S. Bae wrote:
> Extend the instruction emulator to recognize and interpret the REX2
> prefix byte. Also, detect and flag invalid prefix sequences after a REX2
> prefix.
>
> In the existing prefix-decoding loop,
> * The loop exits when the first non-prefix byte is encountered.
> * Any non-REX prefix clears previously recorded REX information.
>
> For REX2, however, once a REX2 prefix is encountered, most subsequent
> prefixes are invalid. So, each subsequent prefix needs to be validated
> before continuing the loop.
>
> Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
> ---
> RFC note:
> The REX2 decoding itself is straightforward. The additional logic is
> mainly to detect and handle invalid prefix sequences. If this seems
> excessive, there is a chance to cut off this check since VMX would raise
> '#UD' on such cases anyway.
> ---
> arch/x86/kvm/emulate.c | 24 +++++++++++++++++++++++-
> 1 file changed, 23 insertions(+), 1 deletion(-)
>
> diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c
> index 9bd61ea496e5..f9381a4055d6 100644
> --- a/arch/x86/kvm/emulate.c
> +++ b/arch/x86/kvm/emulate.c
> @@ -4844,7 +4844,7 @@ int x86_decode_insn(struct x86_emulate_ctxt *ctxt, void *insn, int insn_len, int
> ctxt->op_bytes = def_op_bytes;
> ctxt->ad_bytes = def_ad_bytes;
>
> - /* Legacy prefixes. */
> + /* Legacy and REX/REX2 prefixes. */
> for (;;) {
> switch (ctxt->b = insn_fetch(u8, ctxt)) {
> case 0x66: /* operand-size override */
> @@ -4887,9 +4887,20 @@ int x86_decode_insn(struct x86_emulate_ctxt *ctxt, void *insn, int insn_len, int
> case 0x40 ... 0x4f: /* REX */
> if (mode != X86EMUL_MODE_PROT64)
> goto done_prefixes;
> + if (ctxt->rex_prefix == REX2_PREFIX)
> + break;
> ctxt->rex_prefix = REX_PREFIX;
> ctxt->rex.raw = 0x0f & ctxt->b;
> continue;
> + case 0xd5: /* REX2 */
> + if (mode != X86EMUL_MODE_PROT64)
> + goto done_prefixes;
Here you should also check

if (ctxt->rex_prefix == REX_PREFIX) {
ctxt->rex_prefix = REX2_INVALID;
goto done_prefixes;
}

> + if (ctxt->rex_prefix == REX2_PREFIX &&
> + ctxt->rex.bits.m0 == 0)
> + break;
> + ctxt->rex_prefix = REX2_PREFIX;
> + ctxt->rex.raw = insn_fetch(u8, ctxt);
> + continue;
After REX2 always comes the main opcode byte, so you can "goto
done_prefixes" here. Or even jump here already; in pseudocode:

ctxt->b = insn_fetch(u8, ctxt);
if (rex2 & REX_M)
goto decode_twobyte;
else
goto decode_onebyte;

...

if (ctxt->b == 0x0f) {
decode_twobyte:
...
if (ctxt->b == 0x38 && ctxt->rex_prefix != REX2_PREFIX)
...
} else {
decode_onebyte:
...
}

> case 0xf0: /* LOCK */
> ctxt->lock_prefix = 1;
> break;
> @@ -4901,6 +4912,17 @@ int x86_decode_insn(struct x86_emulate_ctxt *ctxt, void *insn, int insn_len, int
> goto done_prefixes;
> }
>
> + if (ctxt->rex_prefix == REX2_PREFIX) {
> + /*
> + * A legacy or REX prefix following a REX2 prefix
> + * forms an invalid byte sequences. Likewise,
> + * a second REX2 prefix following a REX2 prefix
> + * with M0=0 is invalid.
> + */
> + ctxt->rex_prefix = REX2_INVALID;
> + goto done_prefixes;
> + }

... and this is not needed.

Paolo

> /* Any legacy prefix after a REX prefix nullifies its effect. */
> ctxt->rex_prefix = REX_NONE;
> ctxt->rex.raw = 0;