Re: [kvm-unit-tests PATCH v3 17/17] x86/cet: Add testcases to verify KVM rejects emulation of CET instructions

[Sean Christopherson <seanjc@google.com>]

On Fri, Nov 14, 2025, Mathias Krause wrote:
> On 14.11.25 01:12, Sean Christopherson wrote:
> > +#define SHSTK_TEST_UNSUPPORTED_INSTRUCTION(insn)			\
> > +do {									\
> > +	uint8_t vector = __CET_TEST_UNSUPPORTED_INSTRUCTION(insn);	\
> > +									\
> > +	report(vector == UD_VECTOR, "Wanted #UD on %s, got %s",		\
> > +	       insn, exception_mnemonic(vector));			\
> > +} while (0)
> > +
> > +/*
> > + * Treat IRET as unsupported with IBT even though the minimal interactions with
> > + * IBT _could_ be easily emulated by KVM, as KVM doesn't support emulating IRET
> > + * outside of Real Mode.
> > + */
> > +#define CET_TEST_UNSUPPORTED_INSTRUCTIONS(CET)				\
> > +do {									\
> > +	CET##_TEST_UNSUPPORTED_INSTRUCTION("callq *%%rax");		\
> > +	CET##_TEST_UNSUPPORTED_INSTRUCTION("lcall *%0");		\
> 
> Maybe spell out that this is a 32bit far call ("lcalll *%0"), as only
> these are supported on AMD *and* Intel? (Intel can do 64bit ones too.)
> 
> > +	CET##_TEST_UNSUPPORTED_INSTRUCTION("syscall");			\
> > +	CET##_TEST_UNSUPPORTED_INSTRUCTION("sysenter");			\
> > +	CET##_TEST_UNSUPPORTED_INSTRUCTION("iretq");			\
> 
> These, if emulated, would be rather disastrous to the test. :D
> - SYSCALL messes badly with the register state.
> - SYSENTER should #UD in long mode on AMD anyway but Intel allows it.
>   However, it'll likely #GP early, prior to messing with register state,
>   as IA32_SYSENTER_CS doesn't get initialized as the CPU expects it to
>   be.
> - The stack frame for IRETQ isn't properly set up (lacks CS, RFLAGS, SS,
>   RSP) but even if it'll #GP early for the non-canonical address, it'll
>   have modified RSP, making the recovery "pop %rax" pop an unrelated
>   stack word, likely making the code crash soon after.
> 
> So these instructions really *rely* on KVM to #UD on these early.
> Dunno, but it feels like the tests should be made more robust wrt.
> possible future emulation by KVM?

I don't necessarily disagree, but I'm not convinced it'd be worth the effort to
properly set everything up _and_ keep it functional.  Unless KVM is buggy, the
fallback will be completely untested.  I hate user-hostile tests as much as the
next person (probably more), but for this one my vote is to punt and cross our
fingers that KVM never screws up.

> > +} while (0)
> > +
> > +static uint64_t cet_shstk_emulation(void)
> > +{
> > +	CET_TEST_UNSUPPORTED_INSTRUCTIONS(SHSTK);
> > +
> > +	SHSTK_TEST_UNSUPPORTED_INSTRUCTION("call 1f");
> > +	SHSTK_TEST_UNSUPPORTED_INSTRUCTION("retq");
> > +	SHSTK_TEST_UNSUPPORTED_INSTRUCTION("retq $10");
> > +	SHSTK_TEST_UNSUPPORTED_INSTRUCTION("lretq");
> > +	SHSTK_TEST_UNSUPPORTED_INSTRUCTION("lretq $10");
> > +
> > +	/* Do a handful of JMPs to verify they aren't impacted by SHSTK. */
> > +	asm volatile(KVM_FEP "jmp 1f\n\t"
> > +		     "1:\n\t"
> > +		     KVM_FEP "lea 2f(%%rip), %%rax\n\t"
> > +		     KVM_FEP "jmp *%%rax\n\t"
>                               ^^^^^^^^^^
> Fortunately, this indirect branch runs only with shadow stack enabled,
> no IBT. If it would, than below misses an ENDBR64.

50/50 as to whether I noticed that or just got lucky :-)

>                      vvv
> > +		     "2:\n\t"> +		     KVM_FEP "push $" xstr(USER_CS) "\n\t"
> > +		     KVM_FEP "lea 3f(%%rip), %%rax\n\t"
> > +		     KVM_FEP "push %%rax\n\t"
> > +		     /* Manually encode ljmpq, which gas doesn't recognize :-( */
> 
> LJMPQ is actually only supported on Intel systems. AMD doesn't support
> it and one has to emulate it via "push $cs; push $rip; lretq".

Well that explains why gas wasn't being very nice to me.  I'll tweak the comment.

> Dunno if it's a feature or bug of KVM to emulate LJMPQ fine on AMD -- if it
> does, that is!

Bug?  Sort of?  KVM won't emulate it on #UD, so it's only very limited emulation.
E.g. the guest would have to force emulation, put the jump source in emulated MMIO,
or play TLB games to get KVM to emulate

> > +		     KVM_FEP ".byte 0x48\n\t"
> > +		     "ljmpl *(%%rsp)\n\t"
> > +		     "3:\n\t"
> > +		     KVM_FEP "pop %%rax\n\t"
> > +		     KVM_FEP "pop %%rax\n\t"
> > +		     ::: "eax");
> > +
> > +	return 0;
> > +}
> > +
> 
> > +/*
> > + * Don't invoke printf() or report() in the IBT testcase, as it will likely
> > + * generate an indirect branch without an endbr64 annotation and thus #CP.
> 
> Uhm, x86/Makefile.x86_64 has this:
> 
> fcf_protection_full := $(call cc-option, -fcf-protection=full,)
> COMMON_CFLAGS += -mno-red-zone -mno-sse -mno-sse2 $(fcf_protection_full)
> [...]
> ifneq ($(fcf_protection_full),)
> tests += $(TEST_DIR)/cet.$(exe)
> endif
> 
> So all code that needs it should have a leading ENDBR64 or the cet test
> wouldn't be part of the test suite.
> 
> However, I also notice it isn't working...
> 
> [digging in, hours later...]
> 
> After tweaking the exception handling to be able to tweak the IBT
> tracker state (which would be stuck at WAIT_FOR_ENDBRANCH for a missing
> ENDBR64), it still wouldn't work for me. Bummer! Further instrumentation
> showed, the code triggered within exception_mnemonic() which caused only
> more question marks -- it's just a simple switch, right? Though, looking
> at the disassembly made it crystal clear:
> 
> 000000000040707c <exception_mnemonic>:
>   40707c:       endbr64
>   407080:       cmp    $0x1e,%edi
>   407083:       ja     407117 <exception_mnemonic+0x9b>
>   407089:       mov    %edi,%edi
>   40708b:       notrack jmp *0x4107e0(,%rdi,8)
>     ::
>   4070b1:       mov    $0x411c7c,%eax	# <-- #CP(3) here
> 
> The switch block caused gcc to emit a jump table and an indirect jump
> for it (to 40708b). The jump is prefixed with 'notrack' to keep the size
> of the jump target stubs small, allowing for omitting the ENDBR64
> instruction. However, the IBT test code doesn't enable that in the MSR,
> leading the CPU to raising a #CP(3) for this indirect jump. D'oh!
> 
> So we can either disable the generation of jump tables in gcc
> (-fno-jump-tables) or just allow the 'notrack' handling. I guess the
> latter is easier.

And much more helpful for people like me, who know just enough about IBT to be
dangerous, but don't fully understand how everything works the end-to-end.
 
> Progress, but still something to fix, namely the "ljmp *%0" which is,
> apparently, successfully branching to address 0 and then triggering a
> #GP(0). But maybe that's just because I'm on an old version of the KVM
> CET series, haven't build v6.18-rcX yet.

I suspect it's your old version, it passes on my end.

> Anyhow, one less puzzle, so should be good?

Ya, I'll slot this in for the next version (instead of having it be a fixup).