Re: [kvm-unit-tests PATCH v3 17/17] x86/cet: Add testcases to verify KVM rejects emulation of CET instructions

Sean Christopherson
2025-11-14T20:43:00Z

On Fri, Nov 14, 2025, Mathias Krause wrote:
> On 14.11.25 01:12, Sean Christopherson wrote:
> > +#define SHSTK_TEST_UNSUPPORTED_INSTRUCTION(insn) \
> > +do { \
> > + uint8_t vector = __CET_TEST_UNSUPPORTED_INSTRUCTION(insn); \
> > + \
> > + report(vector == UD_VECTOR, "Wanted #UD on %s, got %s", \
> > + insn, exception_mnemonic(vector)); \
> > +} while (0)
> > +
> > +/*
> > + * Treat IRET as unsupported with IBT even though the minimal interactions with
> > + * IBT _could_ be easily emulated by KVM, as KVM doesn't support emulating IRET
> > + * outside of Real Mode.
> > + */
> > +#define CET_TEST_UNSUPPORTED_INSTRUCTIONS(CET) \
> > +do { \
> > + CET##_TEST_UNSUPPORTED_INSTRUCTION("callq *%%rax"); \
> > + CET##_TEST_UNSUPPORTED_INSTRUCTION("lcall *%0"); \
>
> Maybe spell out that this is a 32bit far call ("lcalll *%0"), as only
> these are supported on AMD *and* Intel? (Intel can do 64bit ones too.)
>
> > + CET##_TEST_UNSUPPORTED_INSTRUCTION("syscall"); \
> > + CET##_TEST_UNSUPPORTED_INSTRUCTION("sysenter"); \
> > + CET##_TEST_UNSUPPORTED_INSTRUCTION("iretq"); \
>
> These, if emulated, would be rather disastrous to the test. :D
> - SYSCALL messes badly with the register state.
> - SYSENTER should #UD in long mode on AMD anyway but Intel allows it.
> However, it'll likely #GP early, prior to messing with register state,
> as IA32_SYSENTER_CS doesn't get initialized as the CPU expects it to
> be.
> - The stack frame for IRETQ isn't properly set up (lacks CS, RFLAGS, SS,
> RSP) but even if it'll #GP early for the non-canonical address, it'll
> have modified RSP, making the recovery "pop %rax" pop an unrelated
> stack word, likely making the code crash soon after.
>
> So these instructions really *rely* on KVM to #UD on these early.
> Dunno, but it feels like the tests should be made more robust wrt.
> possible future emulation by KVM?

I don't necessarily disagree, but I'm not convinced it'd be worth the effort to
properly set everything up _and_ keep it functional. Unless KVM is buggy, the
fallback will be completely untested. I hate user-hostile tests as much as the
next person (probably more), but for this one my vote is to punt and cross our
fingers that KVM never screws up.

> > +} while (0)
> > +
> > +static uint64_t cet_shstk_emulation(void)
> > +{
> > + CET_TEST_UNSUPPORTED_INSTRUCTIONS(SHSTK);
> > +
> > + SHSTK_TEST_UNSUPPORTED_INSTRUCTION("call 1f");
> > + SHSTK_TEST_UNSUPPORTED_INSTRUCTION("retq");
> > + SHSTK_TEST_UNSUPPORTED_INSTRUCTION("retq $10");
> > + SHSTK_TEST_UNSUPPORTED_INSTRUCTION("lretq");
> > + SHSTK_TEST_UNSUPPORTED_INSTRUCTION("lretq $10");
> > +
> > + /* Do a handful of JMPs to verify they aren't impacted by SHSTK. */
> > + asm volatile(KVM_FEP "jmp 1f\n\t"
> > + "1:\n\t"
> > + KVM_FEP "lea 2f(%%rip), %%rax\n\t"
> > + KVM_FEP "jmp *%%rax\n\t"
> ^^^^^^^^^^
> Fortunately, this indirect branch runs only with shadow stack enabled,
> no IBT. If it would, than below misses an ENDBR64.

50/50 as to whether I noticed that or just got lucky :-)

> vvv
> > + "2:\n\t"> + KVM_FEP "push $" xstr(USER_CS) "\n\t"
> > + KVM_FEP "lea 3f(%%rip), %%rax\n\t"
> > + KVM_FEP "push %%rax\n\t"
> > + /* Manually encode ljmpq, which gas doesn't recognize :-( */
>
> LJMPQ is actually only supported on Intel systems. AMD doesn't support
> it and one has to emulate it via "push $cs; push $rip; lretq".

Well that explains why gas wasn't being very nice to me. I'll tweak the comment.

> Dunno if it's a feature or bug of KVM to emulate LJMPQ fine on AMD -- if it
> does, that is!

Bug? Sort of? KVM won't emulate it on #UD, so it's only very limited emulation.
E.g. the guest would have to force emulation, put the jump source in emulated MMIO,
or play TLB games to get KVM to emulate

> > + KVM_FEP ".byte 0x48\n\t"
> > + "ljmpl *(%%rsp)\n\t"
> > + "3:\n\t"
> > + KVM_FEP "pop %%rax\n\t"
> > + KVM_FEP "pop %%rax\n\t"
> > + ::: "eax");
> > +
> > + return 0;
> > +}
> > +
>
> > +/*
> > + * Don't invoke printf() or report() in the IBT testcase, as it will likely
> > + * generate an indirect branch without an endbr64 annotation and thus #CP.
>
> Uhm, x86/Makefile.x86_64 has this:
>
> fcf_protection_full := $(call cc-option, -fcf-protection=full,)
> COMMON_CFLAGS += -mno-red-zone -mno-sse -mno-sse2 $(fcf_protection_full)
> [...]
> ifneq ($(fcf_protection_full),)
> tests += $(TEST_DIR)/cet.$(exe)
> endif
>
> So all code that needs it should have a leading ENDBR64 or the cet test
> wouldn't be part of the test suite.
>
> However, I also notice it isn't working...
>
> [digging in, hours later...]
>
> After tweaking the exception handling to be able to tweak the IBT
> tracker state (which would be stuck at WAIT_FOR_ENDBRANCH for a missing
> ENDBR64), it still wouldn't work for me. Bummer! Further instrumentation
> showed, the code triggered within exception_mnemonic() which caused only
> more question marks -- it's just a simple switch, right? Though, looking
> at the disassembly made it crystal clear:
>
> 000000000040707c <exception_mnemonic>:
> 40707c: endbr64
> 407080: cmp $0x1e,%edi
> 407083: ja 407117 <exception_mnemonic+0x9b>
> 407089: mov %edi,%edi
> 40708b: notrack jmp *0x4107e0(,%rdi,8)
> ::
> 4070b1: mov $0x411c7c,%eax # <-- #CP(3) here
>
> The switch block caused gcc to emit a jump table and an indirect jump
> for it (to 40708b). The jump is prefixed with 'notrack' to keep the size
> of the jump target stubs small, allowing for omitting the ENDBR64
> instruction. However, the IBT test code doesn't enable that in the MSR,
> leading the CPU to raising a #CP(3) for this indirect jump. D'oh!
>
> So we can either disable the generation of jump tables in gcc
> (-fno-jump-tables) or just allow the 'notrack' handling. I guess the
> latter is easier.

And much more helpful for people like me, who know just enough about IBT to be
dangerous, but don't fully understand how everything works the end-to-end.

> Progress, but still something to fix, namely the "ljmp *%0" which is,
> apparently, successfully branching to address 0 and then triggering a
> #GP(0). But maybe that's just because I'm on an old version of the KVM
> CET series, haven't build v6.18-rcX yet.

I suspect it's your old version, it passes on my end.

> Anyhow, one less puzzle, so should be good?

Ya, I'll slot this in for the next version (instead of having it be a fixup).