Re: [kvm-unit-tests PATCH v2 13/13] x86: Avoid top-most page for vmalloc on x86-64

[Chao Gao <chao.gao@intel.com>]

On Fri, Sep 12, 2025 at 07:37:54AM -0700, Sean Christopherson wrote:
>On Thu, Jun 26, 2025, Mathias Krause wrote:
>> The x86-64 implementation if setup_mmu() doesn't initialize 'vfree_top'
>> and leaves it at its zero-value. This isn't wrong per se, however, it
>> leads to odd configurations when the first vmalloc/vmap page gets
>> allocated. It'll be the very last page in the virtual address space --
>> which is an interesting corner case -- but its boundary will probably
>> wrap. It does so, for CET's shadow stack, at least, which loads the
>> shadow stack pointer with the base address of the mapped page plus its
>> size, i.e. 0xffffffff_fffff000 + 4096, which wraps to 0x0.
>> 
>> The CPU seems to handle such configurations just fine. However, it feels
>> odd to set the shadow stack pointer to "NULL".
>> 
>> To avoid the wrapping, ignore the top most page by initializing
>> 'vfree_top' to just one page below.
>> 
>> Reviewed-by: Chao Gao <chao.gao@intel.com>
>> Signed-off-by: Mathias Krause <minipli@grsecurity.net>
>> ---
>> v2:
>> - change comment in x86/lam.c too
>> 
>>  lib/x86/vm.c |  2 ++
>>  x86/lam.c    | 10 +++++-----
>>  2 files changed, 7 insertions(+), 5 deletions(-)
>> 
>> diff --git a/lib/x86/vm.c b/lib/x86/vm.c
>> index 90f73fbb2dfd..27e7bb4004ef 100644
>> --- a/lib/x86/vm.c
>> +++ b/lib/x86/vm.c
>> @@ -191,6 +191,8 @@ void *setup_mmu(phys_addr_t end_of_memory, void *opt_mask)
>>          end_of_memory = (1ul << 32);  /* map mmio 1:1 */
>>  
>>      setup_mmu_range(cr3, 0, end_of_memory);
>> +    /* skip the last page for out-of-bound and wrap-around reasons */
>> +    init_alloc_vpage((void *)(~(PAGE_SIZE - 1)));
>
>This breaks the eventinj test (leads to SHUTDOWN).  I haven't spent any time
>trying to figure out why.

do_iret in eventinj.c is buggy. When crafting an IRET frame, it just sets
RFLAGS, CS and RIP properly, leaving SS and RSP as 0. So after IRET, RSP
becomes 0. Then, a nested NMI is injected right after IRET and accesses the
stack. If the stack (i.e., the page starts from 0xffffffff_fffff000) isn't
mapped, the NMI will cause TRIPLE FAULT.

The issue goes unnoticed because w/o this patch, the handle_irq()->malloc() at
the beginning of eventinj.c:main() allocates and maps the page 0xffffffff_fffff000

The issue can be fixed with below diff:

index 6fbb2d0f..945f7d11 100644
--- a/x86/eventinj.c
+++ b/x86/eventinj.c
@@ -132,7 +132,7 @@ static void nested_nmi_iret_isr(struct ex_regs *r)
 {
        printf("Nested NMI isr running rip=%lx\n", r->rip);

-       if (r->rip == iret_stack[-3])
+       if (r->rip == iret_stack[-4])
                test_count++;
 }

@@ -152,6 +152,7 @@ asm("do_iret:"
        "mov 8(%esp), %edx \n\t"        // virt_stack
 #endif
        "xchg %"R "dx, %"R "sp \n\t"    // point to new stack
+       "push"W" %"R "dx \n\t"
        "pushf"W" \n\t"
        "mov %cs, %ecx \n\t"
        "push"W" %"R "cx \n\t"