Kernel KVM virtualization development
 help / color / mirror / Atom feed
From: Sean Christopherson <seanjc@google.com>
To: Weiming Shi <bestswngs@gmail.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>,
	Kai Huang <kai.huang@intel.com>,
	kvm@vger.kernel.org,  stable@vger.kernel.org,
	Zhang Haoyu <zhanghy@sangfor.com>,
	 Jason Wang <jasowang@redhat.com>,
	Zhong Wang <wangzhong.c0ss4ck@bytedance.com>,
	 Xuanqing Shi <shixuanqing.11@bytedance.com>
Subject: Re: [PATCH v2] KVM: x86: Destroy the PIC and IOAPIC before destroying vCPUs
Date: Mon, 6 Jul 2026 11:20:30 -0700	[thread overview]
Message-ID: <akvx7que1BE5DY-O@google.com> (raw)
In-Reply-To: <20260706180025.2735341-3-bestswngs@gmail.com>

On Tue, Jul 07, 2026, Weiming Shi wrote:
> kvm_ioapic_eoi_inject_work() re-delivers a throttled level-triggered
> interrupt via kvm_irq_delivery_to_apic(), which walks kvm->arch.apic_map
> and dereferences the destination vCPU's APIC.  The work is cancelled only
> in kvm_ioapic_destroy(), which runs after kvm_destroy_vcpus() has freed
> the vCPUs and their APICs.  kvm_free_lapic() does not rebuild apic_map, so
> the map is left with dangling pointers, and a work item that fires during
> that window reads freed memory:
> 
>  BUG: KASAN: slab-use-after-free in __kvm_irq_delivery_to_apic_fast (arch/x86/kvm/lapic.c:1248)
>  Read of size 8 by task kworker/3:1
>  Workqueue: events kvm_ioapic_eoi_inject_work
>   __kvm_irq_delivery_to_apic_fast (arch/x86/kvm/lapic.c:1248)
>   __kvm_irq_delivery_to_apic (arch/x86/kvm/lapic.c:1343)
>   ioapic_service (arch/x86/kvm/ioapic.c:492)
>   kvm_ioapic_eoi_inject_work (arch/x86/kvm/ioapic.c:525)
>   process_one_work
> 
>  Freed by task 153:
>   kvm_arch_vcpu_destroy (arch/x86/kvm/x86.c:12871)
>   kvm_destroy_vcpus (virt/kvm/kvm_main.c:489)
>   kvm_arch_destroy_vm (arch/x86/kvm/x86.c:13402)
>   kvm_destroy_vm (virt/kvm/kvm_main.c:1302)
>   kvm_vm_release (virt/kvm/kvm_main.c:1363)
> 
> A guest arms the work by EOIing a level-triggered pin 10000 times in a
> row, so the window is reachable from guest ring 0 whenever its VM is torn
> down soon after.
> 
> Destroy the in-kernel PIC and IOAPIC in kvm_arch_pre_destroy_vm(),
> before vCPUs are freed, so the eoi_inject work is cancelled while the
> target APICs are still valid.  This also unregisters the PIC/IOAPIC
> MMIO devices while the KVM buses still exist; kvm_destroy_vm() tears
> the buses down right after kvm_free_irq_routing() and before
> kvm_arch_destroy_vm(), so the previous kvm_io_bus_unregister_dev() in
> kvm_ioapic_destroy() was a no-op.
> 
> Fixes: 184564efae4d ("kvm: ioapic: conditionally delay irq delivery duringeoi broadcast")
> Link: https://lore.kernel.org/all/88ba60ad32ba851426a3f6590b0e402210991b4a.e33b58ce.0008.4e1c.aa62.c1024b242cbf@bytedance.com/
> Suggested-by: Kai Huang <kai.huang@intel.com>
> Reported-by: Zhong Wang <wangzhong.c0ss4ck@bytedance.com>
> Reported-by: Xuanqing Shi <shixuanqing.11@bytedance.com>
> Signed-off-by: Weiming Shi <bestswngs@gmail.com>
> ---
> v1: https://lore.kernel.org/all/20260705045450.1325048-2-bestswngs@gmail.com/
> 
> v2:
>  - Per Kai's suggestion, instead of adding a kvm_ioapic_pre_destroy()
>    helper that only cancels the eoi_inject work, move
>    kvm_pic_destroy()/kvm_ioapic_destroy() as a whole into
>    kvm_arch_pre_destroy_vm().  This also fixes the stale
>    kvm_io_bus_unregister_dev() Kai pointed out.

Sadly, it creates an even easier-to-exploit NULL pointer deref.  Your v1 is what
I came up with idependently (the bug got reported off-list), though I eschewed a
helper.

Author:     Sean Christopherson <seanjc@google.com>
AuthorDate: Tue Jun 30 08:13:42 2026 -0700
Commit:     Sean Christopherson <seanjc@google.com>
CommitDate: Tue Jun 30 08:28:19 2026 -0700

    KVM: x86: Cancel delayed I/O APIC EOI handling before destroying vCPUs
    
    Cancel (and flush) the I/O APIC's delayed EOI handling work during the
    "pre VM destroy" phase, before vCPUs are destroyed, as processing the EOI
    broadcast will inject another IRQ if the line is asserted, i.e. will try
    to deliver an IRQ to the target vCPU(s).  Canceling the work after vCPUs
    are destroyed leads to UAF if the delayed work is processed after vCPUs are
    destroyed.
    
      BUG: KASAN: slab-use-after-free in __kvm_irq_delivery_to_apic_fast+0x9bf/0xa20 arch/x86/kvm/lapic.c:1250
      Read of size 8 at addr ffff8880499abea0 by task kworker/1:2/1218
    
      CPU: 1 UID: 0 PID: 1218 Comm: kworker/1:2 Not tainted 7.1.0-rc7 #5 PREEMPT(lazy)
      Hardware name: QEMU Ubuntu 25.10 PC v2 (i440FX + PIIX, + 10.1 machine, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
      Workqueue: events kvm_ioapic_eoi_inject_work
      Call Trace:
       <TASK>
       __dump_stack /root/linux/lib/dump_stack.c:94
       dump_stack_lvl+0x100/0x190 /root/linux/lib/dump_stack.c:120
       print_address_description /root/linux/mm/kasan/report.c:378
       print_report+0x139/0x4ad /root/linux/mm/kasan/report.c:482
       kasan_report+0xe4/0x1d0 /root/linux/mm/kasan/report.c:595
       __kvm_irq_delivery_to_apic_fast+0x9bf/0xa20 /root/linux/arch/x86/kvm/lapic.c:1250
       __kvm_irq_delivery_to_apic+0xd8/0xbf0 /root/linux/arch/x86/kvm/lapic.c:1345
       kvm_irq_delivery_to_apic /root/linux/arch/x86/kvm/lapic.h:129
       ioapic_service+0x308/0x590 /root/linux/arch/x86/kvm/ioapic.c:492
       kvm_ioapic_eoi_inject_work+0x13c/0x190 /root/linux/arch/x86/kvm/ioapic.c:532
       process_one_work+0xa59/0x19a0 /root/linux/kernel/workqueue.c:3314
       process_scheduled_works /root/linux/kernel/workqueue.c:3397
       worker_thread+0x5eb/0xe50 /root/linux/kernel/workqueue.c:3478
       kthread+0x370/0x450 /root/linux/kernel/kthread.c:436
       ret_from_fork+0x72b/0xd30 /root/linux/arch/x86/kernel/process.c:158
       ret_from_fork_asm+0x1a/0x30 /root/linux/arch/x86/entry/entry_64.S:245
       </TASK>
    
    Note, the VM is unreachable once kvm_destroy_vm() starts, and scheduling
    new work via kvm_ioapic_send_eoi() can only be done via KVM_RUN, i.e.
    requires a live vCPU.
    
    Alternatively, KVM could simply destroy the I/O APIC during the "pre" phase
    of VM destruction, but that gets more than a bit sketchy as KVM expects the
    I/O APIC to exist if ioapic_in_kernel() is true, and nested virtualization
    in particular has a bad habit of touching VM-scope state during vCPU
    destruction.  E.g. attempting to free the PIC during the pre phase would
    lead to a NULL pointer dereference in kvm_cpu_has_extint(), and it's not
    hard to imagine the I/O APIC having a similar flaw.
    
    Fixes: 17bcd7144263 ("KVM: x86: Free vCPUs before freeing VM state")
    Reported-by: <zdi-disclosures@trendmicro.com>
    Cc: stable@vger.kernel.org
    Signed-off-by: Sean Christopherson <seanjc@google.com>

diff --git arch/x86/kvm/x86.c arch/x86/kvm/x86.c
index 0626e835e9eb..a0cc74c8ded1 100644
--- arch/x86/kvm/x86.c
+++ arch/x86/kvm/x86.c
@@ -9942,6 +9942,8 @@ void kvm_arch_pre_destroy_vm(struct kvm *kvm)
         */
 #ifdef CONFIG_KVM_IOAPIC
        kvm_free_pit(kvm);
+       if (kvm->arch.vioapic)
+               cancel_delayed_work_sync(&kvm->arch.vioapic->eoi_inject);
 #endif
 
        kvm_mmu_pre_destroy_vm(kvm);

  reply	other threads:[~2026-07-06 18:20 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <20260705045450.1325048-2-bestswngs@gmail.com>
2026-07-06 18:00 ` [PATCH v2] KVM: x86: Destroy the PIC and IOAPIC before destroying vCPUs Weiming Shi
2026-07-06 18:20   ` Sean Christopherson [this message]
2026-07-06 22:06     ` Huang, Kai
2026-07-06 22:10       ` Sean Christopherson
2026-07-06 22:33         ` Huang, Kai
2026-07-06 18:20   ` sashiko-bot
2026-07-07  6:32   ` [syzbot ci] " syzbot ci

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=akvx7que1BE5DY-O@google.com \
    --to=seanjc@google.com \
    --cc=bestswngs@gmail.com \
    --cc=jasowang@redhat.com \
    --cc=kai.huang@intel.com \
    --cc=kvm@vger.kernel.org \
    --cc=pbonzini@redhat.com \
    --cc=shixuanqing.11@bytedance.com \
    --cc=stable@vger.kernel.org \
    --cc=wangzhong.c0ss4ck@bytedance.com \
    --cc=zhanghy@sangfor.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox