Linux virtualization list
 help / color / mirror / Atom feed
* [PATCH 0/1] vfio: fix KVM reference leak on QEMU CPR
@ 2026-11-20 15:47 Andrey Drobyshev
  2026-11-20 15:47 ` [PATCH 1/1] vfio: Take the KVM reference only for drivers that read device->kvm Andrey Drobyshev
  2026-11-20 16:52 ` [PATCH 0/1] vfio: fix KVM reference leak on QEMU CPR Paolo Bonzini
  0 siblings, 2 replies; 5+ messages in thread
From: Andrey Drobyshev @ 2026-11-20 15:47 UTC (permalink / raw)
  To: linux-kernel; +Cc: kvm, virtualization, alex, den, khorenko, andrey.drobyshev

When a VM has a VFIO device and QEMU does a CPR (CheckPoint and Restart)
live update, QEMU creates a new VM and keeps the device open.  The
device holds a reference to the VM it was opened with, so the old VM is
not destroyed.  It stays until the device is closed, which usually means
for as long as the guest runs.

It is one stale VM, not one per update.  It is also visible: after
cpr-exec the kernel logs "kvm: debugfs: duplicate directory <pid>-<fd>",
and the new VM gets no debugfs directory.

This patch makes the device take that reference only if its driver uses
it.

Tested on v7.4-rc3 with QEMU 11.1.50, cpr-exec and the legacy VFIO
container, in a nested setup: the L1 VM has a spare virtio-net device
behind a virtual IOMMU and passes it to an L2 VM with vfio-pci.  The
script below runs cpr-exec and counts the kvm_create_vm() and
kvm_destroy_vm() calls:

$ cat kvmref.sh

> #!/bin/bash
> 
> QEMU=/path/to/build/qemu-system-x86_64
> DEV=0000:08:00.0
> MON="/var/tmp/mon.sock"
> STATE="/var/tmp/vm.state"
> TRACING=/sys/kernel/tracing
> 
> hmp() {
>   echo "$1" | socat - "UNIX-CONNECT:${MON}" | tr -d '\r' \
>     | grep -v -e '^QEMU ' -e '^(qemu)'
> }
> 
> bind_vfio() {
>   dev="/sys/bus/pci/devices/${DEV}"
>   modprobe vfio-pci
>   echo vfio-pci > "${dev}/driver_override"
>   [[ -e "${dev}/driver" ]] && echo "${DEV}" > "${dev}/driver/unbind"
>   echo "${DEV}" > /sys/bus/pci/drivers_probe
> }
> 
> trace_off() {
>   echo nop > "${TRACING}/current_tracer"
>   echo > "${TRACING}/set_ftrace_filter"
> }
> 
> trace_on() {
>   trace_off
>   echo 'kvm_create_vm kvm_destroy_vm' > "${TRACING}/set_ftrace_filter"
>   echo function > "${TRACING}/current_tracer"
>   echo > "${TRACING}/trace"
> }
> 
> # How many VMs were created and destroyed so far
> report() {
>   created="$(grep -c 'kvm_create_vm' "${TRACING}/trace")"
>   destroyed="$(grep -c 'kvm_destroy_vm' "${TRACING}/trace")"
>   printf '%-22s created %s, destroyed %s, alive %s\n' \
>     "$1:" "${created}" "${destroyed}" "$((created - destroyed))"
> }
> 
> main() {
>   mode="${1:-vfio}"
>   cycles="${2:-1}"
> 
>   local -a args=(
>     "${QEMU}" -accel kvm -display none
>     -machine q35,aux-ram-share=on,memory-backend=ram
>     -object memory-backend-memfd,id=ram,size=512M,share=on
>     -monitor "unix:${MON},server=on,wait=off"
>   )
> 
>   if [[ "${mode}" == vfio ]]; then
>     bind_vfio
>     args+=(-device "vfio-pci,host=${DEV}")
>   fi
> 
>   trace_on
>   "${args[@]}" &
>   pid=$!
>   sleep 2
>   report 'QEMU started'
> 
>   for ((i = 1; i <= cycles; i++)); do
>     hmp 'migrate_set_parameter mode cpr-exec'
>     hmp "migrate_set_parameter cpr-exec-command ${args[*]} -incoming file:${STATE}"
>     hmp "migrate file:${STATE}"
>     sleep 3
>     report "after cpr-exec ${i}"
>   done
>   echo "pid ${pid} is still $(readlink "/proc/${pid}/exe"), $(hmp 'info status')"
> 
>   hmp quit
>   wait "${pid}"
>   sleep 1
>   report 'QEMU exited'
>   trace_off
> }
> 
> main "$@"

Control run:

$ ./kvmref.sh novfio 3
QEMU started:          created 1, destroyed 0, alive 1
after cpr-exec 1:      created 2, destroyed 1, alive 1
after cpr-exec 2:      created 3, destroyed 2, alive 1
after cpr-exec 3:      created 4, destroyed 3, alive 1
pid 6057 is still /path/to/build/qemu-system-x86_64, VM status: running
QEMU exited:           created 4, destroyed 4, alive 0

Before patch:

$ ./kvmref.sh vfio 3
QEMU started:          created 1, destroyed 0, alive 1
after cpr-exec 1:      created 2, destroyed 0, alive 2
after cpr-exec 2:      created 3, destroyed 1, alive 2
after cpr-exec 3:      created 4, destroyed 2, alive 2
pid 7607 is still /path/to/build/qemu-system-x86_64, VM status: running
QEMU exited:           created 4, destroyed 4, alive 0

After this patch:

$ ./kvmref.sh vfio 3
QEMU started:          created 1, destroyed 0, alive 1
after cpr-exec 1:      created 2, destroyed 1, alive 1
after cpr-exec 2:      created 3, destroyed 2, alive 1
after cpr-exec 3:      created 4, destroyed 3, alive 1
pid 6164 is still /path/to/build/qemu-system-x86_64, VM status: running
QEMU exited:           created 4, destroyed 4, alive 0

Andrey Drobyshev (1):
  vfio: Take the KVM reference only for drivers that read device->kvm

 drivers/gpu/drm/i915/gvt/kvmgt.c  | 1 +
 drivers/s390/crypto/vfio_ap_ops.c | 1 +
 drivers/vfio/device_cdev.c        | 3 ++-
 drivers/vfio/group.c              | 3 ++-
 drivers/vfio/pci/vfio_pci_core.c  | 1 +
 drivers/vfio/vfio_main.c          | 6 ++++--
 include/linux/vfio.h              | 2 ++
 7 files changed, 13 insertions(+), 4 deletions(-)

-- 
2.47.1


^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2026-11-20 18:54 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20 15:47 [PATCH 0/1] vfio: fix KVM reference leak on QEMU CPR Andrey Drobyshev
2026-11-20 15:47 ` [PATCH 1/1] vfio: Take the KVM reference only for drivers that read device->kvm Andrey Drobyshev
2026-11-20 15:55   ` sashiko-bot
2026-11-20 16:52 ` [PATCH 0/1] vfio: fix KVM reference leak on QEMU CPR Paolo Bonzini
2026-11-20 18:54   ` Jason Gunthorpe

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox