Linux virtualization list
 help / color / mirror / Atom feed
From: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
To: linux-kernel@vger.kernel.org
Cc: kvm@vger.kernel.org, virtualization@lists.linux.dev,
	alex@shazbot.org, den@openvz.org, khorenko@virtuozzo.com,
	andrey.drobyshev@virtuozzo.com
Subject: [PATCH 0/1] vfio: fix KVM reference leak on QEMU CPR
Date: Fri, 20 Nov 2026 17:47:29 +0200	[thread overview]
Message-ID: <20261120154730.882393-1-andrey.drobyshev@virtuozzo.com> (raw)

When a VM has a VFIO device and QEMU does a CPR (CheckPoint and Restart)
live update, QEMU creates a new VM and keeps the device open.  The
device holds a reference to the VM it was opened with, so the old VM is
not destroyed.  It stays until the device is closed, which usually means
for as long as the guest runs.

It is one stale VM, not one per update.  It is also visible: after
cpr-exec the kernel logs "kvm: debugfs: duplicate directory <pid>-<fd>",
and the new VM gets no debugfs directory.

This patch makes the device take that reference only if its driver uses
it.

Tested on v7.4-rc3 with QEMU 11.1.50, cpr-exec and the legacy VFIO
container, in a nested setup: the L1 VM has a spare virtio-net device
behind a virtual IOMMU and passes it to an L2 VM with vfio-pci.  The
script below runs cpr-exec and counts the kvm_create_vm() and
kvm_destroy_vm() calls:

$ cat kvmref.sh

> #!/bin/bash
> 
> QEMU=/path/to/build/qemu-system-x86_64
> DEV=0000:08:00.0
> MON="/var/tmp/mon.sock"
> STATE="/var/tmp/vm.state"
> TRACING=/sys/kernel/tracing
> 
> hmp() {
>   echo "$1" | socat - "UNIX-CONNECT:${MON}" | tr -d '\r' \
>     | grep -v -e '^QEMU ' -e '^(qemu)'
> }
> 
> bind_vfio() {
>   dev="/sys/bus/pci/devices/${DEV}"
>   modprobe vfio-pci
>   echo vfio-pci > "${dev}/driver_override"
>   [[ -e "${dev}/driver" ]] && echo "${DEV}" > "${dev}/driver/unbind"
>   echo "${DEV}" > /sys/bus/pci/drivers_probe
> }
> 
> trace_off() {
>   echo nop > "${TRACING}/current_tracer"
>   echo > "${TRACING}/set_ftrace_filter"
> }
> 
> trace_on() {
>   trace_off
>   echo 'kvm_create_vm kvm_destroy_vm' > "${TRACING}/set_ftrace_filter"
>   echo function > "${TRACING}/current_tracer"
>   echo > "${TRACING}/trace"
> }
> 
> # How many VMs were created and destroyed so far
> report() {
>   created="$(grep -c 'kvm_create_vm' "${TRACING}/trace")"
>   destroyed="$(grep -c 'kvm_destroy_vm' "${TRACING}/trace")"
>   printf '%-22s created %s, destroyed %s, alive %s\n' \
>     "$1:" "${created}" "${destroyed}" "$((created - destroyed))"
> }
> 
> main() {
>   mode="${1:-vfio}"
>   cycles="${2:-1}"
> 
>   local -a args=(
>     "${QEMU}" -accel kvm -display none
>     -machine q35,aux-ram-share=on,memory-backend=ram
>     -object memory-backend-memfd,id=ram,size=512M,share=on
>     -monitor "unix:${MON},server=on,wait=off"
>   )
> 
>   if [[ "${mode}" == vfio ]]; then
>     bind_vfio
>     args+=(-device "vfio-pci,host=${DEV}")
>   fi
> 
>   trace_on
>   "${args[@]}" &
>   pid=$!
>   sleep 2
>   report 'QEMU started'
> 
>   for ((i = 1; i <= cycles; i++)); do
>     hmp 'migrate_set_parameter mode cpr-exec'
>     hmp "migrate_set_parameter cpr-exec-command ${args[*]} -incoming file:${STATE}"
>     hmp "migrate file:${STATE}"
>     sleep 3
>     report "after cpr-exec ${i}"
>   done
>   echo "pid ${pid} is still $(readlink "/proc/${pid}/exe"), $(hmp 'info status')"
> 
>   hmp quit
>   wait "${pid}"
>   sleep 1
>   report 'QEMU exited'
>   trace_off
> }
> 
> main "$@"

Control run:

$ ./kvmref.sh novfio 3
QEMU started:          created 1, destroyed 0, alive 1
after cpr-exec 1:      created 2, destroyed 1, alive 1
after cpr-exec 2:      created 3, destroyed 2, alive 1
after cpr-exec 3:      created 4, destroyed 3, alive 1
pid 6057 is still /path/to/build/qemu-system-x86_64, VM status: running
QEMU exited:           created 4, destroyed 4, alive 0

Before patch:

$ ./kvmref.sh vfio 3
QEMU started:          created 1, destroyed 0, alive 1
after cpr-exec 1:      created 2, destroyed 0, alive 2
after cpr-exec 2:      created 3, destroyed 1, alive 2
after cpr-exec 3:      created 4, destroyed 2, alive 2
pid 7607 is still /path/to/build/qemu-system-x86_64, VM status: running
QEMU exited:           created 4, destroyed 4, alive 0

After this patch:

$ ./kvmref.sh vfio 3
QEMU started:          created 1, destroyed 0, alive 1
after cpr-exec 1:      created 2, destroyed 1, alive 1
after cpr-exec 2:      created 3, destroyed 2, alive 1
after cpr-exec 3:      created 4, destroyed 3, alive 1
pid 6164 is still /path/to/build/qemu-system-x86_64, VM status: running
QEMU exited:           created 4, destroyed 4, alive 0

Andrey Drobyshev (1):
  vfio: Take the KVM reference only for drivers that read device->kvm

 drivers/gpu/drm/i915/gvt/kvmgt.c  | 1 +
 drivers/s390/crypto/vfio_ap_ops.c | 1 +
 drivers/vfio/device_cdev.c        | 3 ++-
 drivers/vfio/group.c              | 3 ++-
 drivers/vfio/pci/vfio_pci_core.c  | 1 +
 drivers/vfio/vfio_main.c          | 6 ++++--
 include/linux/vfio.h              | 2 ++
 7 files changed, 13 insertions(+), 4 deletions(-)

-- 
2.47.1


             reply	other threads:[~2026-11-20 15:47 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-20 15:47 Andrey Drobyshev [this message]
2026-11-20 15:47 ` [PATCH 1/1] vfio: Take the KVM reference only for drivers that read device->kvm Andrey Drobyshev
2026-11-20 15:55   ` sashiko-bot
2026-11-20 16:52 ` [PATCH 0/1] vfio: fix KVM reference leak on QEMU CPR Paolo Bonzini
2026-11-20 18:54   ` Jason Gunthorpe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261120154730.882393-1-andrey.drobyshev@virtuozzo.com \
    --to=andrey.drobyshev@virtuozzo.com \
    --cc=alex@shazbot.org \
    --cc=den@openvz.org \
    --cc=khorenko@virtuozzo.com \
    --cc=kvm@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=virtualization@lists.linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox