* [PATCH 0/1] vfio: fix KVM reference leak on QEMU CPR
@ 2026-11-20 15:47 Andrey Drobyshev
2026-11-20 15:47 ` [PATCH 1/1] vfio: Take the KVM reference only for drivers that read device->kvm Andrey Drobyshev
2026-11-20 16:52 ` [PATCH 0/1] vfio: fix KVM reference leak on QEMU CPR Paolo Bonzini
0 siblings, 2 replies; 5+ messages in thread
From: Andrey Drobyshev @ 2026-11-20 15:47 UTC (permalink / raw)
To: linux-kernel; +Cc: kvm, virtualization, alex, den, khorenko, andrey.drobyshev
When a VM has a VFIO device and QEMU does a CPR (CheckPoint and Restart)
live update, QEMU creates a new VM and keeps the device open. The
device holds a reference to the VM it was opened with, so the old VM is
not destroyed. It stays until the device is closed, which usually means
for as long as the guest runs.
It is one stale VM, not one per update. It is also visible: after
cpr-exec the kernel logs "kvm: debugfs: duplicate directory <pid>-<fd>",
and the new VM gets no debugfs directory.
This patch makes the device take that reference only if its driver uses
it.
Tested on v7.4-rc3 with QEMU 11.1.50, cpr-exec and the legacy VFIO
container, in a nested setup: the L1 VM has a spare virtio-net device
behind a virtual IOMMU and passes it to an L2 VM with vfio-pci. The
script below runs cpr-exec and counts the kvm_create_vm() and
kvm_destroy_vm() calls:
$ cat kvmref.sh
> #!/bin/bash
>
> QEMU=/path/to/build/qemu-system-x86_64
> DEV=0000:08:00.0
> MON="/var/tmp/mon.sock"
> STATE="/var/tmp/vm.state"
> TRACING=/sys/kernel/tracing
>
> hmp() {
> echo "$1" | socat - "UNIX-CONNECT:${MON}" | tr -d '\r' \
> | grep -v -e '^QEMU ' -e '^(qemu)'
> }
>
> bind_vfio() {
> dev="/sys/bus/pci/devices/${DEV}"
> modprobe vfio-pci
> echo vfio-pci > "${dev}/driver_override"
> [[ -e "${dev}/driver" ]] && echo "${DEV}" > "${dev}/driver/unbind"
> echo "${DEV}" > /sys/bus/pci/drivers_probe
> }
>
> trace_off() {
> echo nop > "${TRACING}/current_tracer"
> echo > "${TRACING}/set_ftrace_filter"
> }
>
> trace_on() {
> trace_off
> echo 'kvm_create_vm kvm_destroy_vm' > "${TRACING}/set_ftrace_filter"
> echo function > "${TRACING}/current_tracer"
> echo > "${TRACING}/trace"
> }
>
> # How many VMs were created and destroyed so far
> report() {
> created="$(grep -c 'kvm_create_vm' "${TRACING}/trace")"
> destroyed="$(grep -c 'kvm_destroy_vm' "${TRACING}/trace")"
> printf '%-22s created %s, destroyed %s, alive %s\n' \
> "$1:" "${created}" "${destroyed}" "$((created - destroyed))"
> }
>
> main() {
> mode="${1:-vfio}"
> cycles="${2:-1}"
>
> local -a args=(
> "${QEMU}" -accel kvm -display none
> -machine q35,aux-ram-share=on,memory-backend=ram
> -object memory-backend-memfd,id=ram,size=512M,share=on
> -monitor "unix:${MON},server=on,wait=off"
> )
>
> if [[ "${mode}" == vfio ]]; then
> bind_vfio
> args+=(-device "vfio-pci,host=${DEV}")
> fi
>
> trace_on
> "${args[@]}" &
> pid=$!
> sleep 2
> report 'QEMU started'
>
> for ((i = 1; i <= cycles; i++)); do
> hmp 'migrate_set_parameter mode cpr-exec'
> hmp "migrate_set_parameter cpr-exec-command ${args[*]} -incoming file:${STATE}"
> hmp "migrate file:${STATE}"
> sleep 3
> report "after cpr-exec ${i}"
> done
> echo "pid ${pid} is still $(readlink "/proc/${pid}/exe"), $(hmp 'info status')"
>
> hmp quit
> wait "${pid}"
> sleep 1
> report 'QEMU exited'
> trace_off
> }
>
> main "$@"
Control run:
$ ./kvmref.sh novfio 3
QEMU started: created 1, destroyed 0, alive 1
after cpr-exec 1: created 2, destroyed 1, alive 1
after cpr-exec 2: created 3, destroyed 2, alive 1
after cpr-exec 3: created 4, destroyed 3, alive 1
pid 6057 is still /path/to/build/qemu-system-x86_64, VM status: running
QEMU exited: created 4, destroyed 4, alive 0
Before patch:
$ ./kvmref.sh vfio 3
QEMU started: created 1, destroyed 0, alive 1
after cpr-exec 1: created 2, destroyed 0, alive 2
after cpr-exec 2: created 3, destroyed 1, alive 2
after cpr-exec 3: created 4, destroyed 2, alive 2
pid 7607 is still /path/to/build/qemu-system-x86_64, VM status: running
QEMU exited: created 4, destroyed 4, alive 0
After this patch:
$ ./kvmref.sh vfio 3
QEMU started: created 1, destroyed 0, alive 1
after cpr-exec 1: created 2, destroyed 1, alive 1
after cpr-exec 2: created 3, destroyed 2, alive 1
after cpr-exec 3: created 4, destroyed 3, alive 1
pid 6164 is still /path/to/build/qemu-system-x86_64, VM status: running
QEMU exited: created 4, destroyed 4, alive 0
Andrey Drobyshev (1):
vfio: Take the KVM reference only for drivers that read device->kvm
drivers/gpu/drm/i915/gvt/kvmgt.c | 1 +
drivers/s390/crypto/vfio_ap_ops.c | 1 +
drivers/vfio/device_cdev.c | 3 ++-
drivers/vfio/group.c | 3 ++-
drivers/vfio/pci/vfio_pci_core.c | 1 +
drivers/vfio/vfio_main.c | 6 ++++--
include/linux/vfio.h | 2 ++
7 files changed, 13 insertions(+), 4 deletions(-)
--
2.47.1
^ permalink raw reply [flat|nested] 5+ messages in thread* [PATCH 1/1] vfio: Take the KVM reference only for drivers that read device->kvm
2026-11-20 15:47 [PATCH 0/1] vfio: fix KVM reference leak on QEMU CPR Andrey Drobyshev
@ 2026-11-20 15:47 ` Andrey Drobyshev
2026-11-20 15:55 ` sashiko-bot
2026-11-20 16:52 ` [PATCH 0/1] vfio: fix KVM reference leak on QEMU CPR Paolo Bonzini
1 sibling, 1 reply; 5+ messages in thread
From: Andrey Drobyshev @ 2026-11-20 15:47 UTC (permalink / raw)
To: linux-kernel; +Cc: kvm, virtualization, alex, den, khorenko, andrey.drobyshev
A device takes a reference to the KVM file of its group or device file
at the first open and drops it at the last close, so that device->kvm
stays valid for the driver in between.
QEMU's cpr-exec and cpr-transfer keep the device open while the VM is
re-created. The first VM then stays referenced for as long as the
device is open: its file is not released, and its vcpus, mm and
in-kernel PIT thread stay with it.
To see it, start QEMU with a vfio-pci device, run cpr-exec and trace
kvm_create_vm() and kvm_destroy_vm(). The old VM is not destroyed at
the exec, and two VMs stay alive until QEMU exits. Without the device
the old VM is destroyed at the exec. The new VM also logs
kvm: debugfs: duplicate directory <pid>-<fd>
because it gets the pid and fd number of the old one.
Only three drivers read device->kvm: KVMGT, vfio_ap and vfio-pci's
zdev support. vfio_ap reads it only in open_device(), zdev also at
close, KVMGT also at close and while the guest runs. Let them say so
with needs_kvm_ref. Without it device->kvm stays NULL and no reference
is taken, so the device can't keep a VM alive.
Fixes: 2b48f52f2bff ("vfio: fix deadlock between group lock and kvm lock")
Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
drivers/gpu/drm/i915/gvt/kvmgt.c | 1 +
drivers/s390/crypto/vfio_ap_ops.c | 1 +
drivers/vfio/device_cdev.c | 3 ++-
drivers/vfio/group.c | 3 ++-
drivers/vfio/pci/vfio_pci_core.c | 1 +
drivers/vfio/vfio_main.c | 6 ++++--
include/linux/vfio.h | 2 ++
7 files changed, 13 insertions(+), 4 deletions(-)
diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index ec62db5cc367..3e05ad946c5d 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -1420,6 +1420,7 @@ static int intel_vgpu_init_dev(struct vfio_device *vfio_dev)
int ret;
vgpu->gvt = kdev_to_i915(mdev->type->parent->dev)->gvt;
+ vfio_dev->needs_kvm_ref = true;
ret = intel_gvt_create_vgpu(vgpu, type->conf);
if (ret)
return ret;
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 373ce574c431..5ec858389ba3 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -786,6 +786,7 @@ static int vfio_ap_mdev_init_dev(struct vfio_device *vdev)
container_of(vdev, struct ap_matrix_mdev, vdev);
matrix_mdev->mdev = to_mdev_device(vdev->dev);
+ vdev->needs_kvm_ref = true;
vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->matrix);
matrix_mdev->pqap_hook = handle_pqap;
vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->shadow_apcb);
diff --git a/drivers/vfio/device_cdev.c b/drivers/vfio/device_cdev.c
index 92627295182e..d661bdeb1834 100644
--- a/drivers/vfio/device_cdev.c
+++ b/drivers/vfio/device_cdev.c
@@ -157,7 +157,8 @@ long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df,
* Before the device open, get the KVM pointer currently
* associated with the device file (if there is) and obtain
* a reference. This reference is held until device closed.
- * Save the pointer in the device for use by drivers.
+ * Save the pointer in the device for use by drivers. Both only
+ * if the driver set needs_kvm_ref.
*/
vfio_df_get_kvm_safe(df);
diff --git a/drivers/vfio/group.c b/drivers/vfio/group.c
index 185f359f75d3..1b7b4f907895 100644
--- a/drivers/vfio/group.c
+++ b/drivers/vfio/group.c
@@ -184,7 +184,8 @@ static int vfio_df_group_open(struct vfio_device_file *df)
* Before the first device open, get the KVM pointer currently
* associated with the group (if there is one) and obtain a reference
* now that will be held until the open_count reaches 0 again. Save
- * the pointer in the device for use by drivers.
+ * the pointer in the device for use by drivers. Both only if the
+ * driver set needs_kvm_ref.
*/
if (device->open_count == 0)
vfio_device_group_get_kvm_safe(device);
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index fcbb2da860e2..b1689f9325e4 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -2173,6 +2173,7 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev)
vdev->pdev = to_pci_dev(core_vdev->dev);
vdev->irq_type = VFIO_PCI_NUM_IRQS;
+ core_vdev->needs_kvm_ref = IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM);
mutex_init(&vdev->igate);
spin_lock_init(&vdev->irqlock);
mutex_init(&vdev->ioeventfds_lock);
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index 4e5d738ccf4b..aa26c4e4c6e0 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -476,7 +476,8 @@ void vfio_device_get_kvm(struct vfio_device *device, struct file *kvm)
{
lockdep_assert_held(&device->dev_set->lock);
- if (!kvm)
+ /* Only a driver that reads device->kvm needs the KVM referenced */
+ if (!kvm || !device->needs_kvm_ref)
return;
device->kvm = get_file(kvm);
@@ -1536,7 +1537,8 @@ static void vfio_device_file_set_kvm(struct file *file, struct file *kvm)
* @kvm: KVM file to link
*
* When a VFIO device is first opened the KVM will be available in
- * device->kvm if one was associated with the file.
+ * device->kvm if one was associated with the file and the driver set
+ * needs_kvm_ref.
*/
void vfio_file_set_kvm(struct file *vfio_file, struct file *kvm)
{
diff --git a/include/linux/vfio.h b/include/linux/vfio.h
index 0cc91c6f96d2..debf75e83cae 100644
--- a/include/linux/vfio.h
+++ b/include/linux/vfio.h
@@ -54,6 +54,8 @@ struct vfio_device {
struct list_head dev_set_list;
unsigned int migration_flags;
u8 precopy_info_v2;
+ /* Drivers that use kvm must set this before registering the device */
+ bool needs_kvm_ref;
struct file *kvm;
/* Members below here are private, not for driver use */
--
2.47.1
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [PATCH 1/1] vfio: Take the KVM reference only for drivers that read device->kvm
2026-11-20 15:47 ` [PATCH 1/1] vfio: Take the KVM reference only for drivers that read device->kvm Andrey Drobyshev
@ 2026-11-20 15:55 ` sashiko-bot
0 siblings, 0 replies; 5+ messages in thread
From: sashiko-bot @ 2026-11-20 15:55 UTC (permalink / raw)
To: Andrey Drobyshev; +Cc: virtualization, Michael S. Tsirkin, Eugenio Perez
> A device takes a reference to the KVM file of its group or device file
> at the first open and drops it at the last close, so that device->kvm
> stays valid for the driver in between.
>
> QEMU's cpr-exec and cpr-transfer keep the device open while the VM is
> re-created. The first VM then stays referenced for as long as the
> device is open: its file is not released, and its vcpus, mm and
> in-kernel PIT thread stay with it.
>
> To see it, start QEMU with a vfio-pci device, run cpr-exec and trace
> kvm_create_vm() and kvm_destroy_vm(). The old VM is not destroyed at
> the exec, and two VMs stay alive until QEMU exits. Without the device
> the old VM is destroyed at the exec. The new VM also logs
>
> kvm: debugfs: duplicate directory <pid>-<fd>
>
> because it gets the pid and fd number of the old one.
>
> Only three drivers read device->kvm: KVMGT, vfio_ap and vfio-pci's
> zdev support. vfio_ap reads it only in open_device(), zdev also at
> close, KVMGT also at close and while the guest runs. Let them say so
> with needs_kvm_ref. Without it device->kvm stays NULL and no reference
> is taken, so the device can't keep a VM alive.
>
> Fixes: 2b48f52f2bff ("vfio: fix deadlock between group lock and kvm lock")
> Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261120154730.882393-1-andrey.drobyshev@virtuozzo.com?part=1
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 0/1] vfio: fix KVM reference leak on QEMU CPR
2026-11-20 15:47 [PATCH 0/1] vfio: fix KVM reference leak on QEMU CPR Andrey Drobyshev
2026-11-20 15:47 ` [PATCH 1/1] vfio: Take the KVM reference only for drivers that read device->kvm Andrey Drobyshev
@ 2026-11-20 16:52 ` Paolo Bonzini
2026-11-20 18:54 ` Jason Gunthorpe
1 sibling, 1 reply; 5+ messages in thread
From: Paolo Bonzini @ 2026-11-20 16:52 UTC (permalink / raw)
To: Andrey Drobyshev, linux-kernel; +Cc: kvm, virtualization, alex, den, khorenko
On 11/20/26 16:47, Andrey Drobyshev wrote:
> When a VM has a VFIO device and QEMU does a CPR (CheckPoint and Restart)
> live update, QEMU creates a new VM and keeps the device open. The
> device holds a reference to the VM it was opened with, so the old VM is
> not destroyed. It stays until the device is closed, which usually means
> for as long as the guest runs.
>
> It is one stale VM, not one per update. It is also visible: after
> cpr-exec the kernel logs "kvm: debugfs: duplicate directory <pid>-<fd>",
> and the new VM gets no debugfs directory.
>
> This patch makes the device take that reference only if its driver uses
> it.
There are two problems with this patch:
1) It sweeps the bug under the carpet and only solves the case that
*you* care about. If the device keeps a reference to the VM it was
first bound to, the problem is that there's no handoff from the old file
descriptor to the new one.
2) Even worse, the above text is 101% copied from Claude.
Please come back once you have a patch that works for everyone and
anyway only after having read Documentation/process/generated-content.rst:
"You are expected to understand and to be able to defend
everything you submit. If you are unable to do so, then do not submit
the resulting changes. If you do so anyway, maintainers are entitled to
reject your series without detailed review."
Paolo
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 0/1] vfio: fix KVM reference leak on QEMU CPR
2026-11-20 16:52 ` [PATCH 0/1] vfio: fix KVM reference leak on QEMU CPR Paolo Bonzini
@ 2026-11-20 18:54 ` Jason Gunthorpe
0 siblings, 0 replies; 5+ messages in thread
From: Jason Gunthorpe @ 2026-11-20 18:54 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Andrey Drobyshev, linux-kernel, kvm, virtualization, alex, den,
khorenko
On Fri, Nov 20, 2026 at 05:52:09PM +0100, Paolo Bonzini wrote:
> On 11/20/26 16:47, Andrey Drobyshev wrote:
> > When a VM has a VFIO device and QEMU does a CPR (CheckPoint and Restart)
> > live update, QEMU creates a new VM and keeps the device open. The
> > device holds a reference to the VM it was opened with, so the old VM is
> > not destroyed. It stays until the device is closed, which usually means
> > for as long as the guest runs.
> >
> > It is one stale VM, not one per update. It is also visible: after
> > cpr-exec the kernel logs "kvm: debugfs: duplicate directory <pid>-<fd>",
> > and the new VM gets no debugfs directory.
> >
> > This patch makes the device take that reference only if its driver uses
> > it.
>
> There are two problems with this patch:
>
> 1) It sweeps the bug under the carpet and only solves the case that *you*
> care about. If the device keeps a reference to the VM it was first bound
> to, the problem is that there's no handoff from the old file descriptor to
> the new one.
Yeah, reflowing the kvm fd is not something vfio broadly supports at
all, there a several issues with even trying to do this this I think.
If qemu is doing a live update it probably has to keep the kvm fd
alive across the update too..
Ie when we get to things like confidential VMs there is state inside
that the kvm FD that cannot be checkpointed.
Jason
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-11-20 18:54 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20 15:47 [PATCH 0/1] vfio: fix KVM reference leak on QEMU CPR Andrey Drobyshev
2026-11-20 15:47 ` [PATCH 1/1] vfio: Take the KVM reference only for drivers that read device->kvm Andrey Drobyshev
2026-11-20 15:55 ` sashiko-bot
2026-11-20 16:52 ` [PATCH 0/1] vfio: fix KVM reference leak on QEMU CPR Paolo Bonzini
2026-11-20 18:54 ` Jason Gunthorpe
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox