[PATCH] drm/vmwgfx: Release PRIME import in the BO destroy path

[PATCH] drm/vmwgfx: Release PRIME import in the BO destroy path

[Michal TOMA]

drm_gem_prime_import_dev() attaches to a foreign dma-buf, takes a
reference with get_dma_buf(), maps the attachment and, once
vmw_prime_import_sg_table() has created the TTM buffer object, stores
the attachment in obj->import_attach. Drivers that import this way have
to undo it by calling drm_prime_gem_destroy() when the object is freed.

vmwgfx never does. The TTM destroy callback, vmw_bo_free(), only calls
drm_gem_object_release() and kfree(). For every imported dma-buf, the
sg mapping, the attachment and the dma-buf reference are leaked when
the last GEM reference goes away, and the exporter's backing pages stay
pinned until reboot.

Any process that can open the vmwgfx render node can hit this by
importing a dma-buf from another exporter, for example with
DRM_IOCTL_PRIME_FD_TO_HANDLE. It showed up as a memory drain on a
VirtualBox VMSVGA guest running a Plasma 6.7 Wayland session with the
host's 3D acceleration off. KWin 6.7 wraps wl_shm client buffers in
udmabufs and imports them through EGL on llvmpipe, and it keeps many of
those imports referenced while it runs, which is a separate userspace
problem. Restarting the compositor did not give the memory back,
though: with about 1.5 GiB of client buffers imported, killing KWin
left 211 udmabufs in /sys/kernel/debug/dma_buf/bufinfo with a refcount
of 1, still attached to the vmwgfx device, after every userspace
reference was gone. Their pages are no longer mapped or in any page
cache, so reclaim and swap cannot free them.

Call drm_prime_gem_destroy() for imported objects before
drm_gem_object_release(), as amdgpu, radeon and nouveau do in their TTM
destroy callbacks, and include <drm/drm_prime.h> for it. The check is
safe on the import error path: drm_gem_prime_import_dev() only sets
obj->import_attach after gem_prime_import_sg_table() succeeded, so a
buffer object destroyed before that point is not unmapped, detached or
put a second time. ttm_bo_release() tears down the TTM backing and
drops the reservation lock before calling the destroy callback, which
fits the _unlocked unmap done by drm_prime_gem_destroy().

The leak was found by walking /proc/kpageflags, which attributed the
missing memory to orphaned shmem pages. Those matched the udmabuf
objects in dma_buf/bufinfo, and the remaining reference was traced to
the missing PRIME teardown. It was confirmed without a compositor using
a small reproducer: create a memfd, wrap it with UDMABUF_CREATE, import
it with DRM_IOCTL_PRIME_FD_TO_HANDLE on the vmwgfx render node, close
the GEM handle and every file descriptor, then check bufinfo.

Build tested on drm-misc-fixes (4600b4d1a9ee) with the openSUSE 7.2.3
config and W=1: drivers/gpu/drm/vmwgfx/ builds without warnings before
and after the change, and checkpatch.pl --strict is clean. Runtime
tested on a VirtualBox 7.2 VMSVGA guest with kernel 7.2.3, whose vmwgfx
sources for the files involved are identical to drm-misc-fixes, using a
vmwgfx.ko built from them with this change. With the reproducer, all 8
imported udmabufs stayed pinned without the change and none with it,
with 3D acceleration both on and off. With KWin on llvmpipe, killing
the compositor now released 84 of 92 udmabufs (610 MiB) within 10
seconds, where the same test on the unpatched driver released none.

Fixes: b32233acceff ("drm/vmwgfx: Fix prime import/export")
Cc: stable@vger.kernel.org # v6.6+
Assisted-by: LLM
Signed-off-by: Michal TOMA <michaltoma@sicoop.com>
---
Reproducer
----------
Run as a user with access to the render node and /dev/udmabuf:

#!/usr/bin/env python3
import fcntl, os, struct
N, SIZE = 8, 4 * 1024 * 1024
UDMABUF_CREATE     = 0x40187542  # _IOW('u', 0x42, struct udmabuf_create)
PRIME_FD_TO_HANDLE = 0xC00C642E  # DRM_IOWR(0x2e, struct drm_prime_handle)
GEM_CLOSE          = 0x40086409  # DRM_IOW(0x09, struct drm_gem_close)
render = os.open('/dev/dri/renderD128', os.O_RDWR | os.O_CLOEXEC)
udm = os.open('/dev/udmabuf', os.O_RDWR | os.O_CLOEXEC)
for i in range(N):
    mfd = os.memfd_create(f'prime-leak-{i}', os.MFD_ALLOW_SEALING)
    os.ftruncate(mfd, SIZE)
    fcntl.fcntl(mfd, fcntl.F_ADD_SEALS, fcntl.F_SEAL_SHRINK)
    create = bytearray(struct.pack('IIQQ', mfd, 1, 0, SIZE))
    dfd = fcntl.ioctl(udm, UDMABUF_CREATE, create)
    ph = bytearray(struct.pack('IIi', 0, 0, dfd))
    fcntl.ioctl(render, PRIME_FD_TO_HANDLE, ph)
    handle = struct.unpack('IIi', ph)[0]
    fcntl.ioctl(render, GEM_CLOSE, bytearray(struct.pack('II', handle, 0)))
    os.close(dfd)
    os.close(mfd)
os.close(udm)
os.close(render)

Then look at the 4 MiB udmabuf entries in
/sys/kernel/debug/dma_buf/bufinfo. Without the fix, all 8 are still
listed with a count of 1 and attached to the vmwgfx device after the
script has exited. The same loop without the PRIME_FD_TO_HANDLE and
GEM_CLOSE steps frees every udmabuf.

Test results
------------
Kernel 7.2.3, comparing the stock vmwgfx.ko with one built from the same
sources plus this patch:
- Reproducer, 8 x 4 MiB udmabufs: all 8 left on the stock driver, none
  left with the patch, with 3D acceleration on and off.
- KWin 6.7.4 on llvmpipe (3D off), after about 1.5 GiB of client
  buffers had been imported, killing kwin_wayland (kwin_wayland_wrapper
  restarts it):
    stock driver (7.2.2 at the time): udmabufs went 212 -> 220 and
      nothing was freed. Their refcounts dropped from mostly 3 to mostly
      1, still attached.
    patched: 84 of 92 udmabufs (610 MiB) released within 10 seconds, and
      MemAvailable rose from 1.50 to 2.20 GiB.
  The module used for the KWin run also carried a separate vmwgfx fix
  (see below), but that fix is not on this path: with 3D off, KWin's
  imports go through DRM_IOCTL_PRIME_FD_TO_HANDLE, which kprobes
  confirmed.
- The kernel log showed no vmwgfx warnings with the patched module.

While it runs on llvmpipe, KWin itself keeps many of these imports
referenced through GEM handles on its render node file. A 60-second
trace counted 130 import handles created and 119 deleted. That is a
separate userspace issue and is not addressed here; this patch makes
sure the memory is released once those handles go away.

With 3D acceleration on, KWin's imports go through
DRM_VMW_GB_SURFACE_REF_EXT instead, which has its own handle leak. I'm
sending a separate patch for that: "drm/vmwgfx: Don't leak a GEM handle
when referencing a surface by fd".

Test environment
----------------
- VirtualBox 7.2 (Guest Additions 7.2.16), VMSVGA adapter, 4 vCPUs,
  3.8 GiB RAM. Tested with 3D acceleration on (vmwgfx shader model SM_5)
  and off (shader model Legacy).
- openSUSE Tumbleweed kernel 7.2.3-1-default. vmwgfx_bo.c, vmwgfx_gem.c,
  vmwgfx_prime.c, ttm_object.c, vmwgfx_bo.h and vmwgfx_drv.h are
  byte-identical to drm-misc-fixes 4600b4d1a9ee. The patched module was
  built out of tree from those sources, which adds the E taint.
- The kernel was already tainted W+O before any test: two boot-time
  warnings unrelated to vmwgfx (arch/x86/mm/pat/set_memory.c:727 and
  kernel/rcu/tree_plugin.h:823), plus VirtualBox's out-of-tree
  vboxguest/vboxsf.
- KDE Plasma / KWin 6.7.4, Mesa 26.2.1.
- W=1 build of drivers/gpu/drm/vmwgfx/ on drm-misc-fixes: no warnings
  before or after the patch. checkpatch.pl --strict is clean.

Not tested
----------
- VMware Workstation or ESXi hosts; only VirtualBox VMSVGA.
- A full kernel built from drm-misc-fixes; only the 7.2.3 kernel with a
  module built from identical vmwgfx sources.
- Kernels with KASAN, lockdep or kmemleak enabled. IGT was not run.
- Stable backports were not built.

Classification
--------------
I'm treating this as a regular bug rather than a vulnerability, per
Documentation/process/threat-model.rst. The pinned pages are charged to
the importing process's memory cgroup, so the impact is bounded by
memory limits. The same class of reference leak (f739416dc555,
"drm/vmwgfx: drop dma_buf reference on foreign-fd prime import") was
handled as a regular fix.

Separate observation (not addressed here, untested)
---------------------------------------------------
vmw_prime_import_sg_table() returns NULL rather than an ERR_PTR when
vmw_bo_create() fails, because vmw_bo_create() sets *p_bo = NULL on
error. drm_gem_prime_import_dev() only checks IS_ERR() and then
dereferences the result (obj->import_attach = attach). This could only
trigger on buffer object allocation failure, and I have not tried to
reproduce it with fault injection.

Workaround for affected users
-----------------------------
Setting KWIN_DISABLE_UDMABUF_IMPORT=1 in the session environment makes
KWin 6.7 use its copy path again, which avoids the import.

Tool use
--------
I found the problem because my VM repeatedly ran out of memory and had
to be rebooted. An LLM coding assistant (Claude, Anthropic) did the
diagnostics that traced it back to this code, wrote the patch and this
changelog, and built the reproducer and the test tooling used to confirm
that the patch fixes the issue.

 drivers/gpu/drm/vmwgfx/vmwgfx_bo.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_bo.c b/drivers/gpu/drm/vmwgfx/vmwgfx_bo.c
index 9c7a73c0b..56bc94edc 100644
--- a/drivers/gpu/drm/vmwgfx/vmwgfx_bo.c
+++ b/drivers/gpu/drm/vmwgfx/vmwgfx_bo.c
@@ -30,6 +30,7 @@
 #include "vmwgfx_drv.h"
 #include "vmwgfx_resource_priv.h"
 
+#include <drm/drm_prime.h>
 #include <drm/ttm/ttm_placement.h>
 
 /**
@@ -69,6 +70,8 @@ static void vmw_bo_free(struct ttm_buffer_object *bo)
 		vmw_surface_unreference(&vbo->dumb_surface);
 	}
 	WARN_ON(!RB_EMPTY_ROOT(&vbo->res_tree));
+	if (drm_gem_is_imported(&vbo->tbo.base))
+		drm_prime_gem_destroy(&vbo->tbo.base, vbo->tbo.sg);
 	drm_gem_object_release(&vbo->tbo.base);
 	WARN_ON(vbo->dirty);
 	kfree(vbo);

base-commit: 4600b4d1a9ee730d03ddac5ce409cd2730ce8c0c
-- 
2.55.0

Re: [PATCH] drm/vmwgfx: Release PRIME import in the BO destroy path

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] Double free of the exporter's sg_table triggered by combining the newly added drm_prime_gem_destroy() with a pre-existing incorrect error path in vmw_ttm_map_dma().

Pre-existing issues:
- [High] NULL pointer dereference in drm_gem_prime_import_dev() because vmw_prime_import_sg_table() returns NULL instead of ERR_PTR on failure.
--

commit 9bf7a32b370ca37cb8db240835af8ec49f8b2120
Author: Michal TOMA <michaltoma@sicoop.com>

drm/vmwgfx: Release PRIME import in the BO destroy path

This commit addresses a reference leak for imported dma-buf objects by adding
a call to drm_prime_gem_destroy() in vmw_bo_free(). This ensures the sg mapping,
attachment, and dma-buf reference are properly released when the GEM object
is freed.

> diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_bo.c b/drivers/gpu/drm/vmwgfx/vmwgfx_bo.c
> index 9c7a73c0b0dc3..56bc94edc30b7 100644
> --- a/drivers/gpu/drm/vmwgfx/vmwgfx_bo.c
> +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_bo.c
[ ... ]
> @@ -69,6 +70,8 @@ static void vmw_bo_free(struct ttm_buffer_object *bo)
>  		vmw_surface_unreference(&vbo->dumb_surface);
>  	}
>  	WARN_ON(!RB_EMPTY_ROOT(&vbo->res_tree));
> +	if (drm_gem_is_imported(&vbo->tbo.base))
> +		drm_prime_gem_destroy(&vbo->tbo.base, vbo->tbo.sg);

[Severity: High]
Does this code cause a double free of the exporter's sg_table if mapping fails?

When an imported PRIME buffer (TTM_TT_FLAG_EXTERNAL) is mapped or bound,
vmw_ttm_map_dma() sets up the sg_table using the exporter's sg_table:

drivers/gpu/drm/vmwgfx/vmwgfx_ttm_buffer.c:vmw_ttm_map_dma() {
    ...
    if (vmw_tt->dma_ttm.page_flags & TTM_TT_FLAG_EXTERNAL) {
        vsgt->sgt = vmw_tt->dma_ttm.sg;
    } else {
    ...
    }

    ret = vmw_ttm_map_for_dma(vmw_tt);
    if (unlikely(ret != 0))
        goto out_map_fail;
    ...
out_map_fail:
    drm_warn(&dev_priv->drm, "VSG table map failed!");
    sg_free_table(vsgt->sgt);
    vsgt->sgt = NULL;
    ...
}

In vmw_ttm_map_dma(), vmw_ttm_map_for_dma() attempts to call dma_map_sgtable()
on an uninitialized inline &vmw_tt->sgt, which predictably fails and jumps to
out_map_fail. The error path (as well as vmw_ttm_unmap_dma) incorrectly calls
sg_free_table() on the exporter's sg_table.

When the buffer object is subsequently destroyed, this newly added call to
drm_prime_gem_destroy() asks the exporter to unmap its attachment, causing the
exporter to call sg_free_table() again on the exact same scatterlist.

[Severity: High]
This isn't a bug introduced by this patch, but a pre-existing issue was exposed
by analyzing the error path: can a failure in vmw_bo_create() cause a NULL
pointer dereference?

If a local user imports a dma-buf via DRM_IOCTL_PRIME_FD_TO_HANDLE and
vmw_bo_create() fails (e.g., due to memory pressure),
vmw_prime_import_sg_table() returns NULL instead of propagating an ERR_PTR:

drivers/gpu/drm/vmwgfx/vmwgfx_gem.c:vmw_prime_import_sg_table() {
    ...
    ret = vmw_bo_create(dev_priv, &params, &vbo);
    if (ret != 0)
        goto out_no_bo;
    ...
out_no_bo:
    dma_resv_unlock(params.resv);
    return gem;
}

The caller expects an ERR_PTR on failure. Since IS_ERR(NULL) evaluates to
false, it proceeds to unconditionally assign to obj->import_attach,
dereferencing the NULL pointer:

drivers/gpu/drm/drm_prime.c:drm_gem_prime_import_dev() {
    ...
    obj = dev->driver->gem_prime_import_sg_table(dev, attach, sgt);
    if (IS_ERR(obj)) {
        ret = PTR_ERR(obj);
        goto fail_unmap;
    }

    obj->import_attach = attach;
    ...
}

>  	drm_gem_object_release(&vbo->tbo.base);
>  	WARN_ON(vbo->dirty);
>  	kfree(vbo);

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260910142359.13716-1-michaltoma@sicoop.com?part=1

Re: [PATCH] drm/vmwgfx: Release PRIME import in the BO destroy path

[Michal TOMA]

On Thu, Sep 10, 2026 at 03:00:34PM +0000, sashiko-bot@kernel.org wrote:
> - [High] Double free of the exporter's sg_table triggered by combining
> the newly added drm_prime_gem_destroy() with a pre-existing incorrect
> error path in vmw_ttm_map_dma().

Thanks - this is real, and it means the patch as posted must not be
applied on its own. I reproduced it on the affected guest. The
mechanism differs a little from the one described, and the outcome is
worse than a double free: it is a NULL pointer dereference in a TTM
workqueue, which wedged the machine.

Corrected mechanism
-------------------
sg_free_table() sets table->sgl = NULL but leaves table->orig_nents
alone, so the second sg_free_table() from the exporter is in fact a
no-op. The crash comes one step earlier, in the unmap:

  drm_prime_gem_destroy()
    dma_buf_unmap_attachment_unlocked()
      udmabuf unmap_udmabuf() -> put_sg_table()
        dma_unmap_sgtable(dev, sg, ...)
          dma_unmap_sg_attrs(dev, sgt->sgl = NULL, sgt->orig_nents = N)
            dma_direct_unmap_sg() -> for_each_sg(NULL, sg, N, i)

which dereferences NULL on the first iteration.

Why vmw_ttm_map_dma() fails for every imported buffer
-----------------------------------------------------
For TTM_TT_FLAG_EXTERNAL, vmw_ttm_map_dma() points vsgt->sgt at the
exporter's table:

	if (vmw_tt->dma_ttm.page_flags & TTM_TT_FLAG_EXTERNAL) {
		vsgt->sgt = vmw_tt->dma_ttm.sg;
	} else {
		vsgt->sgt = &vmw_tt->sgt;
		ret = sg_alloc_table_from_pages_segment(&vmw_tt->sgt, ...);

but then calls vmw_ttm_map_for_dma(), which maps &vmw_tt->sgt - the
inline table, not the one vsgt->sgt points to. For an imported buffer
that inline table is never populated (vmw_ttm_tt_create() kzallocs the
vmw_ttm_tt), so dma_map_sgtable() is called with orig_nents == 0,
trips the WARN_ON_ONCE in __dma_map_sg_attrs() and returns -EIO. Then
out_map_fail calls sg_free_table() on the *exporter's* table.

That path is therefore taken on every attempt to bind an imported
buffer object, not only in some rare error case. vmw_ttm_unmap_dma()
would free the exporter's table too, if a map ever succeeded.

Note also that the exporter has already DMA-mapped that table for our
device in dma_buf_map_attachment(), so vmwgfx should not be mapping it
a second time at all.

Reachability
------------
An unprivileged process with access to the render node can get there
with three ioctls, all DRM_RENDER_ALLOW: import a dma-buf
(DRM_IOCTL_PRIME_FD_TO_HANDLE, e.g. a udmabuf), create a GB surface
backed by that buffer (DRM_VMW_GB_SURFACE_CREATE_EXT with
base.buffer_handle set), then submit SVGA_3D_CMD_UPDATE_GB_SURFACE for
that surface with DRM_VMW_EXECBUF. Validation moves the backing buffer
to VMW_BO_DOMAIN_MOB, which binds the TTM tt:

 WARNING: kernel/dma/mapping.c:266 at __dma_map_sg_attrs+0xdd/0x1d0
 Call Trace:
  dma_map_sgtable+0x1d/0x30
  vmw_ttm_map_dma+0xf6/0x140 [vmwgfx]
  vmw_move+0x1cd/0x2c0 [vmwgfx]
  ttm_bo_handle_move_mem+0xc0/0x180 [ttm]
  ttm_bo_validate+0xd2/0x1d0 [ttm]
  vmw_validation_bo_validate+0xb5/0x180 [vmwgfx]
  vmw_execbuf_process+0x852/0x1330 [vmwgfx]
  vmw_execbuf_ioctl+0x10d/0x1d0 [vmwgfx]
  drm_ioctl_kernel+0xa6/0x100
  drm_ioctl+0x2ad/0x590
  __x64_sys_ioctl+0xb9/0x100
  do_syscall_64+0xe1/0x610

 vmwgfx 0000:00:02.0: [drm] VSG table map failed!

execbuf returns -EIO, but the exporter's sg_table has already been
freed. Releasing the buffer afterwards, with my patch applied:

 BUG: kernel NULL pointer dereference, address: 000000000000001c
 Oops: 0000 [#1] SMP NOPTI
 Workqueue: ttm ttm_bo_delayed_delete [ttm]
 RIP: 0010:dma_direct_unmap_sg+0x62/0x200
 Call Trace:
  unmap_udmabuf+0x24/0x40
  dma_buf_unmap_attachment+0x43/0x80
  dma_buf_unmap_attachment_unlocked+0x46/0x70
  drm_prime_gem_destroy+0x28/0x50
  vmw_bo_free+0x15b/0x1f0 [vmwgfx]
  process_one_work+0x19f/0x370
  worker_thread+0x1b1/0x310
  kthread+0xe4/0x120

Since this runs in the TTM delayed-delete worker rather than in the
caller, the closing process exits normally and the worker dies holding
TTM/dma-resv locks. The desktop wedged a few seconds later and the VM
had to be reset.

Without my patch the same sequence only corrupts and leaks the
exporter's mapping, because nothing ever unmaps the attachment - which
is exactly the leak I was fixing. So the patch does not create the bug,
but it does turn it into a locally triggerable crash.

What I intend to send
---------------------
Please do not apply the posted patch as it stands. I will send a
series instead:

  1) fix vmw_ttm_map_dma()/vmw_ttm_unmap_dma() for external TTM tt:
     use the exporter's already-mapped sg_table as-is, and never map,
     unmap or free it. Fixes: b32233acceff ("drm/vmwgfx: Fix prime
     import/export")
  2) the drm_prime_gem_destroy() cleanup posted here, unchanged
  3) a second leak that was ready but held back: the GEM handle
     vmw_buffer_prime_to_surface_base() creates for the fd passed to
     DRM_VMW_GB_SURFACE_REF_EXT is never returned to userspace and
     never deleted. Fixes: d6667f0ddf46 ("drm/vmwgfx: Fix handling of
     dumb buffers")

The other point in the review, vmw_prime_import_sg_table() returning
NULL rather than an ERR_PTR when vmw_bo_create() fails, is real as
well. I am leaving it out of the series: it needs a buffer object
allocation failure to reach, I have not managed to trigger it, and I
would rather not post a patch I could not test. It is called out in the
cover letter.

The proposed 1) looks like this; I will post it with test results:

@@ vmw_ttm_map_dma()
 	case vmw_dma_map_bind:
 	case vmw_dma_map_populate:
 		if (vmw_tt->dma_ttm.page_flags & TTM_TT_FLAG_EXTERNAL) {
+			/*
+			 * The exporter has already mapped its sg_table for
+			 * this device in dma_buf_map_attachment(). Use it
+			 * as-is; it is not ours to map or to free.
+			 */
 			vsgt->sgt = vmw_tt->dma_ttm.sg;
-		} else {
-			vsgt->sgt = &vmw_tt->sgt;
-			ret = sg_alloc_table_from_pages_segment(...);
-			if (ret)
-				goto out_sg_alloc_fail;
+			break;
 		}
+		vsgt->sgt = &vmw_tt->sgt;
+		ret = sg_alloc_table_from_pages_segment(...);
+		if (ret)
+			goto out_sg_alloc_fail;
 		ret = vmw_ttm_map_for_dma(vmw_tt);

@@ vmw_ttm_unmap_dma()
+	if (vmw_tt->dma_ttm.page_flags & TTM_TT_FLAG_EXTERNAL) {
+		vmw_tt->vsgt.sgt = NULL;
+		vmw_tt->mapped = false;
+		return;
+	}

I have 1) built and tested here. With it, the reproducer above logs
nothing at all: no WARN_ON_ONCE(), no "VSG table map failed!", and
DRM_VMW_EXECBUF is accepted instead of returning -EIO, so an imported
buffer object can now be bound at all. Releasing the buffer afterwards
leaves no entry in /sys/kernel/debug/dma_buf/bufinfo and no oops, and
the reproducers for the two leaks from the original posting still
behave as they did.

The reproducer for the crash above, and the ones for the two leaks, are
small C and Python programs; each is included below the cut of the
patch it belongs to in the series.

The analysis and the reproducers here were produced the same way as
described in the tool-use note of the original patch.