[PATCH v2] drm/i915/dmabuf: avoid global wbinvd on dma-buf import

[PATCH v2] drm/i915/dmabuf: avoid global wbinvd on dma-buf import

[Prabhakaran, Krishna]

From: Krishna Prabhakaran <krishna.prabhakaran@intel.com>

When i915 needs to make an imported dma-buf coherent for GPU access on
non-LLC platforms, or for objects that bypass LLC, it currently calls
wbinvd_on_all_cpus(). get_pages() runs whenever an imported buffer is
pinned, so this triggers a whole-cache write-back and invalidate,
broadcast by IPI to every CPU, on every execbuf submission involving an
imported dma-buf. That stalls the entire machine for milliseconds and
starves latency-sensitive work on unrelated cores (e.g. USB isochronous
audio serviced on the VMM's main thread).

Flush only the pages that actually need it instead:

 - If we imported one of our own dma-bufs, the backing object is
   struct-page backed once migrated to SMEM, so flush it directly with
   drm_clflush_sg(), exactly as we flush our other objects. This also
   avoids re-entering the exporter through dma_buf_vmap(), which would
   recurse into i915_gem_object_pin_map() on the source object we
   already hold locked (and fails the igt_dmabuf_import_same_driver
   selftest with -EBUSY).

 - For a foreign dma-buf the sg_table is not guaranteed to be backed by
   struct pages, and the importer has no way to tell, so drm_clflush_sg()
   cannot be used. vmap the buffer and flush that virtual range with
   drm_clflush_virt_range() instead: x86 uses PIPT caches, so flushing
   one virtual alias evicts the cache lines for every alias of the same
   physical pages. The dma_resv lock required by dma_buf_vmap() is
   already held here via the imported object.

Fall back to wbinvd only when the buffer cannot be vmapped or is backed
by I/O memory, where there is no CPU-side range to clflush.

Fixes: a035154da45d ("drm/i915/dmabuf: add paranoid flush-on-acquire")
Signed-off-by: Krishna Prabhakaran <krishna.prabhakaran@intel.com>
---
v2:
 - Flush our own imported dma-bufs directly with drm_clflush_sg() instead of
   dma_buf_vmap(), which re-entered i915_gem_object_pin_map() on the source
   object and failed igt_dmabuf_import_same_driver_smem with -EBUSY (reported
   by Intel CI on v1). Foreign dma-bufs still use dma_buf_vmap() +
   drm_clflush_virt_range(); wbinvd only as fallback.
 - Link to v1: https://patchwork.freedesktop.org/patch/744955/?series=171760

 drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 55 ++++++++++++++++++----
 1 file changed, 47 insertions(+), 8 deletions(-)

diff --git a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
index b43d34c7d641..c798a90f1c0f 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
@@ -10,6 +10,8 @@
 
 #include <asm/smp.h>
 
+#include <drm/drm_cache.h>
+
 #include "gem/i915_gem_dmabuf.h"
 #include "i915_drv.h"
 #include "i915_gem_object.h"
@@ -249,16 +251,53 @@ static int i915_gem_object_get_pages_dmabuf(struct drm_i915_gem_object *obj)
 	 * DG1 is special here since it still snoops transactions even with
 	 * CACHE_NONE. This is not the case with other HAS_SNOOP platforms. We
 	 * might need to revisit this as we add new discrete platforms.
-	 *
-	 * XXX: Consider doing a vmap flush or something, where possible.
-	 * Currently we just do a heavy handed wbinvd_on_all_cpus() here since
-	 * the underlying sg_table might not even point to struct pages, so we
-	 * can't just call drm_clflush_sg or similar, like we do elsewhere in
-	 * the driver.
 	 */
 	if (i915_gem_object_can_bypass_llc(obj) ||
-	    (!HAS_LLC(i915) && !IS_DG1(i915)))
-		wbinvd_on_all_cpus();
+	    (!HAS_LLC(i915) && !IS_DG1(i915))) {
+		struct dma_buf *dma_buf = obj->base.import_attach->dmabuf;
+
+		if (dma_buf->ops == &i915_dmabuf_ops) {
+			struct drm_i915_gem_object *dma_obj =
+				dma_buf_to_obj(dma_buf);
+
+			/*
+			 * We imported one of our own dma-bufs. The backing
+			 * object is struct-page backed once migrated to SMEM,
+			 * so flush it directly, the same way we flush our
+			 * other objects. This also avoids re-entering the
+			 * exporter through dma_buf_vmap(), which would recurse
+			 * into i915_gem_object_pin_map() on the source object
+			 * we already hold locked.
+			 */
+			if (i915_gem_object_has_struct_page(dma_obj))
+				drm_clflush_sg(dma_obj->mm.pages);
+			else
+				wbinvd_on_all_cpus();
+		} else {
+			struct iosys_map map;
+
+			/*
+			 * A foreign sg_table is not guaranteed to be backed by
+			 * struct pages, so we cannot use drm_clflush_sg(). vmap
+			 * the buffer and flush the virtual range instead; x86
+			 * uses PIPT caches, so flushing one alias evicts the
+			 * lines for every alias of the same physical pages.
+			 *
+			 * We already hold the dma_resv lock via the imported
+			 * obj, so use the locked dma_buf_vmap() variant.
+			 */
+			if (!dma_buf_vmap(dma_buf, &map)) {
+				if (!map.is_iomem)
+					drm_clflush_virt_range(map.vaddr,
+							       obj->base.size);
+				else
+					wbinvd_on_all_cpus();
+				dma_buf_vunmap(dma_buf, &map);
+			} else {
+				wbinvd_on_all_cpus();
+			}
+		}
+	}
 
 	__i915_gem_object_set_pages(obj, sgt);
 

base-commit: 682ea2d28d18bb06f9fc663cb5ab7e80dc0e606a
-- 
2.43.0

Re: [PATCH v2] drm/i915/dmabuf: avoid global wbinvd on dma-buf import

[Matthew Auld]

On 16/08/2026 02:45, Prabhakaran, Krishna wrote:
> From: Krishna Prabhakaran <krishna.prabhakaran@intel.com>
> 
> When i915 needs to make an imported dma-buf coherent for GPU access on
> non-LLC platforms, or for objects that bypass LLC, it currently calls
> wbinvd_on_all_cpus(). get_pages() runs whenever an imported buffer is
> pinned, so this triggers a whole-cache write-back and invalidate,
> broadcast by IPI to every CPU, on every execbuf submission involving an
> imported dma-buf. That stalls the entire machine for milliseconds and
> starves latency-sensitive work on unrelated cores (e.g. USB isochronous
> audio serviced on the VMM's main thread).
> 
> Flush only the pages that actually need it instead:
> 
>   - If we imported one of our own dma-bufs, the backing object is
>     struct-page backed once migrated to SMEM, so flush it directly with
>     drm_clflush_sg(), exactly as we flush our other objects. This also
>     avoids re-entering the exporter through dma_buf_vmap(), which would
>     recurse into i915_gem_object_pin_map() on the source object we
>     already hold locked (and fails the igt_dmabuf_import_same_driver
>     selftest with -EBUSY).
> 
>   - For a foreign dma-buf the sg_table is not guaranteed to be backed by
>     struct pages, and the importer has no way to tell, so drm_clflush_sg()
>     cannot be used. vmap the buffer and flush that virtual range with
>     drm_clflush_virt_range() instead: x86 uses PIPT caches, so flushing
>     one virtual alias evicts the cache lines for every alias of the same
>     physical pages. The dma_resv lock required by dma_buf_vmap() is
>     already held here via the imported object.
> 
> Fall back to wbinvd only when the buffer cannot be vmapped or is backed
> by I/O memory, where there is no CPU-side range to clflush.
> 
> Fixes: a035154da45d ("drm/i915/dmabuf: add paranoid flush-on-acquire")
> Signed-off-by: Krishna Prabhakaran <krishna.prabhakaran@intel.com>
> ---
> v2:
>   - Flush our own imported dma-bufs directly with drm_clflush_sg() instead of
>     dma_buf_vmap(), which re-entered i915_gem_object_pin_map() on the source
>     object and failed igt_dmabuf_import_same_driver_smem with -EBUSY (reported
>     by Intel CI on v1). Foreign dma-bufs still use dma_buf_vmap() +
>     drm_clflush_virt_range(); wbinvd only as fallback.
>   - Link to v1: https://patchwork.freedesktop.org/patch/744955/?series=171760
> 
>   drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 55 ++++++++++++++++++----
>   1 file changed, 47 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
> index b43d34c7d641..c798a90f1c0f 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
> @@ -10,6 +10,8 @@
>   
>   #include <asm/smp.h>
>   
> +#include <drm/drm_cache.h>
> +
>   #include "gem/i915_gem_dmabuf.h"
>   #include "i915_drv.h"
>   #include "i915_gem_object.h"
> @@ -249,16 +251,53 @@ static int i915_gem_object_get_pages_dmabuf(struct drm_i915_gem_object *obj)
>   	 * DG1 is special here since it still snoops transactions even with
>   	 * CACHE_NONE. This is not the case with other HAS_SNOOP platforms. We
>   	 * might need to revisit this as we add new discrete platforms.
> -	 *
> -	 * XXX: Consider doing a vmap flush or something, where possible.
> -	 * Currently we just do a heavy handed wbinvd_on_all_cpus() here since
> -	 * the underlying sg_table might not even point to struct pages, so we
> -	 * can't just call drm_clflush_sg or similar, like we do elsewhere in
> -	 * the driver.
>   	 */
>   	if (i915_gem_object_can_bypass_llc(obj) ||
> -	    (!HAS_LLC(i915) && !IS_DG1(i915)))
> -		wbinvd_on_all_cpus();
> +	    (!HAS_LLC(i915) && !IS_DG1(i915))) {

I think we can bump this now for dg2? I think we treat dgfx as always 
coherent with system memory. So maybe s/IS_DG1/IS_DGFX/ in a separate 
patch? Pretty sure the rest of the driver is the same.

> +		struct dma_buf *dma_buf = obj->base.import_attach->dmabuf;
> +
> +		if (dma_buf->ops == &i915_dmabuf_ops) {
> +			struct drm_i915_gem_object *dma_obj =
> +				dma_buf_to_obj(dma_buf);
> +
> +			/*
> +			 * We imported one of our own dma-bufs. The backing
> +			 * object is struct-page backed once migrated to SMEM,
> +			 * so flush it directly, the same way we flush our
> +			 * other objects. This also avoids re-entering the
> +			 * exporter through dma_buf_vmap(), which would recurse
> +			 * into i915_gem_object_pin_map() on the source object
> +			 * we already hold locked.
> +			 */
> +			if (i915_gem_object_has_struct_page(dma_obj))
> +				drm_clflush_sg(dma_obj->mm.pages);
> +			else
> +				wbinvd_on_all_cpus();

Do we need the flush under the else here? If it's not placed in system 
memory what is this flushing, from i915 pov?

> +		} else {
> +			struct iosys_map map;
> +
> +			/*
> +			 * A foreign sg_table is not guaranteed to be backed by
> +			 * struct pages, so we cannot use drm_clflush_sg(). vmap
> +			 * the buffer and flush the virtual range instead; x86
> +			 * uses PIPT caches, so flushing one alias evicts the
> +			 * lines for every alias of the same physical pages.
> +			 *
> +			 * We already hold the dma_resv lock via the imported
> +			 * obj, so use the locked dma_buf_vmap() variant.
> +			 */
> +			if (!dma_buf_vmap(dma_buf, &map)) {
> +				if (!map.is_iomem)
> +					drm_clflush_virt_range(map.vaddr,
> +							       obj->base.size);
> +				else
> +					wbinvd_on_all_cpus();
> +				dma_buf_vunmap(dma_buf, &map);
> +			} else {
> +				wbinvd_on_all_cpus();
> +			}
> +		}
> +	}
>   
>   	__i915_gem_object_set_pages(obj, sgt);
>   
> 
> base-commit: 682ea2d28d18bb06f9fc663cb5ab7e80dc0e606a

Re: [PATCH v2] drm/i915/dmabuf: avoid global wbinvd on dma-buf import

[Thomas Hellström]

On Thu, 2026-08-20 at 10:23 +0100, Matthew Auld wrote:
> On 16/08/2026 02:45, Prabhakaran, Krishna wrote:
> > From: Krishna Prabhakaran <krishna.prabhakaran@intel.com>
> > 
> > When i915 needs to make an imported dma-buf coherent for GPU access
> > on
> > non-LLC platforms, or for objects that bypass LLC, it currently
> > calls
> > wbinvd_on_all_cpus(). get_pages() runs whenever an imported buffer
> > is
> > pinned, so this triggers a whole-cache write-back and invalidate,
> > broadcast by IPI to every CPU, on every execbuf submission
> > involving an
> > imported dma-buf. That stalls the entire machine for milliseconds
> > and
> > starves latency-sensitive work on unrelated cores (e.g. USB
> > isochronous
> > audio serviced on the VMM's main thread).
> > 
> > Flush only the pages that actually need it instead:
> > 
> >   - If we imported one of our own dma-bufs, the backing object is
> >     struct-page backed once migrated to SMEM, so flush it directly
> > with
> >     drm_clflush_sg(), exactly as we flush our other objects. This
> > also
> >     avoids re-entering the exporter through dma_buf_vmap(), which
> > would
> >     recurse into i915_gem_object_pin_map() on the source object we
> >     already hold locked (and fails the
> > igt_dmabuf_import_same_driver
> >     selftest with -EBUSY).
> > 
> >   - For a foreign dma-buf the sg_table is not guaranteed to be
> > backed by
> >     struct pages, and the importer has no way to tell, so
> > drm_clflush_sg()
> >     cannot be used. vmap the buffer and flush that virtual range
> > with
> >     drm_clflush_virt_range() instead: x86 uses PIPT caches, so
> > flushing
> >     one virtual alias evicts the cache lines for every alias of the
> > same
> >     physical pages. The dma_resv lock required by dma_buf_vmap() is
> >     already held here via the imported object.
> > 
> > Fall back to wbinvd only when the buffer cannot be vmapped or is
> > backed
> > by I/O memory, where there is no CPU-side range to clflush.
> > 
> > Fixes: a035154da45d ("drm/i915/dmabuf: add paranoid flush-on-
> > acquire")
> > Signed-off-by: Krishna Prabhakaran <krishna.prabhakaran@intel.com>
> > ---
> > v2:
> >   - Flush our own imported dma-bufs directly with drm_clflush_sg()
> > instead of
> >     dma_buf_vmap(), which re-entered i915_gem_object_pin_map() on
> > the source
> >     object and failed igt_dmabuf_import_same_driver_smem with -
> > EBUSY (reported
> >     by Intel CI on v1). Foreign dma-bufs still use dma_buf_vmap() +
> >     drm_clflush_virt_range(); wbinvd only as fallback.
> >   - Link to v1:
> > https://patchwork.freedesktop.org/patch/744955/?series=171760
> > 
> >   drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 55
> > ++++++++++++++++++----
> >   1 file changed, 47 insertions(+), 8 deletions(-)
> > 
> > diff --git a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
> > b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
> > index b43d34c7d641..c798a90f1c0f 100644
> > --- a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
> > +++ b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
> > @@ -10,6 +10,8 @@
> >   
> >   #include <asm/smp.h>
> >   
> > +#include <drm/drm_cache.h>
> > +
> >   #include "gem/i915_gem_dmabuf.h"
> >   #include "i915_drv.h"
> >   #include "i915_gem_object.h"
> > @@ -249,16 +251,53 @@ static int
> > i915_gem_object_get_pages_dmabuf(struct drm_i915_gem_object *obj)
> >   	 * DG1 is special here since it still snoops transactions
> > even with
> >   	 * CACHE_NONE. This is not the case with other HAS_SNOOP
> > platforms. We
> >   	 * might need to revisit this as we add new discrete
> > platforms.
> > -	 *
> > -	 * XXX: Consider doing a vmap flush or something, where
> > possible.
> > -	 * Currently we just do a heavy handed
> > wbinvd_on_all_cpus() here since
> > -	 * the underlying sg_table might not even point to struct
> > pages, so we
> > -	 * can't just call drm_clflush_sg or similar, like we do
> > elsewhere in
> > -	 * the driver.
> >   	 */
> >   	if (i915_gem_object_can_bypass_llc(obj) ||
> > -	    (!HAS_LLC(i915) && !IS_DG1(i915)))
> > -		wbinvd_on_all_cpus();
> > +	    (!HAS_LLC(i915) && !IS_DG1(i915))) {
> 
> I think we can bump this now for dg2? I think we treat dgfx as always
> coherent with system memory. So maybe s/IS_DG1/IS_DGFX/ in a separate
> patch? Pretty sure the rest of the driver is the same.

+1

> 
> > +		struct dma_buf *dma_buf = obj->base.import_attach-
> > >dmabuf;
> > +
> > +		if (dma_buf->ops == &i915_dmabuf_ops) {
> > +			struct drm_i915_gem_object *dma_obj =
> > +				dma_buf_to_obj(dma_buf);
> > +
> > +			/*
> > +			 * We imported one of our own dma-bufs.
> > The backing
> > +			 * object is struct-page backed once
> > migrated to SMEM,
> > +			 * so flush it directly, the same way we
> > flush our
> > +			 * other objects. This also avoids re-
> > entering the
> > +			 * exporter through dma_buf_vmap(), which
> > would recurse
> > +			 * into i915_gem_object_pin_map() on the
> > source object
> > +			 * we already hold locked.
> > +			 */
> > +			if
> > (i915_gem_object_has_struct_page(dma_obj))
> > +				drm_clflush_sg(dma_obj->mm.pages);
> > +			else
> > +				wbinvd_on_all_cpus();
> 
> Do we need the flush under the else here? If it's not placed in
> system 
> memory what is this flushing, from i915 pov?
> 
> > +		} else {
> > +			struct iosys_map map;
> > +
> > +			/*
> > +			 * A foreign sg_table is not guaranteed to
> > be backed by
> > +			 * struct pages, so we cannot use
> > drm_clflush_sg(). vmap
> > +			 * the buffer and flush the virtual range
> > instead; x86
> > +			 * uses PIPT caches, so flushing one alias
> > evicts the
> > +			 * lines for every alias of the same
> > physical pages.
> > +			 *
> > +			 * We already hold the dma_resv lock via
> > the imported
> > +			 * obj, so use the locked dma_buf_vmap()
> > variant.
> > +			 */
> > +			if (!dma_buf_vmap(dma_buf, &map)) {
> > +				if (!map.is_iomem)
> > +					drm_clflush_virt_range(map
> > .vaddr,
> > +							      
> > obj->base.size);
> > +				else
> > +					wbinvd_on_all_cpus();

The need for this flush is a bit unfortunate. In theory iomem can be
mapped and wiped write-back, But I'm not sure how common that is...



> > +				dma_buf_vunmap(dma_buf, &map);



> > +			} else {
> > +				wbinvd_on_all_cpus();

Random idea, would it make sense to insert a GPU CLFLUSH into the
pipeline using MI_CLFLUSH (on hardware that supports it) or an
uncached but coherent MOCS / GPU PAT setting dummy blit?

Thanks,
Thomas


> > +			}
> > +		}
> > +	}
> >   
> >   	__i915_gem_object_set_pages(obj, sgt);
> >   
> > 
> > base-commit: 682ea2d28d18bb06f9fc663cb5ab7e80dc0e606a

Re: [PATCH v2] drm/i915/dmabuf: avoid global wbinvd on dma-buf import

[Prabhakaran, Krishna]

________________________________________
From: Auld, Matthew <matthew.auld@intel.com>
Sent: Thursday, August 20, 2026 2:23 AM
To: Prabhakaran, Krishna <krishna.prabhakaran@intel.com>; intel-gfx@lists.freedesktop.org <intel-gfx@lists.freedesktop.org>
Cc: jani.nikula@linux.intel.com <jani.nikula@linux.intel.com>; joonas.lahtinen@linux.intel.com <joonas.lahtinen@linux.intel.com>; Vivi, Rodrigo <rodrigo.vivi@intel.com>; tursulin@ursulin.net <tursulin@ursulin.net>; thomas.hellstrom@linux.intel.com <thomas.hellstrom@linux.intel.com>; dri-devel@lists.freedesktop.org <dri-devel@lists.freedesktop.org>
Subject: Re: [PATCH v2] drm/i915/dmabuf: avoid global wbinvd on dma-buf import
 
On 16/08/2026 02:45, Prabhakaran, Krishna wrote:
> From: Krishna Prabhakaran <krishna.prabhakaran@intel.com>
>
> When i915 needs to make an imported dma-buf coherent for GPU access on
> non-LLC platforms, or for objects that bypass LLC, it currently calls
> wbinvd_on_all_cpus(). get_pages() runs whenever an imported buffer is
> pinned, so this triggers a whole-cache write-back and invalidate,
> broadcast by IPI to every CPU, on every execbuf submission involving an
> imported dma-buf. That stalls the entire machine for milliseconds and
> starves latency-sensitive work on unrelated cores (e.g. USB isochronous
> audio serviced on the VMM's main thread).
>
> Flush only the pages that actually need it instead:
>
>   - If we imported one of our own dma-bufs, the backing object is
>     struct-page backed once migrated to SMEM, so flush it directly with
>     drm_clflush_sg(), exactly as we flush our other objects. This also
>     avoids re-entering the exporter through dma_buf_vmap(), which would
>     recurse into i915_gem_object_pin_map() on the source object we
>     already hold locked (and fails the igt_dmabuf_import_same_driver
>     selftest with -EBUSY).
>
>   - For a foreign dma-buf the sg_table is not guaranteed to be backed by
>     struct pages, and the importer has no way to tell, so drm_clflush_sg()
>     cannot be used. vmap the buffer and flush that virtual range with
>     drm_clflush_virt_range() instead: x86 uses PIPT caches, so flushing
>     one virtual alias evicts the cache lines for every alias of the same
>     physical pages. The dma_resv lock required by dma_buf_vmap() is
>     already held here via the imported object.
>
> Fall back to wbinvd only when the buffer cannot be vmapped or is backed
> by I/O memory, where there is no CPU-side range to clflush.
>
> Fixes: a035154da45d ("drm/i915/dmabuf: add paranoid flush-on-acquire")
> Signed-off-by: Krishna Prabhakaran <krishna.prabhakaran@intel.com>
> ---
> v2:
>   - Flush our own imported dma-bufs directly with drm_clflush_sg() instead of
>     dma_buf_vmap(), which re-entered i915_gem_object_pin_map() on the source
>     object and failed igt_dmabuf_import_same_driver_smem with -EBUSY (reported
>     by Intel CI on v1). Foreign dma-bufs still use dma_buf_vmap() +
>     drm_clflush_virt_range(); wbinvd only as fallback.
>   - Link to v1: https://patchwork.freedesktop.org/patch/744955/?series=171760
>
>   drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 55 ++++++++++++++++++----
>   1 file changed, 47 insertions(+), 8 deletions(-)
>
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
> index b43d34c7d641..c798a90f1c0f 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
> @@ -10,6 +10,8 @@
>  
>   #include <asm/smp.h>
>  
> +#include <drm/drm_cache.h>
> +
>   #include "gem/i915_gem_dmabuf.h"
>   #include "i915_drv.h"
>   #include "i915_gem_object.h"
> @@ -249,16 +251,53 @@ static int i915_gem_object_get_pages_dmabuf(struct drm_i915_gem_object *obj)
>         * DG1 is special here since it still snoops transactions even with
>         * CACHE_NONE. This is not the case with other HAS_SNOOP platforms. We
>         * might need to revisit this as we add new discrete platforms.
> -      *
> -      * XXX: Consider doing a vmap flush or something, where possible.
> -      * Currently we just do a heavy handed wbinvd_on_all_cpus() here since
> -      * the underlying sg_table might not even point to struct pages, so we
> -      * can't just call drm_clflush_sg or similar, like we do elsewhere in
> -      * the driver.
>         */
>        if (i915_gem_object_can_bypass_llc(obj) ||
> -         (!HAS_LLC(i915) && !IS_DG1(i915)))
> -             wbinvd_on_all_cpus();
> +         (!HAS_LLC(i915) && !IS_DG1(i915))) {

I think we can bump this now for dg2? I think we treat dgfx as always
coherent with system memory. So maybe s/IS_DG1/IS_DGFX/ in a separate
patch? Pretty sure the rest of the driver is the same.

Agreed. I've will send it as a separate patch, "drm/i915/dmabuf: skip
acquire flush on all discrete GPUs", with you as Suggested-by.

> +             struct dma_buf *dma_buf = obj->base.import_attach->dmabuf;
> +
> +             if (dma_buf->ops == &i915_dmabuf_ops) {
> +                     struct drm_i915_gem_object *dma_obj =
> +                             dma_buf_to_obj(dma_buf);
> +
> +                     /*
> +                      * We imported one of our own dma-bufs. The backing
> +                      * object is struct-page backed once migrated to SMEM,
> +                      * so flush it directly, the same way we flush our
> +                      * other objects. This also avoids re-entering the
> +                      * exporter through dma_buf_vmap(), which would recurse
> +                      * into i915_gem_object_pin_map() on the source object
> +                      * we already hold locked.
> +                      */
> +                     if (i915_gem_object_has_struct_page(dma_obj))
> +                             drm_clflush_sg(dma_obj->mm.pages);
> +                     else
> +                             wbinvd_on_all_cpus();

Do we need the flush under the else here? If it's not placed in system
memory what is this flushing, from i915 pov?

Right. i915_gem_dmabuf_attach() migrates the exporter to SMEM
(INTEL_REGION_SMEM) before get_pages() runs, so dma_obj is always
struct-page backed here and the else was effectively dead code; a wbinvd
of device memory doesn't make sense from i915's point of view. Will drop
in v3 - the own-dma-buf branch is now just:

	drm_clflush_sg(dma_obj->mm.pages);


> +             } else {
> +                     struct iosys_map map;
> +
> +                     /*
> +                      * A foreign sg_table is not guaranteed to be backed by
> +                      * struct pages, so we cannot use drm_clflush_sg(). vmap
> +                      * the buffer and flush the virtual range instead; x86
> +                      * uses PIPT caches, so flushing one alias evicts the
> +                      * lines for every alias of the same physical pages.
> +                      *
> +                      * We already hold the dma_resv lock via the imported
> +                      * obj, so use the locked dma_buf_vmap() variant.
> +                      */
> +                     if (!dma_buf_vmap(dma_buf, &map)) {
> +                             if (!map.is_iomem)
> +                                     drm_clflush_virt_range(map.vaddr,
> +                                                            obj->base.size);
> +                             else
> +                                     wbinvd_on_all_cpus();
> +                             dma_buf_vunmap(dma_buf, &map);
> +                     } else {
> +                             wbinvd_on_all_cpus();
> +                     }
> +             }
> +     }
>  
>        __i915_gem_object_set_pages(obj, sgt);
>  
>
> base-commit: 682ea2d28d18bb06f9fc663cb5ab7e80dc0e606a

Re: [PATCH v2] drm/i915/dmabuf: avoid global wbinvd on dma-buf import

[Prabhakaran, Krishna]

> I think we can bump this now for dg2? I think we treat dgfx as always
> coherent with system memory. So maybe s/IS_DG1/IS_DGFX/ in a separate
> patch? Pretty sure the rest of the driver is the same.

Agreed. I'll send that as a separate patch, "drm/i915/dmabuf: skip
acquire flush on all discrete GPUs", with you as Suggested-by.

> Do we need the flush under the else here? If it's not placed in system
> memory what is this flushing, from i915 pov?

Right. i915_gem_dmabuf_attach() migrates the exporter to SMEM
(INTEL_REGION_SMEM) before get_pages() runs, so dma_obj is always
struct-page backed here and the else was effectively dead code; a wbinvd
of device memory doesn't make sense from i915's point of view. Dropped
in v3 -- the own-dma-buf branch is now just:

	drm_clflush_sg(dma_obj->mm.pages);

Thanks
Krishna

Re: [PATCH v2] drm/i915/dmabuf: avoid global wbinvd on dma-buf import

[Prabhakaran, Krishna]

> +1

Agreed. Will send out a separate patch for that.

> The need for this flush is a bit unfortunate. In theory iomem can be
> mapped and wiped write-back, But I'm not sure how common that is...
> Random idea, would it make sense to insert a GPU CLFLUSH into the
> pipeline using MI_CLFLUSH (on hardware that supports it) or an
> uncached but coherent MOCS / GPU PAT setting dummy blit?

Agreed it's unfortunate. I'd like to keep this patch as the targeted fix
for avoiding the latencies added by wbinvd_on_all_cpus and look at
the GPU-side approach as a follow-up.

To make sure I chase the right thing: did you mean making imported
buffers GPU-coherent via PAT/MOCS so the GPU snoops and no acquire flush
is needed at all, or emitting a pipelined flush at submission time? A CPU
wbinvd/clflush and an MI/PIPE_CONTROL flush act on different caches, so I
want to be sure which direction you had in mind before reworking.

Thanks,
Krishna

[PATCH v3] drm/i915/dmabuf: avoid global wbinvd on dma-buf import

[Prabhakaran, Krishna]

From: Krishna Prabhakaran <krishna.prabhakaran@intel.com>

When i915 needs to make an imported dma-buf coherent for GPU access on
non-LLC platforms, or for objects that bypass LLC, it currently calls
wbinvd_on_all_cpus(). get_pages() runs whenever an imported buffer is
pinned, so this triggers a whole-cache write-back and invalidate,
broadcast by IPI to every CPU, on every execbuf submission involving an
imported dma-buf. That stalls the entire machine for milliseconds and
starves latency-sensitive work on unrelated cores (e.g. USB isochronous
audio serviced on the VMM's main thread).

Flush only the pages that actually need it instead:

 - If we imported one of our own dma-bufs, the backing object is
   struct-page backed once migrated to SMEM, so flush it directly with
   drm_clflush_sg(), exactly as we flush our other objects. This also
   avoids re-entering the exporter through dma_buf_vmap(), which would
   recurse into i915_gem_object_pin_map() on the source object we
   already hold locked (and fails the igt_dmabuf_import_same_driver
   selftest with -EBUSY).

 - For a foreign dma-buf the sg_table is not guaranteed to be backed by
   struct pages, and the importer has no way to tell, so drm_clflush_sg()
   cannot be used. vmap the buffer and flush that virtual range with
   drm_clflush_virt_range() instead: x86 uses PIPT caches, so flushing
   one virtual alias evicts the cache lines for every alias of the same
   physical pages. The dma_resv lock required by dma_buf_vmap() is
   already held here via the imported object.

Fall back to wbinvd only when the buffer cannot be vmapped or is backed
by I/O memory, where there is no CPU-side range to clflush.

Fixes: a035154da45d ("drm/i915/dmabuf: add paranoid flush-on-acquire")
Signed-off-by: Krishna Prabhakaran <krishna.prabhakaran@intel.com>
---
v3:
 - Drop the wbinvd fallback in the own-dma-buf branch: the exporter is
   migrated to SMEM on attach, so it is always struct-page backed here;
   flushing device memory made no sense from i915's point of view
   (Matthew Auld).

v2:
 - Flush our own imported dma-bufs directly with drm_clflush_sg() instead of
   dma_buf_vmap(), which re-entered i915_gem_object_pin_map() on the source
   object and failed igt_dmabuf_import_same_driver_smem with -EBUSY (reported
   by Intel CI on v1). Foreign dma-bufs still use dma_buf_vmap() +
   drm_clflush_virt_range(); wbinvd only as fallback.
 - Link to v1: https://patchwork.freedesktop.org/patch/744955/?series=171760

 drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 52 ++++++++++++++++++----
 1 file changed, 44 insertions(+), 8 deletions(-)

diff --git a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
index b43d34c7d641..fbf0e0d4b026 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
@@ -10,6 +10,8 @@
 
 #include <asm/smp.h>
 
+#include <drm/drm_cache.h>
+
 #include "gem/i915_gem_dmabuf.h"
 #include "i915_drv.h"
 #include "i915_gem_object.h"
@@ -249,16 +251,50 @@ static int i915_gem_object_get_pages_dmabuf(struct drm_i915_gem_object *obj)
 	 * DG1 is special here since it still snoops transactions even with
 	 * CACHE_NONE. This is not the case with other HAS_SNOOP platforms. We
 	 * might need to revisit this as we add new discrete platforms.
-	 *
-	 * XXX: Consider doing a vmap flush or something, where possible.
-	 * Currently we just do a heavy handed wbinvd_on_all_cpus() here since
-	 * the underlying sg_table might not even point to struct pages, so we
-	 * can't just call drm_clflush_sg or similar, like we do elsewhere in
-	 * the driver.
 	 */
 	if (i915_gem_object_can_bypass_llc(obj) ||
-	    (!HAS_LLC(i915) && !IS_DG1(i915)))
-		wbinvd_on_all_cpus();
+	    (!HAS_LLC(i915) && !IS_DG1(i915))) {
+		struct dma_buf *dma_buf = obj->base.import_attach->dmabuf;
+
+		if (dma_buf->ops == &i915_dmabuf_ops) {
+			struct drm_i915_gem_object *dma_obj =
+				dma_buf_to_obj(dma_buf);
+
+			/*
+			 * We imported one of our own dma-bufs. The exporter is
+			 * migrated to SMEM on attach, so it is struct-page
+			 * backed and we can flush it directly, the same way we
+			 * flush our other objects. This also avoids re-entering
+			 * the exporter through dma_buf_vmap(), which would
+			 * recurse into i915_gem_object_pin_map() on the source
+			 * object we already hold locked.
+			 */
+			drm_clflush_sg(dma_obj->mm.pages);
+		} else {
+			struct iosys_map map;
+
+			/*
+			 * A foreign sg_table is not guaranteed to be backed by
+			 * struct pages, so we cannot use drm_clflush_sg(). vmap
+			 * the buffer and flush the virtual range instead; x86
+			 * uses PIPT caches, so flushing one alias evicts the
+			 * lines for every alias of the same physical pages.
+			 *
+			 * We already hold the dma_resv lock via the imported
+			 * obj, so use the locked dma_buf_vmap() variant.
+			 */
+			if (!dma_buf_vmap(dma_buf, &map)) {
+				if (!map.is_iomem)
+					drm_clflush_virt_range(map.vaddr,
+							       obj->base.size);
+				else
+					wbinvd_on_all_cpus();
+				dma_buf_vunmap(dma_buf, &map);
+			} else {
+				wbinvd_on_all_cpus();
+			}
+		}
+	}
 
 	__i915_gem_object_set_pages(obj, sgt);
 

base-commit: 682ea2d28d18bb06f9fc663cb5ab7e80dc0e606a
-- 
2.43.0

Re: [PATCH v3] drm/i915/dmabuf: avoid global wbinvd on dma-buf import

[Matthew Auld]

On 20/08/2026 21:50, Prabhakaran, Krishna wrote:
> From: Krishna Prabhakaran <krishna.prabhakaran@intel.com>
> 
> When i915 needs to make an imported dma-buf coherent for GPU access on
> non-LLC platforms, or for objects that bypass LLC, it currently calls
> wbinvd_on_all_cpus(). get_pages() runs whenever an imported buffer is
> pinned, so this triggers a whole-cache write-back and invalidate,
> broadcast by IPI to every CPU, on every execbuf submission involving an
> imported dma-buf. That stalls the entire machine for milliseconds and
> starves latency-sensitive work on unrelated cores (e.g. USB isochronous
> audio serviced on the VMM's main thread).
> 
> Flush only the pages that actually need it instead:
> 
>   - If we imported one of our own dma-bufs, the backing object is
>     struct-page backed once migrated to SMEM, so flush it directly with
>     drm_clflush_sg(), exactly as we flush our other objects. This also
>     avoids re-entering the exporter through dma_buf_vmap(), which would
>     recurse into i915_gem_object_pin_map() on the source object we
>     already hold locked (and fails the igt_dmabuf_import_same_driver
>     selftest with -EBUSY).
> 
>   - For a foreign dma-buf the sg_table is not guaranteed to be backed by
>     struct pages, and the importer has no way to tell, so drm_clflush_sg()
>     cannot be used. vmap the buffer and flush that virtual range with
>     drm_clflush_virt_range() instead: x86 uses PIPT caches, so flushing
>     one virtual alias evicts the cache lines for every alias of the same
>     physical pages. The dma_resv lock required by dma_buf_vmap() is
>     already held here via the imported object.
> 
> Fall back to wbinvd only when the buffer cannot be vmapped or is backed
> by I/O memory, where there is no CPU-side range to clflush.
> 
> Fixes: a035154da45d ("drm/i915/dmabuf: add paranoid flush-on-acquire")
> Signed-off-by: Krishna Prabhakaran <krishna.prabhakaran@intel.com>
> ---
> v3:
>   - Drop the wbinvd fallback in the own-dma-buf branch: the exporter is
>     migrated to SMEM on attach, so it is always struct-page backed here;
>     flushing device memory made no sense from i915's point of view
>     (Matthew Auld).
> 
> v2:
>   - Flush our own imported dma-bufs directly with drm_clflush_sg() instead of
>     dma_buf_vmap(), which re-entered i915_gem_object_pin_map() on the source
>     object and failed igt_dmabuf_import_same_driver_smem with -EBUSY (reported
>     by Intel CI on v1). Foreign dma-bufs still use dma_buf_vmap() +
>     drm_clflush_virt_range(); wbinvd only as fallback.
>   - Link to v1: https://patchwork.freedesktop.org/patch/744955/?series=171760
> 
>   drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 52 ++++++++++++++++++----
>   1 file changed, 44 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
> index b43d34c7d641..fbf0e0d4b026 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
> @@ -10,6 +10,8 @@
>   
>   #include <asm/smp.h>
>   
> +#include <drm/drm_cache.h>
> +
>   #include "gem/i915_gem_dmabuf.h"
>   #include "i915_drv.h"
>   #include "i915_gem_object.h"
> @@ -249,16 +251,50 @@ static int i915_gem_object_get_pages_dmabuf(struct drm_i915_gem_object *obj)
>   	 * DG1 is special here since it still snoops transactions even with
>   	 * CACHE_NONE. This is not the case with other HAS_SNOOP platforms. We
>   	 * might need to revisit this as we add new discrete platforms.
> -	 *
> -	 * XXX: Consider doing a vmap flush or something, where possible.
> -	 * Currently we just do a heavy handed wbinvd_on_all_cpus() here since
> -	 * the underlying sg_table might not even point to struct pages, so we
> -	 * can't just call drm_clflush_sg or similar, like we do elsewhere in
> -	 * the driver.
>   	 */
>   	if (i915_gem_object_can_bypass_llc(obj) ||
> -	    (!HAS_LLC(i915) && !IS_DG1(i915)))
> -		wbinvd_on_all_cpus();
> +	    (!HAS_LLC(i915) && !IS_DG1(i915))) {
> +		struct dma_buf *dma_buf = obj->base.import_attach->dmabuf;
> +
> +		if (dma_buf->ops == &i915_dmabuf_ops) {
> +			struct drm_i915_gem_object *dma_obj =
> +				dma_buf_to_obj(dma_buf);
> +
> +			/*
> +			 * We imported one of our own dma-bufs. The exporter is
> +			 * migrated to SMEM on attach, so it is struct-page
> +			 * backed and we can flush it directly, the same way we
> +			 * flush our other objects. This also avoids re-entering
> +			 * the exporter through dma_buf_vmap(), which would
> +			 * recurse into i915_gem_object_pin_map() on the source
> +			 * object we already hold locked.
> +			 */
> +			drm_clflush_sg(dma_obj->mm.pages);

I think we still need to wrap this with has_struct_page()?

> +		} else {
> +			struct iosys_map map;
> +
> +			/*
> +			 * A foreign sg_table is not guaranteed to be backed by
> +			 * struct pages, so we cannot use drm_clflush_sg(). vmap
> +			 * the buffer and flush the virtual range instead; x86
> +			 * uses PIPT caches, so flushing one alias evicts the
> +			 * lines for every alias of the same physical pages.
> +			 *
> +			 * We already hold the dma_resv lock via the imported
> +			 * obj, so use the locked dma_buf_vmap() variant.
> +			 */
> +			if (!dma_buf_vmap(dma_buf, &map)) {
> +				if (!map.is_iomem)
> +					drm_clflush_virt_range(map.vaddr,
> +							       obj->base.size);
> +				else
> +					wbinvd_on_all_cpus();
> +				dma_buf_vunmap(dma_buf, &map);
> +			} else {
> +				wbinvd_on_all_cpus();
> +			}
> +		}
> +	}
>   
>   	__i915_gem_object_set_pages(obj, sgt);
>   
> 
> base-commit: 682ea2d28d18bb06f9fc663cb5ab7e80dc0e606a

Re: [PATCH v3] drm/i915/dmabuf: avoid global wbinvd on dma-buf import

[Prabhakaran, Krishna]

[-- Attachment #1: Type: text/plain, Size: 437 bytes --]

> I think we still need to wrap this with has_struct_page()?

You're right. SMEM doesn't guarantee the STRUCT_PAGE flag (phys objects clear it), so I'll guard the direct-flush with i915_gem_object_has_struct_page() and fall back to wbinvd for the !struct_page case — I can't fall through to the vmap branch there since that's the recursion into pin_map() on the locked source object that v2 fixed. Will send a v4.

Regards
Krishna

[-- Attachment #2: Type: text/html, Size: 1455 bytes --]

[PATCH v4] drm/i915/dmabuf: avoid global wbinvd on dma-buf import

[Prabhakaran, Krishna]

From: Krishna Prabhakaran <krishna.prabhakaran@intel.com>

When i915 needs to make an imported dma-buf coherent for GPU access on
non-LLC platforms, or for objects that bypass LLC, it currently calls
wbinvd_on_all_cpus(). get_pages() runs whenever an imported buffer is
pinned, so this triggers a whole-cache write-back and invalidate,
broadcast by IPI to every CPU, on every execbuf submission involving an
imported dma-buf. That stalls the entire machine for milliseconds and
starves latency-sensitive work on unrelated cores (e.g. USB isochronous
audio serviced on the VMM's main thread).

Flush only the pages that actually need it instead:

 - If we imported one of our own dma-bufs, the backing object is
   normally struct-page backed once migrated to SMEM, so flush it
   directly with drm_clflush_sg(), exactly as we flush our other
   objects. This also avoids re-entering the exporter through
   dma_buf_vmap(), which would recurse into i915_gem_object_pin_map()
   on the source object we already hold locked (and fails the
   igt_dmabuf_import_same_driver selftest with -EBUSY). SMEM does not
   guarantee the struct-page flag though: phys objects clear it and it
   is dropped transiently during TTM moves, and drm_clflush_sg() walks
   the sgt with sg_page(), so guard the direct flush with
   i915_gem_object_has_struct_page() and fall back to wbinvd when it is
   not set.

 - For a foreign dma-buf the sg_table is not guaranteed to be backed by
   struct pages, and the importer has no way to tell, so drm_clflush_sg()
   cannot be used. vmap the buffer and flush that virtual range with
   drm_clflush_virt_range() instead: x86 uses PIPT caches, so flushing
   one virtual alias evicts the cache lines for every alias of the same
   physical pages. The dma_resv lock required by dma_buf_vmap() is
   already held here via the imported object.

Fall back to wbinvd only when the buffer cannot be vmapped or is backed
by I/O memory, where there is no CPU-side range to clflush.

Fixes: a035154da45d ("drm/i915/dmabuf: add paranoid flush-on-acquire")
Signed-off-by: Krishna Prabhakaran <krishna.prabhakaran@intel.com>
---
v4: Guard the own-dma-buf direct drm_clflush_sg() with
    i915_gem_object_has_struct_page() and fall back to wbinvd when the
    SMEM object is not struct-page backed (phys objects, transient TTM
    moves); sg_page() would otherwise walk bogus pages. (review feedback)
v3: Drop the wbinvd fallback inside the own-dma-buf branch; the exporter is
    migrated to SMEM on attach, so flushing device memory made no sense.
    (Matthew Auld)
v2: Special-case our own dma-bufs with drm_clflush_sg() instead of
    dma_buf_vmap(), which recursed into i915_gem_object_pin_map() on the
    already-locked source object and failed igt_dmabuf_import_same_driver
    with -EBUSY; foreign dma-bufs use vmap + drm_clflush_virt_range().
v1: https://patchwork.freedesktop.org/series/171760/
 drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 62 +++++++++++++++++++---
 1 file changed, 54 insertions(+), 8 deletions(-)

diff --git a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
index b43d34c7d641..d1695171019c 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
@@ -10,6 +10,8 @@
 
 #include <asm/smp.h>
 
+#include <drm/drm_cache.h>
+
 #include "gem/i915_gem_dmabuf.h"
 #include "i915_drv.h"
 #include "i915_gem_object.h"
@@ -249,16 +251,60 @@ static int i915_gem_object_get_pages_dmabuf(struct drm_i915_gem_object *obj)
 	 * DG1 is special here since it still snoops transactions even with
 	 * CACHE_NONE. This is not the case with other HAS_SNOOP platforms. We
 	 * might need to revisit this as we add new discrete platforms.
-	 *
-	 * XXX: Consider doing a vmap flush or something, where possible.
-	 * Currently we just do a heavy handed wbinvd_on_all_cpus() here since
-	 * the underlying sg_table might not even point to struct pages, so we
-	 * can't just call drm_clflush_sg or similar, like we do elsewhere in
-	 * the driver.
 	 */
 	if (i915_gem_object_can_bypass_llc(obj) ||
-	    (!HAS_LLC(i915) && !IS_DG1(i915)))
-		wbinvd_on_all_cpus();
+	    (!HAS_LLC(i915) && !IS_DG1(i915))) {
+		struct dma_buf *dma_buf = obj->base.import_attach->dmabuf;
+
+		if (dma_buf->ops == &i915_dmabuf_ops) {
+			struct drm_i915_gem_object *dma_obj =
+				dma_buf_to_obj(dma_buf);
+
+			/*
+			 * We imported one of our own dma-bufs. The exporter is
+			 * migrated to SMEM on attach, so it is normally struct-page
+			 * backed and we can flush it directly, the same way we
+			 * flush our other objects. This also avoids re-entering
+			 * the exporter through dma_buf_vmap(), which would
+			 * recurse into i915_gem_object_pin_map() on the source
+			 * object we already hold locked.
+			 *
+			 * SMEM does not guarantee the STRUCT_PAGE flag though: phys
+			 * objects clear it and it is dropped transiently during TTM
+			 * moves. drm_clflush_sg() walks the sgt with sg_page(), so
+			 * without struct pages those pointers are bogus; fall back to
+			 * wbinvd in that case (not the vmap path, which would recurse
+			 * on the already-locked source object).
+			 */
+			if (i915_gem_object_has_struct_page(dma_obj))
+				drm_clflush_sg(dma_obj->mm.pages);
+			else
+				wbinvd_on_all_cpus();
+		} else {
+			struct iosys_map map;
+
+			/*
+			 * A foreign sg_table is not guaranteed to be backed by
+			 * struct pages, so we cannot use drm_clflush_sg(). vmap
+			 * the buffer and flush the virtual range instead; x86
+			 * uses PIPT caches, so flushing one alias evicts the
+			 * lines for every alias of the same physical pages.
+			 *
+			 * We already hold the dma_resv lock via the imported
+			 * obj, so use the locked dma_buf_vmap() variant.
+			 */
+			if (!dma_buf_vmap(dma_buf, &map)) {
+				if (!map.is_iomem)
+					drm_clflush_virt_range(map.vaddr,
+							       obj->base.size);
+				else
+					wbinvd_on_all_cpus();
+				dma_buf_vunmap(dma_buf, &map);
+			} else {
+				wbinvd_on_all_cpus();
+			}
+		}
+	}
 
 	__i915_gem_object_set_pages(obj, sgt);
 
-- 
2.43.0