[PATCH v2] drm/i915/dmabuf: avoid global wbinvd on dma-buf import

[PATCH v2] drm/i915/dmabuf: avoid global wbinvd on dma-buf import

[Prabhakaran, Krishna]

From: Krishna Prabhakaran <krishna.prabhakaran@intel.com>

When i915 needs to make an imported dma-buf coherent for GPU access on
non-LLC platforms, or for objects that bypass LLC, it currently calls
wbinvd_on_all_cpus(). get_pages() runs whenever an imported buffer is
pinned, so this triggers a whole-cache write-back and invalidate,
broadcast by IPI to every CPU, on every execbuf submission involving an
imported dma-buf. That stalls the entire machine for milliseconds and
starves latency-sensitive work on unrelated cores (e.g. USB isochronous
audio serviced on the VMM's main thread).

Flush only the pages that actually need it instead:

 - If we imported one of our own dma-bufs, the backing object is
   struct-page backed once migrated to SMEM, so flush it directly with
   drm_clflush_sg(), exactly as we flush our other objects. This also
   avoids re-entering the exporter through dma_buf_vmap(), which would
   recurse into i915_gem_object_pin_map() on the source object we
   already hold locked (and fails the igt_dmabuf_import_same_driver
   selftest with -EBUSY).

 - For a foreign dma-buf the sg_table is not guaranteed to be backed by
   struct pages, and the importer has no way to tell, so drm_clflush_sg()
   cannot be used. vmap the buffer and flush that virtual range with
   drm_clflush_virt_range() instead: x86 uses PIPT caches, so flushing
   one virtual alias evicts the cache lines for every alias of the same
   physical pages. The dma_resv lock required by dma_buf_vmap() is
   already held here via the imported object.

Fall back to wbinvd only when the buffer cannot be vmapped or is backed
by I/O memory, where there is no CPU-side range to clflush.

Fixes: a035154da45d ("drm/i915/dmabuf: add paranoid flush-on-acquire")
Signed-off-by: Krishna Prabhakaran <krishna.prabhakaran@intel.com>
---
v2:
 - Flush our own imported dma-bufs directly with drm_clflush_sg() instead of
   dma_buf_vmap(), which re-entered i915_gem_object_pin_map() on the source
   object and failed igt_dmabuf_import_same_driver_smem with -EBUSY (reported
   by Intel CI on v1). Foreign dma-bufs still use dma_buf_vmap() +
   drm_clflush_virt_range(); wbinvd only as fallback.
 - Link to v1: https://patchwork.freedesktop.org/patch/744955/?series=171760

 drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 55 ++++++++++++++++++----
 1 file changed, 47 insertions(+), 8 deletions(-)

diff --git a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
index b43d34c7d641..c798a90f1c0f 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
@@ -10,6 +10,8 @@
 
 #include <asm/smp.h>
 
+#include <drm/drm_cache.h>
+
 #include "gem/i915_gem_dmabuf.h"
 #include "i915_drv.h"
 #include "i915_gem_object.h"
@@ -249,16 +251,53 @@ static int i915_gem_object_get_pages_dmabuf(struct drm_i915_gem_object *obj)
 	 * DG1 is special here since it still snoops transactions even with
 	 * CACHE_NONE. This is not the case with other HAS_SNOOP platforms. We
 	 * might need to revisit this as we add new discrete platforms.
-	 *
-	 * XXX: Consider doing a vmap flush or something, where possible.
-	 * Currently we just do a heavy handed wbinvd_on_all_cpus() here since
-	 * the underlying sg_table might not even point to struct pages, so we
-	 * can't just call drm_clflush_sg or similar, like we do elsewhere in
-	 * the driver.
 	 */
 	if (i915_gem_object_can_bypass_llc(obj) ||
-	    (!HAS_LLC(i915) && !IS_DG1(i915)))
-		wbinvd_on_all_cpus();
+	    (!HAS_LLC(i915) && !IS_DG1(i915))) {
+		struct dma_buf *dma_buf = obj->base.import_attach->dmabuf;
+
+		if (dma_buf->ops == &i915_dmabuf_ops) {
+			struct drm_i915_gem_object *dma_obj =
+				dma_buf_to_obj(dma_buf);
+
+			/*
+			 * We imported one of our own dma-bufs. The backing
+			 * object is struct-page backed once migrated to SMEM,
+			 * so flush it directly, the same way we flush our
+			 * other objects. This also avoids re-entering the
+			 * exporter through dma_buf_vmap(), which would recurse
+			 * into i915_gem_object_pin_map() on the source object
+			 * we already hold locked.
+			 */
+			if (i915_gem_object_has_struct_page(dma_obj))
+				drm_clflush_sg(dma_obj->mm.pages);
+			else
+				wbinvd_on_all_cpus();
+		} else {
+			struct iosys_map map;
+
+			/*
+			 * A foreign sg_table is not guaranteed to be backed by
+			 * struct pages, so we cannot use drm_clflush_sg(). vmap
+			 * the buffer and flush the virtual range instead; x86
+			 * uses PIPT caches, so flushing one alias evicts the
+			 * lines for every alias of the same physical pages.
+			 *
+			 * We already hold the dma_resv lock via the imported
+			 * obj, so use the locked dma_buf_vmap() variant.
+			 */
+			if (!dma_buf_vmap(dma_buf, &map)) {
+				if (!map.is_iomem)
+					drm_clflush_virt_range(map.vaddr,
+							       obj->base.size);
+				else
+					wbinvd_on_all_cpus();
+				dma_buf_vunmap(dma_buf, &map);
+			} else {
+				wbinvd_on_all_cpus();
+			}
+		}
+	}
 
 	__i915_gem_object_set_pages(obj, sgt);
 

base-commit: 682ea2d28d18bb06f9fc663cb5ab7e80dc0e606a
-- 
2.43.0

✓ i915.CI.BAT: success for drm/i915/dmabuf: avoid global wbinvd on dma-buf import

[Patchwork]

[-- Attachment #1: Type: text/plain, Size: 3310 bytes --]

== Series Details ==

Series: drm/i915/dmabuf: avoid global wbinvd on dma-buf import
URL   : https://patchwork.freedesktop.org/series/172276/
State : success

== Summary ==

CI Bug Log - changes from CI_DRM_19004 -> Patchwork_172276v1
====================================================

Summary
-------

  **SUCCESS**

  No regressions found.

  External URL: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_172276v1/index.html

Participating hosts (35 -> 32)
------------------------------

  Missing    (3): bat-dg2-13 fi-snb-2520m bat-adlp-6 

Known issues
------------

  Here are the changes found in Patchwork_172276v1 that come from known issues:

### IGT changes ###

#### Possible fixes ####

  * igt@i915_selftest@live@ring_submission:
    - fi-cfl-8109u:       [DMESG-WARN][1] ([i915#13735]) -> [PASS][2] +80 other tests pass
   [1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_19004/fi-cfl-8109u/igt@i915_selftest@live@ring_submission.html
   [2]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_172276v1/fi-cfl-8109u/igt@i915_selftest@live@ring_submission.html

  * igt@i915_selftest@live@sanitycheck:
    - fi-kbl-7567u:       [DMESG-WARN][3] ([i915#13735]) -> [PASS][4] +79 other tests pass
   [3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_19004/fi-kbl-7567u/igt@i915_selftest@live@sanitycheck.html
   [4]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_172276v1/fi-kbl-7567u/igt@i915_selftest@live@sanitycheck.html

  * igt@kms_busy@basic@flip:
    - fi-kbl-7567u:       [DMESG-WARN][5] ([i915#13735] / [i915#180]) -> [PASS][6]
   [5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_19004/fi-kbl-7567u/igt@kms_busy@basic@flip.html
   [6]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_172276v1/fi-kbl-7567u/igt@kms_busy@basic@flip.html

  * igt@kms_pipe_crc_basic@read-crc:
    - fi-cfl-8109u:       [DMESG-WARN][7] ([i915#13735] / [i915#15673]) -> [PASS][8] +49 other tests pass
   [7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_19004/fi-cfl-8109u/igt@kms_pipe_crc_basic@read-crc.html
   [8]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_172276v1/fi-cfl-8109u/igt@kms_pipe_crc_basic@read-crc.html

  * igt@kms_pm_rpm@basic-pci-d3-state:
    - fi-kbl-7567u:       [DMESG-WARN][9] ([i915#13735] / [i915#15673] / [i915#180]) -> [PASS][10] +52 other tests pass
   [9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_19004/fi-kbl-7567u/igt@kms_pm_rpm@basic-pci-d3-state.html
   [10]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_172276v1/fi-kbl-7567u/igt@kms_pm_rpm@basic-pci-d3-state.html

  
  [i915#13735]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/13735
  [i915#15673]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/15673
  [i915#180]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/180


Build changes
-------------

  * Linux: CI_DRM_19004 -> Patchwork_172276v1

  CI-20190529: 20190529
  CI_DRM_19004: ceef0899403ceea5643d80a02bfdf9587f3845b8 @ git://anongit.freedesktop.org/gfx-ci/linux
  IGT_9057: 8f7d5198bdb1c5af7c97b967a0bcbe77b653e8a0 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
  Patchwork_172276v1: ceef0899403ceea5643d80a02bfdf9587f3845b8 @ git://anongit.freedesktop.org/gfx-ci/linux

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_172276v1/index.html

[-- Attachment #2: Type: text/html, Size: 4466 bytes --]