[RFC 0/7] drm/virtio: Import scanout buffers from other devices

[RFC 0/7] drm/virtio: Import scanout buffers from other devices

[Vivek Kasireddy]

Having virtio-gpu import scanout buffers (via prime) from other
devices means that we'd be adding a head to headless GPUs assigned
to a Guest VM or additional heads to regular GPU devices that are
passthrough'd to the Guest. In these cases, the Guest compositor
can render into the scanout buffer using a primary GPU and has the
secondary GPU (virtio-gpu) import it for display purposes.

The main advantage with this is that the imported scanout buffer can
either be displayed locally on the Host (e.g, using Qemu + GTK UI)
or encoded and streamed to a remote client (e.g, Qemu + Spice UI).
Note that since Qemu uses udmabuf driver, there would be no copies
made of the scanout buffer as it is displayed. This should be
possible even when it might reside in device memory such has VRAM.

The specific use-case that can be supported with this series is when
running Weston or other guest compositors with "additional-devices"
feature (./weston --drm-device=card1 --additional-devices=card0).
More info about this feature can be found at:
https://gitlab.freedesktop.org/wayland/weston/-/merge_requests/736

In the above scenario, card1 could be a dGPU or an iGPU and card0
would be virtio-gpu in KMS only mode. However, the case where this
patch series could be particularly useful is when card1 is a GPU VF
that needs to share its scanout buffer (in a zero-copy way) with the
GPU PF on the Host. Or, it can also be useful when the scanout buffer
needs to be shared between any two GPU devices (assuming one of them
is assigned to a Guest VM) as long as they are P2P DMA compatible.

As part of the import, the virtio-gpu driver shares the dma
addresses and lengths with Qemu which then determines whether the
memory region they belong to is owned by a PCI device or whether it
is part of the Guest's system ram. If it is the former, it identifies
the devid (or bdf) and bar and provides this info (along with offsets
and sizes) to the udmabuf driver. In the latter case, instead of the
the devid and bar it provides the memfd. The udmabuf driver then
creates a dmabuf using this info that Qemu shares with Spice for
encode via Gstreamer.

Note that the virtio-gpu driver registers a move_notify() callback
to track location changes associated with the scanout buffer and
sends attach/detach backing cmds to Qemu when appropriate. And,
synchronization (that is, ensuring that Guest and Host are not
using the scanout buffer at the same time) is ensured by pinning/
unpinning the dmabuf as part of plane update and using a fence
in resource_flush cmd. 

This series is available at:
https://gitlab.freedesktop.org/Vivek/drm-tip/-/commits/virtgpu_import_rfc

along with additional patches for Qemu and Spice here:
https://gitlab.freedesktop.org/Vivek/qemu/-/commits/virtgpu_dmabuf_pcidev
https://gitlab.freedesktop.org/Vivek/spice/-/commits/encode_dmabuf_v4 

Patchset overview:

Patch 1:   Implement VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING cmd
Patch 2-3: Helpers to initalize, import, free imported object
Patch 4-5: Import and use buffers from other devices for scanout
Patch 6-7: Have udmabuf driver create dmabuf from PCI bars for P2P DMA

This series is tested using the following method:
- Run Qemu with the following relevant options:
  qemu-system-x86_64 -m 4096m ....
  -device vfio-pci,host=0000:03:00.0
  -device virtio-vga,max_outputs=1,blob=true,xres=1920,yres=1080
  -spice port=3001,gl=on,disable-ticketing=on,preferred-codec=gstreamer:h264
  -object memory-backend-memfd,id=mem1,size=4096M
  -machine memory-backend=mem1 ...
- Run upstream Weston with the following options in the Guest VM:
  ./weston --drm-device=card1 --additional-devices=card0

where card1 is a DG2 dGPU (passthrough'd and using xe driver in Guest VM),
card0 is virtio-gpu and the Host is using a RPL iGPU.

Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: Dongwon Kim <dongwon.kim@intel.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Cc: Rob Clark <robdclark@chromium.org>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Oded Gabbay <ogabbay@kernel.org>
Cc: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Michael Tretter <m.tretter@pengutronix.de>

Vivek Kasireddy (7):
  drm/virtio: Implement VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING cmd
  drm/virtio: Add a helper to map and note the dma addrs and lengths
  drm/virtio: Add helpers to initialize and free the imported object
  drm/virtio: Import prime buffers from other devices as guest blobs
  drm/virtio: Ensure that bo's backing store is valid while updating
    plane
  udmabuf/uapi: Add new ioctl to create a dmabuf from PCI bar regions
  udmabuf: Implement UDMABUF_CREATE_LIST_FOR_PCIDEV ioctl

 drivers/dma-buf/udmabuf.c              | 122 ++++++++++++++++--
 drivers/gpu/drm/virtio/virtgpu_drv.h   |   8 ++
 drivers/gpu/drm/virtio/virtgpu_plane.c |  56 ++++++++-
 drivers/gpu/drm/virtio/virtgpu_prime.c | 167 ++++++++++++++++++++++++-
 drivers/gpu/drm/virtio/virtgpu_vq.c    |  15 +++
 include/uapi/linux/udmabuf.h           |  11 +-
 6 files changed, 368 insertions(+), 11 deletions(-)

-- 
2.43.0

[RFC 1/7] drm/virtio: Implement VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING cmd

[Vivek Kasireddy]

This cmd is useful to let the VMM (i.e, Qemu) know that the backing
store associated with a resource is no longer valid, so that the VMM
can perform any cleanup or unmap operations.

Cc: Gerd Hoffmann <kraxel@redhat.com>
Signed-off-by: Vivek Kasireddy <vivek.kasireddy@intel.com>
---
 drivers/gpu/drm/virtio/virtgpu_drv.h |  2 ++
 drivers/gpu/drm/virtio/virtgpu_vq.c  | 15 +++++++++++++++
 2 files changed, 17 insertions(+)

diff --git a/drivers/gpu/drm/virtio/virtgpu_drv.h b/drivers/gpu/drm/virtio/virtgpu_drv.h
index bb7d86a0c6a1..7347835e4fbe 100644
--- a/drivers/gpu/drm/virtio/virtgpu_drv.h
+++ b/drivers/gpu/drm/virtio/virtgpu_drv.h
@@ -349,6 +349,8 @@ void virtio_gpu_object_attach(struct virtio_gpu_device *vgdev,
 			      struct virtio_gpu_object *obj,
 			      struct virtio_gpu_mem_entry *ents,
 			      unsigned int nents);
+void virtio_gpu_cmd_resource_detach_backing(struct virtio_gpu_device *vgdev,
+			      uint32_t resource_id);
 void virtio_gpu_cursor_ping(struct virtio_gpu_device *vgdev,
 			    struct virtio_gpu_output *output);
 int virtio_gpu_cmd_get_display_info(struct virtio_gpu_device *vgdev);
diff --git a/drivers/gpu/drm/virtio/virtgpu_vq.c b/drivers/gpu/drm/virtio/virtgpu_vq.c
index b1a00c0c25a7..17e2e7ab231a 100644
--- a/drivers/gpu/drm/virtio/virtgpu_vq.c
+++ b/drivers/gpu/drm/virtio/virtgpu_vq.c
@@ -645,6 +645,21 @@ virtio_gpu_cmd_resource_attach_backing(struct virtio_gpu_device *vgdev,
 	virtio_gpu_queue_fenced_ctrl_buffer(vgdev, vbuf, fence);
 }
 
+void virtio_gpu_cmd_resource_detach_backing(struct virtio_gpu_device *vgdev,
+					    uint32_t resource_id)
+{
+	struct virtio_gpu_resource_detach_backing *cmd_p;
+	struct virtio_gpu_vbuffer *vbuf;
+
+	cmd_p = virtio_gpu_alloc_cmd(vgdev, &vbuf, sizeof(*cmd_p));
+	memset(cmd_p, 0, sizeof(*cmd_p));
+
+	cmd_p->hdr.type = cpu_to_le32(VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING);
+	cmd_p->resource_id = cpu_to_le32(resource_id);
+
+	virtio_gpu_queue_ctrl_buffer(vgdev, vbuf);
+}
+
 static void virtio_gpu_cmd_get_display_info_cb(struct virtio_gpu_device *vgdev,
 					       struct virtio_gpu_vbuffer *vbuf)
 {
-- 
2.43.0

[RFC 2/7] drm/virtio: Add a helper to map and note the dma addrs and lengths

[Vivek Kasireddy]

This helper would be used when first initializing the object as
part of import and also when updating the plane where we need to
ensure that the imported object's backing is valid.

Cc: Gerd Hoffmann <kraxel@redhat.com>
Signed-off-by: Vivek Kasireddy <vivek.kasireddy@intel.com>
---
 drivers/gpu/drm/virtio/virtgpu_drv.h   |  6 ++++
 drivers/gpu/drm/virtio/virtgpu_prime.c | 44 ++++++++++++++++++++++++++
 2 files changed, 50 insertions(+)

diff --git a/drivers/gpu/drm/virtio/virtgpu_drv.h b/drivers/gpu/drm/virtio/virtgpu_drv.h
index 7347835e4fbe..ca4cb166b509 100644
--- a/drivers/gpu/drm/virtio/virtgpu_drv.h
+++ b/drivers/gpu/drm/virtio/virtgpu_drv.h
@@ -89,9 +89,11 @@ struct virtio_gpu_object_params {
 
 struct virtio_gpu_object {
 	struct drm_gem_shmem_object base;
+	struct sg_table *sgt;
 	uint32_t hw_res_handle;
 	bool dumb;
 	bool created;
+	bool has_backing;
 	bool host3d_blob, guest_blob;
 	uint32_t blob_mem, blob_flags;
 
@@ -470,6 +472,10 @@ struct drm_gem_object *virtgpu_gem_prime_import(struct drm_device *dev,
 struct drm_gem_object *virtgpu_gem_prime_import_sg_table(
 	struct drm_device *dev, struct dma_buf_attachment *attach,
 	struct sg_table *sgt);
+long virtgpu_dma_buf_import_sgt(struct virtio_gpu_mem_entry **ents,
+				unsigned int *nents,
+				struct virtio_gpu_object *bo,
+				struct dma_buf_attachment *attach);
 
 /* virtgpu_debugfs.c */
 void virtio_gpu_debugfs_init(struct drm_minor *minor);
diff --git a/drivers/gpu/drm/virtio/virtgpu_prime.c b/drivers/gpu/drm/virtio/virtgpu_prime.c
index 44425f20d91a..2a90df39c5de 100644
--- a/drivers/gpu/drm/virtio/virtgpu_prime.c
+++ b/drivers/gpu/drm/virtio/virtgpu_prime.c
@@ -27,6 +27,8 @@
 
 #include "virtgpu_drv.h"
 
+MODULE_IMPORT_NS(DMA_BUF);
+
 static int virtgpu_virtio_get_uuid(struct dma_buf *buf,
 				   uuid_t *uuid)
 {
@@ -142,6 +144,48 @@ struct dma_buf *virtgpu_gem_prime_export(struct drm_gem_object *obj,
 	return buf;
 }
 
+long virtgpu_dma_buf_import_sgt(struct virtio_gpu_mem_entry **ents,
+				unsigned int *nents,
+				struct virtio_gpu_object *bo,
+				struct dma_buf_attachment *attach)
+{
+	struct scatterlist *sl;
+	struct sg_table *sgt;
+	long i, ret;
+
+	dma_resv_assert_held(attach->dmabuf->resv);
+
+	ret = dma_resv_wait_timeout(attach->dmabuf->resv,
+				    DMA_RESV_USAGE_KERNEL,
+				    false, MAX_SCHEDULE_TIMEOUT);
+	if (ret < 0)
+		return ret;
+
+	sgt = dma_buf_map_attachment(attach, DMA_BIDIRECTIONAL);
+	if (IS_ERR(sgt))
+		return PTR_ERR(sgt);
+
+	*ents = kvmalloc_array(sgt->nents,
+			       sizeof(struct virtio_gpu_mem_entry),
+			       GFP_KERNEL);
+	if (!(*ents)) {
+		dma_buf_unmap_attachment(attach, sgt, DMA_BIDIRECTIONAL);
+		return -ENOMEM;
+	}
+
+	*nents = sgt->nents;
+	for_each_sgtable_dma_sg(sgt, sl, i) {
+		(*ents)[i].addr = cpu_to_le64(sg_dma_address(sl));
+		(*ents)[i].length = cpu_to_le32(sg_dma_len(sl));
+		(*ents)[i].padding = 0;
+	}
+
+	bo->sgt = sgt;
+	bo->has_backing = true;
+
+	return 0;
+}
+
 struct drm_gem_object *virtgpu_gem_prime_import(struct drm_device *dev,
 						struct dma_buf *buf)
 {
-- 
2.43.0

[RFC 3/7] drm/virtio: Add helpers to initialize and free the imported object

[Vivek Kasireddy]

The imported object can be considered a guest blob resource;
therefore, we use create_blob cmd while creating it. These helpers
are used in the next patch which does the actual import.

Cc: Gerd Hoffmann <kraxel@redhat.com>
Signed-off-by: Vivek Kasireddy <vivek.kasireddy@intel.com>
---
 drivers/gpu/drm/virtio/virtgpu_prime.c | 69 ++++++++++++++++++++++++++
 1 file changed, 69 insertions(+)

diff --git a/drivers/gpu/drm/virtio/virtgpu_prime.c b/drivers/gpu/drm/virtio/virtgpu_prime.c
index 2a90df39c5de..1e87dbc9a897 100644
--- a/drivers/gpu/drm/virtio/virtgpu_prime.c
+++ b/drivers/gpu/drm/virtio/virtgpu_prime.c
@@ -186,6 +186,75 @@ long virtgpu_dma_buf_import_sgt(struct virtio_gpu_mem_entry **ents,
 	return 0;
 }
 
+static void virtgpu_dma_buf_free_obj(struct drm_gem_object *obj)
+{
+	struct virtio_gpu_object *bo = gem_to_virtio_gpu_obj(obj);
+	struct virtio_gpu_device *vgdev = obj->dev->dev_private;
+	struct dma_buf_attachment *attach = obj->import_attach;
+
+	if (bo->created) {
+		virtio_gpu_cmd_unref_resource(vgdev, bo);
+		virtio_gpu_notify(vgdev);
+	}
+
+	if (attach) {
+		dma_buf_detach(attach->dmabuf, attach);
+		dma_buf_put(attach->dmabuf);
+	}
+
+	drm_gem_object_release(&bo->base.base);
+	kfree(bo);
+}
+
+static int virtgpu_dma_buf_init_obj(struct drm_device *dev,
+				    struct virtio_gpu_object *bo,
+				    struct dma_buf_attachment *attach)
+{
+	struct virtio_gpu_device *vgdev = dev->dev_private;
+	struct virtio_gpu_object_params params = { 0 };
+	struct dma_resv *resv = attach->dmabuf->resv;
+	struct virtio_gpu_mem_entry *ents = NULL;
+	unsigned int nents;
+	int ret;
+
+	ret = virtio_gpu_resource_id_get(vgdev, &bo->hw_res_handle);
+	if (ret) {
+		virtgpu_dma_buf_free_obj(&bo->base.base);
+		return ret;
+	}
+
+	dma_resv_lock(resv, NULL);
+
+	ret = dma_buf_pin(attach);
+	if (ret)
+		goto err_pin;
+
+	ret = virtgpu_dma_buf_import_sgt(&ents, &nents, bo, attach);
+	if (ret)
+		goto err_import;
+
+	bo->guest_blob = true;
+	params.blob = true;
+	params.blob_mem = VIRTGPU_BLOB_MEM_GUEST;
+	params.blob_flags = VIRTGPU_BLOB_FLAG_USE_SHAREABLE;
+	params.size = attach->dmabuf->size;
+
+	virtio_gpu_cmd_resource_create_blob(vgdev, bo, &params,
+					    ents, nents);
+	dma_buf_unpin(attach);
+	dma_resv_unlock(resv);
+
+	return 0;
+
+err_import:
+	dma_buf_unpin(attach);
+err_pin:
+	dma_resv_unlock(resv);
+	ida_free(&vgdev->resource_ida, bo->hw_res_handle - 1);
+	virtgpu_dma_buf_free_obj(&bo->base.base);
+	return ret;
+}
+
 struct drm_gem_object *virtgpu_gem_prime_import(struct drm_device *dev,
 						struct dma_buf *buf)
 {
-- 
2.43.0

[RFC 4/7] drm/virtio: Import prime buffers from other devices as guest blobs

[Vivek Kasireddy]

By importing scanout buffers from other devices, we should be able
to use the virtio-gpu driver in KMS only mode. Note that we attach
dynamically and register a move_notify() callback so that we can
let the VMM know of any location changes associated with the backing
store of the imported object by sending detach_backing cmd.

Cc: Gerd Hoffmann <kraxel@redhat.com>
Signed-off-by: Vivek Kasireddy <vivek.kasireddy@intel.com>
---
 drivers/gpu/drm/virtio/virtgpu_prime.c | 54 +++++++++++++++++++++++++-
 1 file changed, 53 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/virtio/virtgpu_prime.c b/drivers/gpu/drm/virtio/virtgpu_prime.c
index 1e87dbc9a897..c65dacc1b2b5 100644
--- a/drivers/gpu/drm/virtio/virtgpu_prime.c
+++ b/drivers/gpu/drm/virtio/virtgpu_prime.c
@@ -255,10 +255,36 @@ static int virtgpu_dma_buf_init_obj(struct drm_device *dev,
 	return ret;
 }
 
+static const struct drm_gem_object_funcs virtgpu_gem_dma_buf_funcs = {
+	.free = virtgpu_dma_buf_free_obj,
+};
+
+static void virtgpu_dma_buf_move_notify(struct dma_buf_attachment *attach)
+{
+	struct drm_gem_object *obj = attach->importer_priv;
+	struct virtio_gpu_device *vgdev = obj->dev->dev_private;
+	struct virtio_gpu_object *bo = gem_to_virtio_gpu_obj(obj);
+
+	if (bo->created) {
+		virtio_gpu_cmd_resource_detach_backing(vgdev,
+						       bo->hw_res_handle);
+		bo->has_backing = false;
+	}
+}
+
+static const struct dma_buf_attach_ops virtgpu_dma_buf_attach_ops = {
+	.allow_peer2peer = true,
+	.move_notify = virtgpu_dma_buf_move_notify
+};
+
 struct drm_gem_object *virtgpu_gem_prime_import(struct drm_device *dev,
 						struct dma_buf *buf)
 {
+	struct virtio_gpu_device *vgdev = dev->dev_private;
+	struct dma_buf_attachment *attach;
+	struct virtio_gpu_object *bo;
 	struct drm_gem_object *obj;
+	int ret;
 
 	if (buf->ops == &virtgpu_dmabuf_ops.ops) {
 		obj = buf->priv;
@@ -272,7 +298,32 @@ struct drm_gem_object *virtgpu_gem_prime_import(struct drm_device *dev,
 		}
 	}
 
-	return drm_gem_prime_import(dev, buf);
+	if (!vgdev->has_resource_blob || vgdev->has_virgl_3d)
+		return drm_gem_prime_import(dev, buf);
+
+	bo = kzalloc(sizeof(*bo), GFP_KERNEL);
+	if (!bo)
+		return ERR_PTR(-ENOMEM);
+
+	obj = &bo->base.base;
+	obj->funcs = &virtgpu_gem_dma_buf_funcs;
+	drm_gem_private_object_init(dev, obj, buf->size);
+
+	attach = dma_buf_dynamic_attach(buf, dev->dev,
+					&virtgpu_dma_buf_attach_ops, obj);
+	if (IS_ERR(attach)) {
+		kfree(bo);
+		return ERR_CAST(attach);
+	}
+
+	obj->import_attach = attach;
+	get_dma_buf(buf);
+
+	ret = virtgpu_dma_buf_init_obj(dev, bo, attach);
+	if (ret < 0)
+		return ERR_PTR(ret);
+
+	return obj;
 }
 
 struct drm_gem_object *virtgpu_gem_prime_import_sg_table(
@@ -281,3 +332,4 @@ struct drm_gem_object *virtgpu_gem_prime_import_sg_table(
 {
 	return ERR_PTR(-ENODEV);
 }
+
-- 
2.43.0

[RFC 5/7] drm/virtio: Ensure that bo's backing store is valid while updating plane

[Vivek Kasireddy]

To make sure that the imported bo's backing store is valid, we first
pin the associated dmabuf, import the sgt if need be and then unpin
it after the update is complete. Note that we pin/unpin the dmabuf
even when the backing store is valid to ensure that it does not move
when the host update (resource_flush) is in progress.

Cc: Gerd Hoffmann <kraxel@redhat.com>
Signed-off-by: Vivek Kasireddy <vivek.kasireddy@intel.com>
---
 drivers/gpu/drm/virtio/virtgpu_plane.c | 56 +++++++++++++++++++++++++-
 1 file changed, 55 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/virtio/virtgpu_plane.c b/drivers/gpu/drm/virtio/virtgpu_plane.c
index a72a2dbda031..3ccf88f9addc 100644
--- a/drivers/gpu/drm/virtio/virtgpu_plane.c
+++ b/drivers/gpu/drm/virtio/virtgpu_plane.c
@@ -26,6 +26,7 @@
 #include <drm/drm_atomic_helper.h>
 #include <drm/drm_damage_helper.h>
 #include <drm/drm_fourcc.h>
+#include <linux/virtio_dma_buf.h>
 
 #include "virtgpu_drv.h"
 
@@ -131,6 +132,45 @@ static void virtio_gpu_update_dumb_bo(struct virtio_gpu_device *vgdev,
 					   objs, NULL);
 }
 
+static bool virtio_gpu_update_dmabuf_bo(struct virtio_gpu_device *vgdev,
+					struct drm_gem_object *obj)
+{
+	struct virtio_gpu_object *bo = gem_to_virtio_gpu_obj(obj);
+	struct dma_buf_attachment *attach = obj->import_attach;
+	struct dma_resv *resv = attach->dmabuf->resv;
+	struct virtio_gpu_mem_entry *ents = NULL;
+	unsigned int nents;
+	int ret;
+
+	dma_resv_lock(resv, NULL);
+
+	ret = dma_buf_pin(attach);
+	if (ret) {
+		dma_resv_unlock(resv);
+		return false;
+	}
+
+	if (!bo->has_backing) {
+		if (bo->sgt)
+			dma_buf_unmap_attachment(attach,
+						 bo->sgt,
+						 DMA_BIDIRECTIONAL);
+
+		ret = virtgpu_dma_buf_import_sgt(&ents, &nents,
+						 bo, attach);
+		if (ret)
+			goto err_import;
+
+		virtio_gpu_object_attach(vgdev, bo, ents, nents);
+	}
+	return true;
+
+err_import:
+	dma_buf_unpin(attach);
+	dma_resv_unlock(resv);
+	return false;
+}
+
 static void virtio_gpu_resource_flush(struct drm_plane *plane,
 				      uint32_t x, uint32_t y,
 				      uint32_t width, uint32_t height)
@@ -174,7 +214,9 @@ static void virtio_gpu_primary_plane_update(struct drm_plane *plane,
 	struct virtio_gpu_device *vgdev = dev->dev_private;
 	struct virtio_gpu_output *output = NULL;
 	struct virtio_gpu_object *bo;
+	struct drm_gem_object *obj;
 	struct drm_rect rect;
+	bool updated = false;
 
 	if (plane->state->crtc)
 		output = drm_crtc_to_virtio_gpu_output(plane->state->crtc);
@@ -196,10 +238,17 @@ static void virtio_gpu_primary_plane_update(struct drm_plane *plane,
 	if (!drm_atomic_helper_damage_merged(old_state, plane->state, &rect))
 		return;
 
-	bo = gem_to_virtio_gpu_obj(plane->state->fb->obj[0]);
+	obj = plane->state->fb->obj[0];
+	bo = gem_to_virtio_gpu_obj(obj);
 	if (bo->dumb)
 		virtio_gpu_update_dumb_bo(vgdev, plane->state, &rect);
 
+	if (obj->import_attach) {
+		updated = virtio_gpu_update_dmabuf_bo(vgdev, obj);
+		if (!updated)
+			return;
+	}
+
 	if (plane->state->fb != old_state->fb ||
 	    plane->state->src_w != old_state->src_w ||
 	    plane->state->src_h != old_state->src_h ||
@@ -239,6 +288,11 @@ static void virtio_gpu_primary_plane_update(struct drm_plane *plane,
 				  rect.y1,
 				  rect.x2 - rect.x1,
 				  rect.y2 - rect.y1);
+
+	if (obj->import_attach && updated) {
+		dma_buf_unpin(obj->import_attach);
+		dma_resv_unlock(obj->import_attach->dmabuf->resv);
+	}
 }
 
 static int virtio_gpu_plane_prepare_fb(struct drm_plane *plane,
-- 
2.43.0

[RFC 6/7] udmabuf/uapi: Add new ioctl to create a dmabuf from PCI bar regions

[Vivek Kasireddy]

This new ioctl can be used by a VMM such as Qemu or other userspace
applications to create a dmabuf from a PCI device's memory regions.
The PCI device's id that the userspace app is required to provide
needs to be encoded in the format specified by the following macro
(defined in include/linux/pci.h):
define PCI_DEVID(bus, devfn)	((((u16)(bus)) << 8) | (devfn))

where devfn is defined (in include/uapi/linux/pci.h) as
define PCI_DEVFN(slot, func)	((((slot) & 0x1f) << 3) | ((func) & 0x07))

In addition to the devid, the userspace needs to include the
offsets and sizes and also the bar number as part of this request.

Cc: Gerd Hoffmann <kraxel@redhat.com>
Signed-off-by: Vivek Kasireddy <vivek.kasireddy@intel.com>
---
 include/uapi/linux/udmabuf.h | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/include/uapi/linux/udmabuf.h b/include/uapi/linux/udmabuf.h
index 46b6532ed855..16fe41fdc4b9 100644
--- a/include/uapi/linux/udmabuf.h
+++ b/include/uapi/linux/udmabuf.h
@@ -15,7 +15,15 @@ struct udmabuf_create {
 };
 
 struct udmabuf_create_item {
-	__u32 memfd;
+	union {
+		struct {
+			__u32 memfd;
+		};
+		struct {
+			__u16 devid;
+			__u16 bar;
+		};
+	};
 	__u32 __pad;
 	__u64 offset;
 	__u64 size;
@@ -29,5 +37,6 @@ struct udmabuf_create_list {
 
 #define UDMABUF_CREATE       _IOW('u', 0x42, struct udmabuf_create)
 #define UDMABUF_CREATE_LIST  _IOW('u', 0x43, struct udmabuf_create_list)
+#define UDMABUF_CREATE_LIST_FOR_PCIDEV  _IOW('u', 0x44, struct udmabuf_create_list)
 
 #endif /* _UAPI_LINUX_UDMABUF_H */
-- 
2.43.0

[RFC 7/7] udmabuf: Implement UDMABUF_CREATE_LIST_FOR_PCIDEV ioctl

[Vivek Kasireddy]

By implementing this request, the udmabuf driver would be able
to support creating a dmabuf from a PCI device's bar region. This
would facilitate P2P DMA operations between any two PCI devices
as long as they are compatible.

Based on the information (devid, bar) provided by the VMM, once
the PCI device known as the provider is identified, we create a
page pool associated with the requested bar region by calling
pci_p2pdma_add_resource(). We then populate the ubuf->pages[]
array with the pages from the pool that would eventually be
included in a sgt which would be shared with the importers.

Cc: Gerd Hoffmann <kraxel@redhat.com>
Signed-off-by: Vivek Kasireddy <vivek.kasireddy@intel.com>
---
 drivers/dma-buf/udmabuf.c | 122 +++++++++++++++++++++++++++++++++++---
 1 file changed, 114 insertions(+), 8 deletions(-)

diff --git a/drivers/dma-buf/udmabuf.c b/drivers/dma-buf/udmabuf.c
index 274defd3fa3e..7355451ed337 100644
--- a/drivers/dma-buf/udmabuf.c
+++ b/drivers/dma-buf/udmabuf.c
@@ -9,6 +9,7 @@
 #include <linux/memfd.h>
 #include <linux/miscdevice.h>
 #include <linux/module.h>
+#include <linux/pci-p2pdma.h>
 #include <linux/shmem_fs.h>
 #include <linux/hugetlb.h>
 #include <linux/slab.h>
@@ -27,6 +28,7 @@ MODULE_PARM_DESC(size_limit_mb, "Max size of a dmabuf, in megabytes. Default is
 struct udmabuf {
 	pgoff_t pagecount;
 	struct page **pages;
+	struct pci_dev *pdev;
 	struct sg_table *sg;
 	struct miscdevice *device;
 	pgoff_t *offsets;
@@ -129,9 +131,28 @@ static void put_sg_table(struct device *dev, struct sg_table *sg,
 	kfree(sg);
 }
 
+static int check_p2p_support(struct dma_buf_attachment *attach)
+{
+	struct udmabuf *ubuf = attach->dmabuf->priv;
+	struct pci_dev *provider = ubuf->pdev;
+	struct device *client = attach->dev;
+	int ret = -1;
+
+	if (!provider)
+		return 0;
+
+	if (attach->peer2peer)
+		ret = pci_p2pdma_distance(provider, client, true);
+
+	return ret < 0 ? ret : 0;
+}
+
 static struct sg_table *map_udmabuf(struct dma_buf_attachment *at,
 				    enum dma_data_direction direction)
 {
+	if (check_p2p_support(at) < 0)
+		return ERR_PTR(-EOPNOTSUPP);
+
 	return get_sg_table(at->dev, at->dmabuf, direction);
 }
 
@@ -151,8 +172,15 @@ static void release_udmabuf(struct dma_buf *buf)
 	if (ubuf->sg)
 		put_sg_table(dev, ubuf->sg, DMA_BIDIRECTIONAL);
 
-	for (pg = 0; pg < ubuf->pagecount; pg++)
-		put_page(ubuf->pages[pg]);
+	for (pg = 0; pg < ubuf->pagecount; pg++) {
+		if (ubuf->pdev)
+			pci_free_p2pmem(ubuf->pdev,
+					page_to_virt(ubuf->pages[pg]),
+					PAGE_SIZE);
+		else
+			put_page(ubuf->pages[pg]);
+	}
+
 	kfree(ubuf->offsets);
 	kfree(ubuf->pages);
 	kfree(ubuf);
@@ -269,9 +297,74 @@ static int handle_shmem_pages(struct udmabuf *ubuf, struct file *memfd,
 	return 0;
 }
 
+static int handle_pcidev_pages(struct udmabuf *ubuf,
+			       struct udmabuf_create_list *head,
+			       struct udmabuf_create_item *list)
+{
+	struct pci_dev *pdev = NULL;
+	resource_size_t bar_size;
+	pgoff_t pgbuf = 0;
+	struct page *page;
+	int i, ret;
+	size_t size;
+	void *addr;
+
+	for (i = 0; i < head->count; i++) {
+		if (!ubuf->pdev) {
+			pdev = pci_get_domain_bus_and_slot(0,
+						PCI_BUS_NUM(list[i].devid),
+						list[i].devid & 0xff);
+			if (!pdev) {
+				ret = -ENODEV;
+				goto err;
+			}
+
+			ubuf->pdev = pdev;
+		}
+
+		bar_size = pci_resource_len(pdev, list[i].bar);
+		if (list[i].offset > bar_size ||
+		    list[i].offset + list[i].size > bar_size) {
+			ret = -EINVAL;
+			goto err;
+		}
+
+		ret = pci_p2pdma_add_resource(pdev,
+					      list[i].bar,
+					      list[i].size,
+					      list[i].offset);
+		if (ret)
+			goto err;
+
+		addr = pci_alloc_p2pmem(pdev, list[i].size);
+		if (!addr) {
+			ret = -EINVAL;
+			goto err;
+		}
+
+		size = 0;
+		while (size < list[i].size) {
+			page = virt_to_page((unsigned long)addr + size);
+			ubuf->pages[pgbuf++] = page;
+
+			size += PAGE_SIZE;
+		}
+	}
+
+err:
+	while (pgbuf > 0 && ubuf->pages[--pgbuf])
+		pci_free_p2pmem(pdev,
+				page_to_virt(ubuf->pages[pgbuf]),
+				PAGE_SIZE);
+	if (pdev)
+		pci_dev_put(pdev);
+	return ret;
+}
+
 static long udmabuf_create(struct miscdevice *device,
 			   struct udmabuf_create_list *head,
-			   struct udmabuf_create_item *list)
+			   struct udmabuf_create_item *list,
+			   bool for_pcidev)
 {
 	DEFINE_DMA_BUF_EXPORT_INFO(exp_info);
 	struct file *memfd = NULL;
@@ -312,6 +405,14 @@ static long udmabuf_create(struct miscdevice *device,
 		goto err;
 	}
 
+	if (for_pcidev) {
+		ret = handle_pcidev_pages(ubuf, head, list);
+		if (ret)
+			goto err;
+
+		goto create_dmabuf;
+	}
+
 	pgbuf = 0;
 	for (i = 0; i < head->count; i++) {
 		ret = -EBADFD;
@@ -344,6 +445,7 @@ static long udmabuf_create(struct miscdevice *device,
 		memfd = NULL;
 	}
 
+create_dmabuf:
 	exp_info.ops  = &udmabuf_ops;
 	exp_info.size = ubuf->pagecount << PAGE_SHIFT;
 	exp_info.priv = ubuf;
@@ -362,7 +464,7 @@ static long udmabuf_create(struct miscdevice *device,
 	return dma_buf_fd(buf, flags);
 
 err:
-	while (pgbuf > 0)
+	while (pgbuf > 0 && !ubuf->pdev)
 		put_page(ubuf->pages[--pgbuf]);
 	if (memfd)
 		fput(memfd);
@@ -388,10 +490,11 @@ static long udmabuf_ioctl_create(struct file *filp, unsigned long arg)
 	list.offset = create.offset;
 	list.size   = create.size;
 
-	return udmabuf_create(filp->private_data, &head, &list);
+	return udmabuf_create(filp->private_data, &head, &list, false);
 }
 
-static long udmabuf_ioctl_create_list(struct file *filp, unsigned long arg)
+static long udmabuf_ioctl_create_list(struct file *filp, unsigned long arg,
+				      bool for_pcidev)
 {
 	struct udmabuf_create_list head;
 	struct udmabuf_create_item *list;
@@ -407,7 +510,7 @@ static long udmabuf_ioctl_create_list(struct file *filp, unsigned long arg)
 	if (IS_ERR(list))
 		return PTR_ERR(list);
 
-	ret = udmabuf_create(filp->private_data, &head, list);
+	ret = udmabuf_create(filp->private_data, &head, list, for_pcidev);
 	kfree(list);
 	return ret;
 }
@@ -422,7 +525,10 @@ static long udmabuf_ioctl(struct file *filp, unsigned int ioctl,
 		ret = udmabuf_ioctl_create(filp, arg);
 		break;
 	case UDMABUF_CREATE_LIST:
-		ret = udmabuf_ioctl_create_list(filp, arg);
+		ret = udmabuf_ioctl_create_list(filp, arg, false);
+		break;
+	case UDMABUF_CREATE_LIST_FOR_PCIDEV:
+		ret = udmabuf_ioctl_create_list(filp, arg, true);
 		break;
 	default:
 		ret = -ENOTTY;
-- 
2.43.0

Re: [RFC 5/7] drm/virtio: Ensure that bo's backing store is valid while updating plane

[Weifeng Liu]

On Thu, 2024-03-28 at 01:32 -0700, Vivek Kasireddy wrote:
> To make sure that the imported bo's backing store is valid, we first
> pin the associated dmabuf, import the sgt if need be and then unpin
> it after the update is complete. Note that we pin/unpin the dmabuf
> even when the backing store is valid to ensure that it does not move
> when the host update (resource_flush) is in progress.
> 
> Cc: Gerd Hoffmann <kraxel@redhat.com>
> Signed-off-by: Vivek Kasireddy <vivek.kasireddy@intel.com>
> ---
>  drivers/gpu/drm/virtio/virtgpu_plane.c | 56 +++++++++++++++++++++++++-
>  1 file changed, 55 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/gpu/drm/virtio/virtgpu_plane.c b/drivers/gpu/drm/virtio/virtgpu_plane.c
> index a72a2dbda031..3ccf88f9addc 100644
> --- a/drivers/gpu/drm/virtio/virtgpu_plane.c
> +++ b/drivers/gpu/drm/virtio/virtgpu_plane.c
> @@ -26,6 +26,7 @@
>  #include <drm/drm_atomic_helper.h>
>  #include <drm/drm_damage_helper.h>
>  #include <drm/drm_fourcc.h>
> +#include <linux/virtio_dma_buf.h>
>  
>  #include "virtgpu_drv.h"
>  
> @@ -131,6 +132,45 @@ static void virtio_gpu_update_dumb_bo(struct virtio_gpu_device *vgdev,
>  					   objs, NULL);
>  }
>  
> +static bool virtio_gpu_update_dmabuf_bo(struct virtio_gpu_device *vgdev,
> +					struct drm_gem_object *obj)
> +{
> +	struct virtio_gpu_object *bo = gem_to_virtio_gpu_obj(obj);
> +	struct dma_buf_attachment *attach = obj->import_attach;
> +	struct dma_resv *resv = attach->dmabuf->resv;
> +	struct virtio_gpu_mem_entry *ents = NULL;
> +	unsigned int nents;
> +	int ret;
> +
> +	dma_resv_lock(resv, NULL);
> +
> +	ret = dma_buf_pin(attach);
> +	if (ret) {
> +		dma_resv_unlock(resv);
> +		return false;
> +	}
> +
> +	if (!bo->has_backing) {
> +		if (bo->sgt)
> +			dma_buf_unmap_attachment(attach,
> +						 bo->sgt,
> +						 DMA_BIDIRECTIONAL);
> +
> +		ret = virtgpu_dma_buf_import_sgt(&ents, &nents,
> +						 bo, attach);
> +		if (ret)
> +			goto err_import;
> +
> +		virtio_gpu_object_attach(vgdev, bo, ents, nents);
> +	}
> +	return true;
> +
> +err_import:
> +	dma_buf_unpin(attach);
> +	dma_resv_unlock(resv);
> +	return false;
> +}
> +
>  static void virtio_gpu_resource_flush(struct drm_plane *plane,
>  				      uint32_t x, uint32_t y,
>  				      uint32_t width, uint32_t height)
> @@ -174,7 +214,9 @@ static void virtio_gpu_primary_plane_update(struct drm_plane *plane,
>  	struct virtio_gpu_device *vgdev = dev->dev_private;
>  	struct virtio_gpu_output *output = NULL;
>  	struct virtio_gpu_object *bo;
> +	struct drm_gem_object *obj;
>  	struct drm_rect rect;
> +	bool updated = false;
>  
>  	if (plane->state->crtc)
>  		output = drm_crtc_to_virtio_gpu_output(plane->state->crtc);
> @@ -196,10 +238,17 @@ static void virtio_gpu_primary_plane_update(struct drm_plane *plane,
>  	if (!drm_atomic_helper_damage_merged(old_state, plane->state, &rect))
>  		return;
>  
> -	bo = gem_to_virtio_gpu_obj(plane->state->fb->obj[0]);
> +	obj = plane->state->fb->obj[0];
> +	bo = gem_to_virtio_gpu_obj(obj);
>  	if (bo->dumb)
>  		virtio_gpu_update_dumb_bo(vgdev, plane->state, &rect);
>  
> +	if (obj->import_attach) {
> +		updated = virtio_gpu_update_dmabuf_bo(vgdev, obj);
Hi Vivek,

It's possible that the objects imported from other devices are used in
other ways apart from being scanned out (e.g., they might act as
texture resources in 3D contexts in the virtio-GPU back-end).  Thus I
think we should find a better way of updating DMA-BUF objects, like
doing so in move_notify callback.

BTW, this patch set is very useful in implementing virtual display for
the case of SR-IOV, especially it supports sharing device local memory
between host and guest.  Thanks for your work and I am really hoping it
gets merged one day!

Best regards,
-Weifeng
> +		if (!updated)
> +			return;
> +	}
> +
>  	if (plane->state->fb != old_state->fb ||
>  	    plane->state->src_w != old_state->src_w ||
>  	    plane->state->src_h != old_state->src_h ||
> @@ -239,6 +288,11 @@ static void virtio_gpu_primary_plane_update(struct drm_plane *plane,
>  				  rect.y1,
>  				  rect.x2 - rect.x1,
>  				  rect.y2 - rect.y1);
> +
> +	if (obj->import_attach && updated) {
> +		dma_buf_unpin(obj->import_attach);
> +		dma_resv_unlock(obj->import_attach->dmabuf->resv);
> +	}
>  }
>  
>  static int virtio_gpu_plane_prepare_fb(struct drm_plane *plane,

Re: [RFC 4/7] drm/virtio: Import prime buffers from other devices as guest blobs

[Daniel Vetter]

On Thu, Mar 28, 2024 at 01:32:57AM -0700, Vivek Kasireddy wrote:
> By importing scanout buffers from other devices, we should be able
> to use the virtio-gpu driver in KMS only mode. Note that we attach
> dynamically and register a move_notify() callback so that we can
> let the VMM know of any location changes associated with the backing
> store of the imported object by sending detach_backing cmd.
> 
> Cc: Gerd Hoffmann <kraxel@redhat.com>
> Signed-off-by: Vivek Kasireddy <vivek.kasireddy@intel.com>
> ---
>  drivers/gpu/drm/virtio/virtgpu_prime.c | 54 +++++++++++++++++++++++++-
>  1 file changed, 53 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/gpu/drm/virtio/virtgpu_prime.c b/drivers/gpu/drm/virtio/virtgpu_prime.c
> index 1e87dbc9a897..c65dacc1b2b5 100644
> --- a/drivers/gpu/drm/virtio/virtgpu_prime.c
> +++ b/drivers/gpu/drm/virtio/virtgpu_prime.c
> @@ -255,10 +255,36 @@ static int virtgpu_dma_buf_init_obj(struct drm_device *dev,
>  	return ret;
>  }
>  
> +static const struct drm_gem_object_funcs virtgpu_gem_dma_buf_funcs = {
> +	.free = virtgpu_dma_buf_free_obj,
> +};
> +
> +static void virtgpu_dma_buf_move_notify(struct dma_buf_attachment *attach)
> +{
> +	struct drm_gem_object *obj = attach->importer_priv;
> +	struct virtio_gpu_device *vgdev = obj->dev->dev_private;
> +	struct virtio_gpu_object *bo = gem_to_virtio_gpu_obj(obj);
> +
> +	if (bo->created) {
> +		virtio_gpu_cmd_resource_detach_backing(vgdev,
> +						       bo->hw_res_handle);
> +		bo->has_backing = false;
> +	}
> +}
> +
> +static const struct dma_buf_attach_ops virtgpu_dma_buf_attach_ops = {
> +	.allow_peer2peer = true,
> +	.move_notify = virtgpu_dma_buf_move_notify
> +};
> +
>  struct drm_gem_object *virtgpu_gem_prime_import(struct drm_device *dev,
>  						struct dma_buf *buf)
>  {
> +	struct virtio_gpu_device *vgdev = dev->dev_private;
> +	struct dma_buf_attachment *attach;
> +	struct virtio_gpu_object *bo;
>  	struct drm_gem_object *obj;
> +	int ret;
>  
>  	if (buf->ops == &virtgpu_dmabuf_ops.ops) {
>  		obj = buf->priv;
> @@ -272,7 +298,32 @@ struct drm_gem_object *virtgpu_gem_prime_import(struct drm_device *dev,
>  		}
>  	}
>  
> -	return drm_gem_prime_import(dev, buf);

I think overall this (entire series) makes sense, but needs someone with
overall virtio understanding to make sure it all fits correctly. Just a
refactor thought here: I think instead of open-coding should we have a
drm_gem_prime_dynamic_import?

Similar in another patch for the dma_buf_pin, should that be also in the
gem helpers to automatically forward to dma_buf if it's imported?

Cheers, Sima

> +	if (!vgdev->has_resource_blob || vgdev->has_virgl_3d)
> +		return drm_gem_prime_import(dev, buf);
> +
> +	bo = kzalloc(sizeof(*bo), GFP_KERNEL);
> +	if (!bo)
> +		return ERR_PTR(-ENOMEM);
> +
> +	obj = &bo->base.base;
> +	obj->funcs = &virtgpu_gem_dma_buf_funcs;
> +	drm_gem_private_object_init(dev, obj, buf->size);
> +
> +	attach = dma_buf_dynamic_attach(buf, dev->dev,
> +					&virtgpu_dma_buf_attach_ops, obj);
> +	if (IS_ERR(attach)) {
> +		kfree(bo);
> +		return ERR_CAST(attach);
> +	}
> +
> +	obj->import_attach = attach;
> +	get_dma_buf(buf);
> +
> +	ret = virtgpu_dma_buf_init_obj(dev, bo, attach);
> +	if (ret < 0)
> +		return ERR_PTR(ret);
> +
> +	return obj;
>  }
>  
>  struct drm_gem_object *virtgpu_gem_prime_import_sg_table(
> @@ -281,3 +332,4 @@ struct drm_gem_object *virtgpu_gem_prime_import_sg_table(
>  {
>  	return ERR_PTR(-ENODEV);
>  }
> +
> -- 
> 2.43.0
> 

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Re: [RFC 0/7] drm/virtio: Import scanout buffers from other devices

[Gurchetan Singh]

[-- Attachment #1: Type: text/plain, Size: 6990 bytes --]

On Thu, Mar 28, 2024 at 2:01 AM Vivek Kasireddy <vivek.kasireddy@intel.com>
wrote:

> Having virtio-gpu import scanout buffers (via prime) from other
> devices means that we'd be adding a head to headless GPUs assigned
> to a Guest VM or additional heads to regular GPU devices that are
> passthrough'd to the Guest. In these cases, the Guest compositor
> can render into the scanout buffer using a primary GPU and has the
> secondary GPU (virtio-gpu) import it for display purposes.
>
> The main advantage with this is that the imported scanout buffer can
> either be displayed locally on the Host (e.g, using Qemu + GTK UI)
> or encoded and streamed to a remote client (e.g, Qemu + Spice UI).
> Note that since Qemu uses udmabuf driver, there would be no copies
> made of the scanout buffer as it is displayed. This should be
> possible even when it might reside in device memory such has VRAM.
>
> The specific use-case that can be supported with this series is when
> running Weston or other guest compositors with "additional-devices"
> feature (./weston --drm-device=card1 --additional-devices=card0).
> More info about this feature can be found at:
> https://gitlab.freedesktop.org/wayland/weston/-/merge_requests/736
>
> In the above scenario, card1 could be a dGPU or an iGPU and card0
> would be virtio-gpu in KMS only mode. However, the case where this
> patch series could be particularly useful is when card1 is a GPU VF
> that needs to share its scanout buffer (in a zero-copy way) with the
> GPU PF on the Host. Or, it can also be useful when the scanout buffer
> needs to be shared between any two GPU devices (assuming one of them
> is assigned to a Guest VM) as long as they are P2P DMA compatible.
>

Is passthrough iGPU-only or passthrough dGPU-only something you intend to
use?

If it's a dGPU + iGPU setup, then the way other people seem to do it is a
"virtualized" iGPU (via virgl/gfxstream/take your pick) and pass-through
the dGPU.

For example, AMD seems to use virgl to allocate and import into the dGPU.

https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23896
https://lore.kernel.org/all/20231221100016.4022353-1-julia.zhang@amd.com/

ChromeOS also uses that method (see crrev.com/c/3764931) [cc: dGPU
architect +Dominik Behr <dbehr@google.com>]

So if iGPU + dGPU is the primary use case, you should be able to use these
methods as well.  The model would "virtualized iGPU" + passthrough dGPU,
not split SoCs.


> As part of the import, the virtio-gpu driver shares the dma
> addresses and lengths with Qemu which then determines whether the
> memory region they belong to is owned by a PCI device or whether it
> is part of the Guest's system ram. If it is the former, it identifies
> the devid (or bdf) and bar and provides this info (along with offsets
> and sizes) to the udmabuf driver. In the latter case, instead of the
> the devid and bar it provides the memfd. The udmabuf driver then
> creates a dmabuf using this info that Qemu shares with Spice for
> encode via Gstreamer.
>
> Note that the virtio-gpu driver registers a move_notify() callback
> to track location changes associated with the scanout buffer and
> sends attach/detach backing cmds to Qemu when appropriate. And,
> synchronization (that is, ensuring that Guest and Host are not
> using the scanout buffer at the same time) is ensured by pinning/
> unpinning the dmabuf as part of plane update and using a fence
> in resource_flush cmd.


I'm not sure how QEMU's display paths work, but with crosvm if you share
the guest-created dmabuf with the display, and the guest moves the backing
pages, the only recourse is the destroy the surface and show a black screen
to the user: not the best thing experience wise.

Only amdgpu calls dma_buf_move_notfiy(..), and you're probably testing on
Intel only, so you may not be hitting that code path anyways.  I forgot the
exact reason, but apparently udmabuf may not work with amdgpu displays and
it seems the virtualized iGPU + dGPU is the way to go for amdgpu anyways.
So I recommend just pinning the buffer for the lifetime of the import for
simplicity and correctness.


> This series is available at:
> https://gitlab.freedesktop.org/Vivek/drm-tip/-/commits/virtgpu_import_rfc
>
> along with additional patches for Qemu and Spice here:
> https://gitlab.freedesktop.org/Vivek/qemu/-/commits/virtgpu_dmabuf_pcidev
> https://gitlab.freedesktop.org/Vivek/spice/-/commits/encode_dmabuf_v4
>
> Patchset overview:
>
> Patch 1:   Implement VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING cmd
> Patch 2-3: Helpers to initalize, import, free imported object
> Patch 4-5: Import and use buffers from other devices for scanout
> Patch 6-7: Have udmabuf driver create dmabuf from PCI bars for P2P DMA
>
> This series is tested using the following method:
> - Run Qemu with the following relevant options:
>   qemu-system-x86_64 -m 4096m ....
>   -device vfio-pci,host=0000:03:00.0
>   -device virtio-vga,max_outputs=1,blob=true,xres=1920,yres=1080
>   -spice
> port=3001,gl=on,disable-ticketing=on,preferred-codec=gstreamer:h264
>   -object memory-backend-memfd,id=mem1,size=4096M
>   -machine memory-backend=mem1 ...
> - Run upstream Weston with the following options in the Guest VM:
>   ./weston --drm-device=card1 --additional-devices=card0
>
> where card1 is a DG2 dGPU (passthrough'd and using xe driver in Guest VM),
> card0 is virtio-gpu and the Host is using a RPL iGPU.
>
> Cc: Gerd Hoffmann <kraxel@redhat.com>
> Cc: Dongwon Kim <dongwon.kim@intel.com>
> Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
> Cc: Christian Koenig <christian.koenig@amd.com>
> Cc: Dmitry Osipenko <dmitry.osipenko@collabora.com>
> Cc: Rob Clark <robdclark@chromium.org>
> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> Cc: Oded Gabbay <ogabbay@kernel.org>
> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com>
> Cc: Michael Tretter <m.tretter@pengutronix.de>
>
> Vivek Kasireddy (7):
>   drm/virtio: Implement VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING cmd
>   drm/virtio: Add a helper to map and note the dma addrs and lengths
>   drm/virtio: Add helpers to initialize and free the imported object
>   drm/virtio: Import prime buffers from other devices as guest blobs
>   drm/virtio: Ensure that bo's backing store is valid while updating
>     plane
>   udmabuf/uapi: Add new ioctl to create a dmabuf from PCI bar regions
>   udmabuf: Implement UDMABUF_CREATE_LIST_FOR_PCIDEV ioctl
>
>  drivers/dma-buf/udmabuf.c              | 122 ++++++++++++++++--
>  drivers/gpu/drm/virtio/virtgpu_drv.h   |   8 ++
>  drivers/gpu/drm/virtio/virtgpu_plane.c |  56 ++++++++-
>  drivers/gpu/drm/virtio/virtgpu_prime.c | 167 ++++++++++++++++++++++++-
>  drivers/gpu/drm/virtio/virtgpu_vq.c    |  15 +++
>  include/uapi/linux/udmabuf.h           |  11 +-
>  6 files changed, 368 insertions(+), 11 deletions(-)
>
> --
> 2.43.0
>
>

[-- Attachment #2: Type: text/html, Size: 9595 bytes --]

RE: [RFC 0/7] drm/virtio: Import scanout buffers from other devices

[Kasireddy, Vivek]

[-- Attachment #1: Type: text/plain, Size: 8639 bytes --]

Hi Gurchetan,

Thank you for taking a look at this patch series!



On Thu, Mar 28, 2024 at 2:01 AM Vivek Kasireddy <vivek.kasireddy@intel.com<mailto:vivek.kasireddy@intel.com>> wrote:
Having virtio-gpu import scanout buffers (via prime) from other
devices means that we'd be adding a head to headless GPUs assigned
to a Guest VM or additional heads to regular GPU devices that are
passthrough'd to the Guest. In these cases, the Guest compositor
can render into the scanout buffer using a primary GPU and has the
secondary GPU (virtio-gpu) import it for display purposes.

The main advantage with this is that the imported scanout buffer can
either be displayed locally on the Host (e.g, using Qemu + GTK UI)
or encoded and streamed to a remote client (e.g, Qemu + Spice UI).
Note that since Qemu uses udmabuf driver, there would be no copies
made of the scanout buffer as it is displayed. This should be
possible even when it might reside in device memory such has VRAM.

The specific use-case that can be supported with this series is when
running Weston or other guest compositors with "additional-devices"
feature (./weston --drm-device=card1 --additional-devices=card0).
More info about this feature can be found at:
https://gitlab.freedesktop.org/wayland/weston/-/merge_requests/736

In the above scenario, card1 could be a dGPU or an iGPU and card0
would be virtio-gpu in KMS only mode. However, the case where this
patch series could be particularly useful is when card1 is a GPU VF
that needs to share its scanout buffer (in a zero-copy way) with the
GPU PF on the Host. Or, it can also be useful when the scanout buffer
needs to be shared between any two GPU devices (assuming one of them
is assigned to a Guest VM) as long as they are P2P DMA compatible.

Is passthrough iGPU-only or passthrough dGPU-only something you intend to use?
Our main use-case involves passthrough’g a headless dGPU VF device and sharing
the Guest compositor’s scanout buffer with dGPU PF device on the Host. Same goal for
headless iGPU VF to iGPU PF device as well.

However, using a combination of iGPU and dGPU where either of them can be passthrough’d
to the Guest is something I think can be supported with this patch series as well.

If it's a dGPU + iGPU setup, then the way other people seem to do it is a "virtualized" iGPU (via virgl/gfxstream/take your pick) and pass-through the dGPU.

For example, AMD seems to use virgl to allocate and import into the dGPU.

https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23896
https://lore.kernel.org/all/20231221100016.4022353-1-julia.zhang@amd.com/

ChromeOS also uses that method (see crrev.com/c/3764931<http://crrev.com/c/3764931>) [cc: dGPU architect +Dominik Behr<mailto:dbehr@google.com>]

So if iGPU + dGPU is the primary use case, you should be able to use these methods as well.  The model would "virtualized iGPU" + passthrough dGPU, not split SoCs.
In our use-case, the goal is to have only one primary GPU (passthrough’d iGPU/dGPU)
do all the rendering (using native DRI drivers) for clients/compositor and all the outputs
and share the scanout buffers with the secondary GPU (virtio-gpu). Since this is mostly
how Mutter (and also Weston) work in a multi-GPU setup, I am not sure if virgl is needed.

As part of the import, the virtio-gpu driver shares the dma
addresses and lengths with Qemu which then determines whether the
memory region they belong to is owned by a PCI device or whether it
is part of the Guest's system ram. If it is the former, it identifies
the devid (or bdf) and bar and provides this info (along with offsets
and sizes) to the udmabuf driver. In the latter case, instead of the
the devid and bar it provides the memfd. The udmabuf driver then
creates a dmabuf using this info that Qemu shares with Spice for
encode via Gstreamer.

Note that the virtio-gpu driver registers a move_notify() callback
to track location changes associated with the scanout buffer and
sends attach/detach backing cmds to Qemu when appropriate. And,
synchronization (that is, ensuring that Guest and Host are not
using the scanout buffer at the same time) is ensured by pinning/
unpinning the dmabuf as part of plane update and using a fence
in resource_flush cmd.

I'm not sure how QEMU's display paths work, but with crosvm if you share the guest-created dmabuf with the display, and the guest moves the backing pages, the only recourse is the destroy the surface and show a black screen to the user: not the best thing experience wise.
Since Qemu GTK UI uses EGL, there is a blit done from the guest’s scanout buffer onto an EGL
backed buffer on the Host. So, this problem would not happen as of now.

Only amdgpu calls dma_buf_move_notfiy(..), and you're probably testing on Intel only, so you may not be hitting that code path anyways.
I have tested with the Xe driver in the Guest which also calls dma_buf_move_notfiy(). However,
note that for dGPUs, both Xe and amdgpu migrate the scanout buffer from vram to system
memory as part of export, because virtio-gpu is not P2P compatible.

I forgot the exact reason, but apparently udmabuf may not work with amdgpu displays and it seems the virtualized iGPU + dGPU is the way to go for amdgpu anyways.
I am curious why udmabuf would not work with amdgpu?

So I recommend just pinning the buffer for the lifetime of the import for simplicity and correctness.
Yeah, in this patch series, the dmabuf is indeed pinned but only for a short duration in the Guest –
just until the Host is done using it (blit or encode).

Thanks,
Vivek

This series is available at:
https://gitlab.freedesktop.org/Vivek/drm-tip/-/commits/virtgpu_import_rfc

along with additional patches for Qemu and Spice here:
https://gitlab.freedesktop.org/Vivek/qemu/-/commits/virtgpu_dmabuf_pcidev
https://gitlab.freedesktop.org/Vivek/spice/-/commits/encode_dmabuf_v4

Patchset overview:

Patch 1:   Implement VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING cmd
Patch 2-3: Helpers to initalize, import, free imported object
Patch 4-5: Import and use buffers from other devices for scanout
Patch 6-7: Have udmabuf driver create dmabuf from PCI bars for P2P DMA

This series is tested using the following method:
- Run Qemu with the following relevant options:
  qemu-system-x86_64 -m 4096m ....
  -device vfio-pci,host=0000:03:00.0
  -device virtio-vga,max_outputs=1,blob=true,xres=1920,yres=1080
  -spice port=3001,gl=on,disable-ticketing=on,preferred-codec=gstreamer:h264
  -object memory-backend-memfd,id=mem1,size=4096M
  -machine memory-backend=mem1 ...
- Run upstream Weston with the following options in the Guest VM:
  ./weston --drm-device=card1 --additional-devices=card0

where card1 is a DG2 dGPU (passthrough'd and using xe driver in Guest VM),
card0 is virtio-gpu and the Host is using a RPL iGPU.

Cc: Gerd Hoffmann <kraxel@redhat.com<mailto:kraxel@redhat.com>>
Cc: Dongwon Kim <dongwon.kim@intel.com<mailto:dongwon.kim@intel.com>>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch<mailto:daniel.vetter@ffwll.ch>>
Cc: Christian Koenig <christian.koenig@amd.com<mailto:christian.koenig@amd.com>>
Cc: Dmitry Osipenko <dmitry.osipenko@collabora.com<mailto:dmitry.osipenko@collabora.com>>
Cc: Rob Clark <robdclark@chromium.org<mailto:robdclark@chromium.org>>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com<mailto:thomas.hellstrom@linux.intel.com>>
Cc: Oded Gabbay <ogabbay@kernel.org<mailto:ogabbay@kernel.org>>
Cc: Michal Wajdeczko <michal.wajdeczko@intel.com<mailto:michal.wajdeczko@intel.com>>
Cc: Michael Tretter <m.tretter@pengutronix.de<mailto:m.tretter@pengutronix.de>>

Vivek Kasireddy (7):
  drm/virtio: Implement VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING cmd
  drm/virtio: Add a helper to map and note the dma addrs and lengths
  drm/virtio: Add helpers to initialize and free the imported object
  drm/virtio: Import prime buffers from other devices as guest blobs
  drm/virtio: Ensure that bo's backing store is valid while updating
    plane
  udmabuf/uapi: Add new ioctl to create a dmabuf from PCI bar regions
  udmabuf: Implement UDMABUF_CREATE_LIST_FOR_PCIDEV ioctl

 drivers/dma-buf/udmabuf.c              | 122 ++++++++++++++++--
 drivers/gpu/drm/virtio/virtgpu_drv.h   |   8 ++
 drivers/gpu/drm/virtio/virtgpu_plane.c |  56 ++++++++-
 drivers/gpu/drm/virtio/virtgpu_prime.c | 167 ++++++++++++++++++++++++-
 drivers/gpu/drm/virtio/virtgpu_vq.c    |  15 +++
 include/uapi/linux/udmabuf.h           |  11 +-
 6 files changed, 368 insertions(+), 11 deletions(-)

--
2.43.0

[-- Attachment #2: Type: text/html, Size: 15351 bytes --]

RE: [RFC 0/7] drm/virtio: Import scanout buffers from other devices

[Kasireddy, Vivek]

Hi,

Sorry, my previous reply got messed up as a result of HTML formatting. This is
a plain text version of the same reply.

> 
> 
> 	Having virtio-gpu import scanout buffers (via prime) from other
> 	devices means that we'd be adding a head to headless GPUs assigned
> 	to a Guest VM or additional heads to regular GPU devices that are
> 	passthrough'd to the Guest. In these cases, the Guest compositor
> 	can render into the scanout buffer using a primary GPU and has the
> 	secondary GPU (virtio-gpu) import it for display purposes.
> 
> 	The main advantage with this is that the imported scanout buffer can
> 	either be displayed locally on the Host (e.g, using Qemu + GTK UI)
> 	or encoded and streamed to a remote client (e.g, Qemu + Spice UI).
> 	Note that since Qemu uses udmabuf driver, there would be no
> copies
> 	made of the scanout buffer as it is displayed. This should be
> 	possible even when it might reside in device memory such has
> VRAM.
> 
> 	The specific use-case that can be supported with this series is when
> 	running Weston or other guest compositors with "additional-devices"
> 	feature (./weston --drm-device=card1 --additional-devices=card0).
> 	More info about this feature can be found at:
> 	https://gitlab.freedesktop.org/wayland/weston/-
> /merge_requests/736
> 
> 	In the above scenario, card1 could be a dGPU or an iGPU and card0
> 	would be virtio-gpu in KMS only mode. However, the case where this
> 	patch series could be particularly useful is when card1 is a GPU VF
> 	that needs to share its scanout buffer (in a zero-copy way) with the
> 	GPU PF on the Host. Or, it can also be useful when the scanout buffer
> 	needs to be shared between any two GPU devices (assuming one of
> them
> 	is assigned to a Guest VM) as long as they are P2P DMA compatible.
> 
> 
> 
> Is passthrough iGPU-only or passthrough dGPU-only something you intend to
> use?
Our main use-case involves passthrough’g a headless dGPU VF device and sharing
the Guest compositor’s scanout buffer with dGPU PF device on the Host. Same goal for
headless iGPU VF to iGPU PF device as well.

However, using a combination of iGPU and dGPU where either of them can be passthrough’d
to the Guest is something I think can be supported with this patch series as well.

> 
> If it's a dGPU + iGPU setup, then the way other people seem to do it is a
> "virtualized" iGPU (via virgl/gfxstream/take your pick) and pass-through the
> dGPU.
> 
> For example, AMD seems to use virgl to allocate and import into the dGPU.
> 
> https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23896
> 
> https://lore.kernel.org/all/20231221100016.4022353-1-
> julia.zhang@amd.com/
> 
> 
> ChromeOS also uses that method (see crrev.com/c/3764931
> <http://crrev.com/c/3764931> ) [cc: dGPU architect +Dominik Behr
> <mailto:dbehr@google.com> ]
> 
> So if iGPU + dGPU is the primary use case, you should be able to use these
> methods as well.  The model would "virtualized iGPU" + passthrough dGPU,
> not split SoCs.
In our use-case, the goal is to have only one primary GPU (passthrough’d iGPU/dGPU)
do all the rendering (using native DRI drivers) for clients/compositor and all the outputs
and share the scanout buffers with the secondary GPU (virtio-gpu). Since this is mostly
how Mutter (and also Weston) work in a multi-GPU setup, I am not sure if virgl is needed.

And, doing it this way means that no other userspace components need to be modified
on both the Guest and the Host.

> 
> 
> 
> 	As part of the import, the virtio-gpu driver shares the dma
> 	addresses and lengths with Qemu which then determines whether
> the
> 	memory region they belong to is owned by a PCI device or whether it
> 	is part of the Guest's system ram. If it is the former, it identifies
> 	the devid (or bdf) and bar and provides this info (along with offsets
> 	and sizes) to the udmabuf driver. In the latter case, instead of the
> 	the devid and bar it provides the memfd. The udmabuf driver then
> 	creates a dmabuf using this info that Qemu shares with Spice for
> 	encode via Gstreamer.
> 
> 	Note that the virtio-gpu driver registers a move_notify() callback
> 	to track location changes associated with the scanout buffer and
> 	sends attach/detach backing cmds to Qemu when appropriate. And,
> 	synchronization (that is, ensuring that Guest and Host are not
> 	using the scanout buffer at the same time) is ensured by pinning/
> 	unpinning the dmabuf as part of plane update and using a fence
> 	in resource_flush cmd.
> 
> 
> I'm not sure how QEMU's display paths work, but with crosvm if you share
> the guest-created dmabuf with the display, and the guest moves the backing
> pages, the only recourse is the destroy the surface and show a black screen
> to the user: not the best thing experience wise.
Since Qemu GTK UI uses EGL, there is a blit done from the guest’s scanout buffer onto an EGL
backed buffer on the Host. So, this problem would not happen as of now.

> 
> Only amdgpu calls dma_buf_move_notfiy(..), and you're probably testing on
> Intel only, so you may not be hitting that code path anyways.
I have tested with the Xe driver in the Guest which also calls dma_buf_move_notfiy(). But
note that for dGPUs, both Xe and amdgpu migrate the scanout buffer from vram to system
memory as part of export, because virtio-gpu is not P2P compatible. However, I am hoping
to relax this (p2p check against virtio-gpu) in Xe driver if it detects that it is running in
VF mode once the following patch series is merged:
https://lore.kernel.org/dri-devel/20240422063602.3690124-1-vivek.kasireddy@intel.com/

> I forgot the
> exact reason, but apparently udmabuf may not work with amdgpu displays
> and it seems the virtualized iGPU + dGPU is the way to go for amdgpu
> anyways. 
I would really like to know why udmabuf would not work with amdgpu?

> So I recommend just pinning the buffer for the lifetime of the
> import for simplicity and correctness.
Yeah, in this patch series, the dmabuf is indeed pinned, but only for a short duration in the Guest –
just until the Host is done using it (blit or encode).

Thanks,
Vivek

> 
> 
> 	This series is available at:
> 	https://gitlab.freedesktop.org/Vivek/drm-tip/-
> /commits/virtgpu_import_rfc
> 
> 	along with additional patches for Qemu and Spice here:
> 	https://gitlab.freedesktop.org/Vivek/qemu/-
> /commits/virtgpu_dmabuf_pcidev
> 	https://gitlab.freedesktop.org/Vivek/spice/-
> /commits/encode_dmabuf_v4
> 
> 	Patchset overview:
> 
> 	Patch 1:   Implement
> VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING cmd
> 	Patch 2-3: Helpers to initalize, import, free imported object
> 	Patch 4-5: Import and use buffers from other devices for scanout
> 	Patch 6-7: Have udmabuf driver create dmabuf from PCI bars for P2P
> DMA
> 
> 	This series is tested using the following method:
> 	- Run Qemu with the following relevant options:
> 	  qemu-system-x86_64 -m 4096m ....
> 	  -device vfio-pci,host=0000:03:00.0
> 	  -device virtio-vga,max_outputs=1,blob=true,xres=1920,yres=1080
> 	  -spice port=3001,gl=on,disable-ticketing=on,preferred-
> codec=gstreamer:h264
> 	  -object memory-backend-memfd,id=mem1,size=4096M
> 	  -machine memory-backend=mem1 ...
> 	- Run upstream Weston with the following options in the Guest VM:
> 	  ./weston --drm-device=card1 --additional-devices=card0
> 
> 	where card1 is a DG2 dGPU (passthrough'd and using xe driver in
> Guest VM),
> 	card0 is virtio-gpu and the Host is using a RPL iGPU.
> 
> 	Cc: Gerd Hoffmann <kraxel@redhat.com
> <mailto:kraxel@redhat.com> >
> 	Cc: Dongwon Kim <dongwon.kim@intel.com
> <mailto:dongwon.kim@intel.com> >
> 	Cc: Daniel Vetter <daniel.vetter@ffwll.ch
> <mailto:daniel.vetter@ffwll.ch> >
> 	Cc: Christian Koenig <christian.koenig@amd.com
> <mailto:christian.koenig@amd.com> >
> 	Cc: Dmitry Osipenko <dmitry.osipenko@collabora.com
> <mailto:dmitry.osipenko@collabora.com> >
> 	Cc: Rob Clark <robdclark@chromium.org
> <mailto:robdclark@chromium.org> >
> 	Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com
> <mailto:thomas.hellstrom@linux.intel.com> >
> 	Cc: Oded Gabbay <ogabbay@kernel.org
> <mailto:ogabbay@kernel.org> >
> 	Cc: Michal Wajdeczko <michal.wajdeczko@intel.com
> <mailto:michal.wajdeczko@intel.com> >
> 	Cc: Michael Tretter <m.tretter@pengutronix.de
> <mailto:m.tretter@pengutronix.de> >
> 
> 	Vivek Kasireddy (7):
> 	  drm/virtio: Implement
> VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING cmd
> 	  drm/virtio: Add a helper to map and note the dma addrs and
> lengths
> 	  drm/virtio: Add helpers to initialize and free the imported object
> 	  drm/virtio: Import prime buffers from other devices as guest blobs
> 	  drm/virtio: Ensure that bo's backing store is valid while updating
> 	    plane
> 	  udmabuf/uapi: Add new ioctl to create a dmabuf from PCI bar
> regions
> 	  udmabuf: Implement UDMABUF_CREATE_LIST_FOR_PCIDEV ioctl
> 
> 	 drivers/dma-buf/udmabuf.c              | 122 ++++++++++++++++--
> 	 drivers/gpu/drm/virtio/virtgpu_drv.h   |   8 ++
> 	 drivers/gpu/drm/virtio/virtgpu_plane.c |  56 ++++++++-
> 	 drivers/gpu/drm/virtio/virtgpu_prime.c | 167
> ++++++++++++++++++++++++-
> 	 drivers/gpu/drm/virtio/virtgpu_vq.c    |  15 +++
> 	 include/uapi/linux/udmabuf.h           |  11 +-
> 	 6 files changed, 368 insertions(+), 11 deletions(-)
> 
> 	--
> 	2.43.0
> 
> 

Re: [RFC 0/7] drm/virtio: Import scanout buffers from other devices

[Gurchetan Singh]

[-- Attachment #1: Type: text/plain, Size: 12156 bytes --]

On Fri, May 24, 2024 at 11:33 AM Kasireddy, Vivek <vivek.kasireddy@intel.com>
wrote:

> Hi,
>
> Sorry, my previous reply got messed up as a result of HTML formatting.
> This is
> a plain text version of the same reply.
>
> >
> >
> >       Having virtio-gpu import scanout buffers (via prime) from other
> >       devices means that we'd be adding a head to headless GPUs assigned
> >       to a Guest VM or additional heads to regular GPU devices that are
> >       passthrough'd to the Guest. In these cases, the Guest compositor
> >       can render into the scanout buffer using a primary GPU and has the
> >       secondary GPU (virtio-gpu) import it for display purposes.
> >
> >       The main advantage with this is that the imported scanout buffer
> can
> >       either be displayed locally on the Host (e.g, using Qemu + GTK UI)
> >       or encoded and streamed to a remote client (e.g, Qemu + Spice UI).
> >       Note that since Qemu uses udmabuf driver, there would be no
> > copies
> >       made of the scanout buffer as it is displayed. This should be
> >       possible even when it might reside in device memory such has
> > VRAM.
> >
> >       The specific use-case that can be supported with this series is
> when
> >       running Weston or other guest compositors with "additional-devices"
> >       feature (./weston --drm-device=card1 --additional-devices=card0).
> >       More info about this feature can be found at:
> >       https://gitlab.freedesktop.org/wayland/weston/-
> > /merge_requests/736
> >
> >       In the above scenario, card1 could be a dGPU or an iGPU and card0
> >       would be virtio-gpu in KMS only mode. However, the case where this
> >       patch series could be particularly useful is when card1 is a GPU VF
> >       that needs to share its scanout buffer (in a zero-copy way) with
> the
> >       GPU PF on the Host. Or, it can also be useful when the scanout
> buffer
> >       needs to be shared between any two GPU devices (assuming one of
> > them
> >       is assigned to a Guest VM) as long as they are P2P DMA compatible.
> >
> >
> >
> > Is passthrough iGPU-only or passthrough dGPU-only something you intend to
> > use?
> Our main use-case involves passthrough’g a headless dGPU VF device and
> sharing
> the Guest compositor’s scanout buffer with dGPU PF device on the Host.
> Same goal for
> headless iGPU VF to iGPU PF device as well.
>

Just to check my understanding: the same physical {i, d}GPU is partitioned
into the VF and PF, but the PF handles host-side display integration and
rendering?


> However, using a combination of iGPU and dGPU where either of them can be
> passthrough’d
> to the Guest is something I think can be supported with this patch series
> as well.
>
> >
> > If it's a dGPU + iGPU setup, then the way other people seem to do it is a
> > "virtualized" iGPU (via virgl/gfxstream/take your pick) and pass-through
> the
> > dGPU.
> >
> > For example, AMD seems to use virgl to allocate and import into the dGPU.
> >
> > https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23896
> >
> > https://lore.kernel.org/all/20231221100016.4022353-1-
> > julia.zhang@amd.com/
> >
> >
> > ChromeOS also uses that method (see crrev.com/c/3764931
> > <http://crrev.com/c/3764931> ) [cc: dGPU architect +Dominik Behr
> > <mailto:dbehr@google.com> ]
> >
> > So if iGPU + dGPU is the primary use case, you should be able to use
> these
> > methods as well.  The model would "virtualized iGPU" + passthrough dGPU,
> > not split SoCs.
> In our use-case, the goal is to have only one primary GPU (passthrough’d
> iGPU/dGPU)
> do all the rendering (using native DRI drivers) for clients/compositor and
> all the outputs
> and share the scanout buffers with the secondary GPU (virtio-gpu). Since
> this is mostly
> how Mutter (and also Weston) work in a multi-GPU setup, I am not sure if
> virgl is needed.
>

I think you can probably use virgl with the PF and others probably will,
but supporting multiple methods in Linux is not unheard of.

Does your patchset need the Mesa kmsro patchset to function correctly?

https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9592

If so, I would try to get that reviewed first to meet DRM requirements (
https://dri.freedesktop.org/docs/drm/gpu/drm-uapi.html#open-source-userspace-requirements).
You might explicitly call out the design decision you're making: ("We can
probably use virgl as the virtualized iGPU via PF, but that adds
unnecessary complexity b/c ______").

And, doing it this way means that no other userspace components need to be
> modified
> on both the Guest and the Host.
>
> >
> >
> >
> >       As part of the import, the virtio-gpu driver shares the dma
> >       addresses and lengths with Qemu which then determines whether
> > the
> >       memory region they belong to is owned by a PCI device or whether it
> >       is part of the Guest's system ram. If it is the former, it
> identifies
> >       the devid (or bdf) and bar and provides this info (along with
> offsets
> >       and sizes) to the udmabuf driver. In the latter case, instead of
> the
> >       the devid and bar it provides the memfd. The udmabuf driver then
> >       creates a dmabuf using this info that Qemu shares with Spice for
> >       encode via Gstreamer.
> >
> >       Note that the virtio-gpu driver registers a move_notify() callback
> >       to track location changes associated with the scanout buffer and
> >       sends attach/detach backing cmds to Qemu when appropriate. And,
> >       synchronization (that is, ensuring that Guest and Host are not
> >       using the scanout buffer at the same time) is ensured by pinning/
> >       unpinning the dmabuf as part of plane update and using a fence
> >       in resource_flush cmd.
> >
> >
> > I'm not sure how QEMU's display paths work, but with crosvm if you share
> > the guest-created dmabuf with the display, and the guest moves the
> backing
> > pages, the only recourse is the destroy the surface and show a black
> screen
> > to the user: not the best thing experience wise.
> Since Qemu GTK UI uses EGL, there is a blit done from the guest’s scanout
> buffer onto an EGL
> backed buffer on the Host. So, this problem would not happen as of now.
>

The guest kernel doesn't know you're using the QEMU GTK UI + EGL
host-side.

If somebody wants to use the virtio-gpu import mechanism with lower-level
Wayland-based display integration, then the problem would occur.

Perhaps, do that just to be safe unless you have performance concerns.

>
> > Only amdgpu calls dma_buf_move_notfiy(..), and you're probably testing on
> > Intel only, so you may not be hitting that code path anyways.
> I have tested with the Xe driver in the Guest which also calls
> dma_buf_move_notfiy(). But
> note that for dGPUs, both Xe and amdgpu migrate the scanout buffer from
> vram to system
> memory as part of export, because virtio-gpu is not P2P compatible.
> However, I am hoping
> to relax this (p2p check against virtio-gpu) in Xe driver if it detects
> that it is running in
> VF mode once the following patch series is merged:
>
> https://lore.kernel.org/dri-devel/20240422063602.3690124-1-vivek.kasireddy@intel.com/
>
> > I forgot the
> > exact reason, but apparently udmabuf may not work with amdgpu displays
> > and it seems the virtualized iGPU + dGPU is the way to go for amdgpu
> > anyways.
> I would really like to know why udmabuf would not work with amdgpu?
>

It's just a rumor I heard, but the idea is udmabuf would be imported into
AMDGPU_GEM_DOMAIN_CPU only.

https://cgit.freedesktop.org/drm/drm-misc/tree/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c#n333

"AMDGPU_GEM_DOMAIN_CPU: System memory that is not GPU accessible. Memory in
this pool could be swapped out to disk if there is pressure."

https://dri.freedesktop.org/docs/drm/gpu/amdgpu.html

Perhaps that limitation is artificial and unnecessary, and it may indeed
work.  I don't think anybody has tried...


>
> > So I recommend just pinning the buffer for the lifetime of the
> > import for simplicity and correctness.
> Yeah, in this patch series, the dmabuf is indeed pinned, but only for a
> short duration in the Guest –
> just until the Host is done using it (blit or encode).
>
> Thanks,
> Vivek
>
> >
> >
> >       This series is available at:
> >       https://gitlab.freedesktop.org/Vivek/drm-tip/-
> > /commits/virtgpu_import_rfc
> >
> >       along with additional patches for Qemu and Spice here:
> >       https://gitlab.freedesktop.org/Vivek/qemu/-
> > /commits/virtgpu_dmabuf_pcidev
> >       https://gitlab.freedesktop.org/Vivek/spice/-
> > /commits/encode_dmabuf_v4
> >
> >       Patchset overview:
> >
> >       Patch 1:   Implement
> > VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING cmd
> >       Patch 2-3: Helpers to initalize, import, free imported object
> >       Patch 4-5: Import and use buffers from other devices for scanout
> >       Patch 6-7: Have udmabuf driver create dmabuf from PCI bars for P2P
> > DMA
> >
> >       This series is tested using the following method:
> >       - Run Qemu with the following relevant options:
> >         qemu-system-x86_64 -m 4096m ....
> >         -device vfio-pci,host=0000:03:00.0
> >         -device virtio-vga,max_outputs=1,blob=true,xres=1920,yres=1080
> >         -spice port=3001,gl=on,disable-ticketing=on,preferred-
> > codec=gstreamer:h264
> >         -object memory-backend-memfd,id=mem1,size=4096M
> >         -machine memory-backend=mem1 ...
> >       - Run upstream Weston with the following options in the Guest VM:
> >         ./weston --drm-device=card1 --additional-devices=card0
> >
> >       where card1 is a DG2 dGPU (passthrough'd and using xe driver in
> > Guest VM),
> >       card0 is virtio-gpu and the Host is using a RPL iGPU.
> >
> >       Cc: Gerd Hoffmann <kraxel@redhat.com
> > <mailto:kraxel@redhat.com> >
> >       Cc: Dongwon Kim <dongwon.kim@intel.com
> > <mailto:dongwon.kim@intel.com> >
> >       Cc: Daniel Vetter <daniel.vetter@ffwll.ch
> > <mailto:daniel.vetter@ffwll.ch> >
> >       Cc: Christian Koenig <christian.koenig@amd.com
> > <mailto:christian.koenig@amd.com> >
> >       Cc: Dmitry Osipenko <dmitry.osipenko@collabora.com
> > <mailto:dmitry.osipenko@collabora.com> >
> >       Cc: Rob Clark <robdclark@chromium.org
> > <mailto:robdclark@chromium.org> >
> >       Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com
> > <mailto:thomas.hellstrom@linux.intel.com> >
> >       Cc: Oded Gabbay <ogabbay@kernel.org
> > <mailto:ogabbay@kernel.org> >
> >       Cc: Michal Wajdeczko <michal.wajdeczko@intel.com
> > <mailto:michal.wajdeczko@intel.com> >
> >       Cc: Michael Tretter <m.tretter@pengutronix.de
> > <mailto:m.tretter@pengutronix.de> >
> >
> >       Vivek Kasireddy (7):
> >         drm/virtio: Implement
> > VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING cmd
> >         drm/virtio: Add a helper to map and note the dma addrs and
> > lengths
> >         drm/virtio: Add helpers to initialize and free the imported
> object
> >         drm/virtio: Import prime buffers from other devices as guest
> blobs
> >         drm/virtio: Ensure that bo's backing store is valid while
> updating
> >           plane
> >         udmabuf/uapi: Add new ioctl to create a dmabuf from PCI bar
> > regions
> >         udmabuf: Implement UDMABUF_CREATE_LIST_FOR_PCIDEV ioctl
> >
> >        drivers/dma-buf/udmabuf.c              | 122 ++++++++++++++++--
> >        drivers/gpu/drm/virtio/virtgpu_drv.h   |   8 ++
> >        drivers/gpu/drm/virtio/virtgpu_plane.c |  56 ++++++++-
> >        drivers/gpu/drm/virtio/virtgpu_prime.c | 167
> > ++++++++++++++++++++++++-
> >        drivers/gpu/drm/virtio/virtgpu_vq.c    |  15 +++
> >        include/uapi/linux/udmabuf.h           |  11 +-
> >        6 files changed, 368 insertions(+), 11 deletions(-)
> >
> >       --
> >       2.43.0
> >
> >
>
>

[-- Attachment #2: Type: text/html, Size: 17531 bytes --]

RE: [RFC 0/7] drm/virtio: Import scanout buffers from other devices

[Kasireddy, Vivek]

Hi Gurchetan,

> 
> On Fri, May 24, 2024 at 11:33 AM Kasireddy, Vivek
> <vivek.kasireddy@intel.com <mailto:vivek.kasireddy@intel.com> > wrote:
> 
> 
> 	Hi,
> 
> 	Sorry, my previous reply got messed up as a result of HTML
> formatting. This is
> 	a plain text version of the same reply.
> 
> 	>
> 	>
> 	>       Having virtio-gpu import scanout buffers (via prime) from other
> 	>       devices means that we'd be adding a head to headless GPUs
> assigned
> 	>       to a Guest VM or additional heads to regular GPU devices that
> are
> 	>       passthrough'd to the Guest. In these cases, the Guest
> compositor
> 	>       can render into the scanout buffer using a primary GPU and has
> the
> 	>       secondary GPU (virtio-gpu) import it for display purposes.
> 	>
> 	>       The main advantage with this is that the imported scanout
> buffer can
> 	>       either be displayed locally on the Host (e.g, using Qemu + GTK
> UI)
> 	>       or encoded and streamed to a remote client (e.g, Qemu + Spice
> UI).
> 	>       Note that since Qemu uses udmabuf driver, there would be no
> 	> copies
> 	>       made of the scanout buffer as it is displayed. This should be
> 	>       possible even when it might reside in device memory such has
> 	> VRAM.
> 	>
> 	>       The specific use-case that can be supported with this series is
> when
> 	>       running Weston or other guest compositors with "additional-
> devices"
> 	>       feature (./weston --drm-device=card1 --additional-
> devices=card0).
> 	>       More info about this feature can be found at:
> 	>       https://gitlab.freedesktop.org/wayland/weston/-
> 	> /merge_requests/736
> 	>
> 	>       In the above scenario, card1 could be a dGPU or an iGPU and
> card0
> 	>       would be virtio-gpu in KMS only mode. However, the case
> where this
> 	>       patch series could be particularly useful is when card1 is a GPU
> VF
> 	>       that needs to share its scanout buffer (in a zero-copy way) with
> the
> 	>       GPU PF on the Host. Or, it can also be useful when the scanout
> buffer
> 	>       needs to be shared between any two GPU devices (assuming
> one of
> 	> them
> 	>       is assigned to a Guest VM) as long as they are P2P DMA
> compatible.
> 	>
> 	>
> 	>
> 	> Is passthrough iGPU-only or passthrough dGPU-only something you
> intend to
> 	> use?
> 	Our main use-case involves passthrough’g a headless dGPU VF device
> and sharing
> 	the Guest compositor’s scanout buffer with dGPU PF device on the
> Host. Same goal for
> 	headless iGPU VF to iGPU PF device as well.
> 
> 
> 
> Just to check my understanding: the same physical {i, d}GPU is partitioned
> into the VF and PF, but the PF handles host-side display integration and
> rendering?
Yes, that is mostly right. In a nutshell, the same physical GPU is partitioned
into one PF device and multiple VF devices. Only the PF device has access to
the display hardware and can do KMS (on the Host). The VF devices are
headless with no access to display hardware (cannot do KMS but can do render/
encode/decode) and are generally assigned (or passthrough'd) to the Guest VMs.
Some more details about this model can be found here:
https://lore.kernel.org/dri-devel/20231110182231.1730-1-michal.wajdeczko@intel.com/

> 
> 
> 	However, using a combination of iGPU and dGPU where either of
> them can be passthrough’d
> 	to the Guest is something I think can be supported with this patch
> series as well.
> 
> 	>
> 	> If it's a dGPU + iGPU setup, then the way other people seem to do it
> is a
> 	> "virtualized" iGPU (via virgl/gfxstream/take your pick) and pass-
> through the
> 	> dGPU.
> 	>
> 	> For example, AMD seems to use virgl to allocate and import into
> the dGPU.
> 	>
> 	> https://gitlab.freedesktop.org/mesa/mesa/-
> /merge_requests/23896
> 	>
> 	> https://lore.kernel.org/all/20231221100016.4022353-1-
> 	> julia.zhang@amd.com/ <http://julia.zhang@amd.com/>
> 	>
> 	>
> 	> ChromeOS also uses that method (see crrev.com/c/3764931
> <http://crrev.com/c/3764931>
> 	> <http://crrev.com/c/3764931> ) [cc: dGPU architect +Dominik Behr
> 	> <mailto:dbehr@google.com <mailto:dbehr@google.com> > ]
> 	>
> 	> So if iGPU + dGPU is the primary use case, you should be able to
> use these
> 	> methods as well.  The model would "virtualized iGPU" +
> passthrough dGPU,
> 	> not split SoCs.
> 	In our use-case, the goal is to have only one primary GPU
> (passthrough’d iGPU/dGPU)
> 	do all the rendering (using native DRI drivers) for clients/compositor
> and all the outputs
> 	and share the scanout buffers with the secondary GPU (virtio-gpu).
> Since this is mostly
> 	how Mutter (and also Weston) work in a multi-GPU setup, I am not
> sure if virgl is needed.
> 
> 
> 
> I think you can probably use virgl with the PF and others probably will, but
> supporting multiple methods in Linux is not unheard of.
In our case, we have an alternative SR-IOV based GPU virtualization/partitioning
model (as described above) where a Guest VM will have access to a hardware-accelerated
GPU VF device for its rendering/encode/decode needs. So, in this situation, using
virgl will become redundant and unnecessary.

And, in this model, we intend to use virtio-gpu for KMS in the Guest VM (since the
GPU VF device cannot do KMS) with the addition of this patchset. However, note that,
since not all GPU SKUs/versions have the SRIOV capability, we plan on using virgl in
those cases where it becomes necessary.

> 
> Does your patchset need the Mesa kmsro patchset to function correctly?
> 
> https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9592
This patchset is an alternative proposal. So, KMSRO would not be needed.
AFAICS, the above MR is mainly stalled because KMSRO uses dumb buffers
which are not suitable for hardware-based rendering in all cases. And, KMSRO
is not really helpful performance-wise with dGPUs, as it forces most buffers to
be allocated from system memory.

> 
> 
> If so, I would try to get that reviewed first to meet DRM requirements
> (https://dri.freedesktop.org/docs/drm/gpu/drm-uapi.html#open-source-
> userspace-requirements).  You might explicitly call out the design decision
> you're making: ("We can probably use virgl as the virtualized iGPU via PF, but
> that adds unnecessary complexity b/c ______").
As I described above, what we have is an alternative GPU virtualization scheme
where virgl is not necessary if SRIOV capability is available. And, as mentioned
earlier, I have tested this series with Mutter/Gnome-shell (upstream master)
(plus one small patch: https://gitlab.gnome.org/GNOME/mutter/-/merge_requests/3745)
and no other changes to any other userspace components on Host and Guest.

> 
> 
> 	And, doing it this way means that no other userspace components
> need to be modified
> 	on both the Guest and the Host.
> 
> 	>
> 	>
> 	>
> 	>       As part of the import, the virtio-gpu driver shares the dma
> 	>       addresses and lengths with Qemu which then determines
> whether
> 	> the
> 	>       memory region they belong to is owned by a PCI device or
> whether it
> 	>       is part of the Guest's system ram. If it is the former, it identifies
> 	>       the devid (or bdf) and bar and provides this info (along with
> offsets
> 	>       and sizes) to the udmabuf driver. In the latter case, instead of
> the
> 	>       the devid and bar it provides the memfd. The udmabuf driver
> then
> 	>       creates a dmabuf using this info that Qemu shares with Spice
> for
> 	>       encode via Gstreamer.
> 	>
> 	>       Note that the virtio-gpu driver registers a move_notify() callback
> 	>       to track location changes associated with the scanout buffer and
> 	>       sends attach/detach backing cmds to Qemu when appropriate.
> And,
> 	>       synchronization (that is, ensuring that Guest and Host are not
> 	>       using the scanout buffer at the same time) is ensured by
> pinning/
> 	>       unpinning the dmabuf as part of plane update and using a fence
> 	>       in resource_flush cmd.
> 	>
> 	>
> 	> I'm not sure how QEMU's display paths work, but with crosvm if
> you share
> 	> the guest-created dmabuf with the display, and the guest moves
> the backing
> 	> pages, the only recourse is the destroy the surface and show a
> black screen
> 	> to the user: not the best thing experience wise.
> 	Since Qemu GTK UI uses EGL, there is a blit done from the guest’s
> scanout buffer onto an EGL
> 	backed buffer on the Host. So, this problem would not happen as of
> now.
> 
> 
> 
> The guest kernel doesn't know you're using the QEMU GTK UI + EGL host-
> side.
So, with blob=true, there is a dma fence in resource_flush() that gets associated
with the Blit/Encode on the Host. This guest dma fence should eventually be signalled
only when the Host is done using guest's scanout buffer.

> 
> If somebody wants to use the virtio-gpu import mechanism with lower-level
> Wayland-based display integration, then the problem would occur.
Right, one way to address this issue is to prevent the Guest compositor from
reusing the scanout buffer (until the Host is done) and forcing it to pick a new
buffer (since Mesa GBM allows 4 backbuffers). 
I have tried this experiment with KMSRO and Wayland-based Qemu UI previously
on iGPUs (and Weston) and noticed that the Guest FPS was getting halved:
https://lore.kernel.org/qemu-devel/20210913222036.3193732-1-vivek.kasireddy@intel.com/

and also discussed and proposed a solution which did not go anywhere:
https://lore.kernel.org/dri-devel/20210913233529.3194401-1-vivek.kasireddy@intel.com/

> 
> Perhaps, do that just to be safe unless you have performance concerns.
If you meant pinning the imported scanout buffer in the Guest, then yes,
that is something I am already doing in this patchset.

> 
> 
> 	>
> 	> Only amdgpu calls dma_buf_move_notfiy(..), and you're probably
> testing on
> 	> Intel only, so you may not be hitting that code path anyways.
> 	I have tested with the Xe driver in the Guest which also calls
> dma_buf_move_notfiy(). But
> 	note that for dGPUs, both Xe and amdgpu migrate the scanout buffer
> from vram to system
> 	memory as part of export, because virtio-gpu is not P2P compatible.
> However, I am hoping
> 	to relax this (p2p check against virtio-gpu) in Xe driver if it detects
> that it is running in
> 	VF mode once the following patch series is merged:
> 	https://lore.kernel.org/dri-devel/20240422063602.3690124-1-
> vivek.kasireddy@intel.com/
> 
> 	> I forgot the
> 	> exact reason, but apparently udmabuf may not work with amdgpu
> displays
> 	> and it seems the virtualized iGPU + dGPU is the way to go for
> amdgpu
> 	> anyways.
> 	I would really like to know why udmabuf would not work with
> amdgpu?
> 
> 
> 
> It's just a rumor I heard, but the idea is udmabuf would be imported into
> AMDGPU_GEM_DOMAIN_CPU only.
> 
> https://cgit.freedesktop.org/drm/drm-
> misc/tree/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c#n333
> 
> "AMDGPU_GEM_DOMAIN_CPU: System memory that is not GPU accessible.
> Memory in this pool could be swapped out to disk if there is pressure."
> 
> https://dri.freedesktop.org/docs/drm/gpu/amdgpu.html
> 
> 
> Perhaps that limitation is artificial and unnecessary, and it may indeed work.
> I don't think anybody has tried...
Since udmabuf driver properly pins the backing pages (from memfd) for DMA,
I don't see any reason why amdgpu would not be able to import.

Thanks,
Vivek

> 
> 
> 
> 
> 	> So I recommend just pinning the buffer for the lifetime of the
> 	> import for simplicity and correctness.
> 	Yeah, in this patch series, the dmabuf is indeed pinned, but only for a
> short duration in the Guest –
> 	just until the Host is done using it (blit or encode).
> 
> 	Thanks,
> 	Vivek
> 
> 	>
> 	>
> 	>       This series is available at:
> 	>       https://gitlab.freedesktop.org/Vivek/drm-tip/-
> 	> /commits/virtgpu_import_rfc
> 	>
> 	>       along with additional patches for Qemu and Spice here:
> 	>       https://gitlab.freedesktop.org/Vivek/qemu/-
> 	> /commits/virtgpu_dmabuf_pcidev
> 	>       https://gitlab.freedesktop.org/Vivek/spice/-
> 	> /commits/encode_dmabuf_v4
> 	>
> 	>       Patchset overview:
> 	>
> 	>       Patch 1:   Implement
> 	> VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING cmd
> 	>       Patch 2-3: Helpers to initalize, import, free imported object
> 	>       Patch 4-5: Import and use buffers from other devices for
> scanout
> 	>       Patch 6-7: Have udmabuf driver create dmabuf from PCI bars
> for P2P
> 	> DMA
> 	>
> 	>       This series is tested using the following method:
> 	>       - Run Qemu with the following relevant options:
> 	>         qemu-system-x86_64 -m 4096m ....
> 	>         -device vfio-pci,host=0000:03:00.0
> 	>         -device virtio-
> vga,max_outputs=1,blob=true,xres=1920,yres=1080
> 	>         -spice port=3001,gl=on,disable-ticketing=on,preferred-
> 	> codec=gstreamer:h264
> 	>         -object memory-backend-memfd,id=mem1,size=4096M
> 	>         -machine memory-backend=mem1 ...
> 	>       - Run upstream Weston with the following options in the Guest
> VM:
> 	>         ./weston --drm-device=card1 --additional-devices=card0
> 	>
> 	>       where card1 is a DG2 dGPU (passthrough'd and using xe driver
> in
> 	> Guest VM),
> 	>       card0 is virtio-gpu and the Host is using a RPL iGPU.
> 	>
> 	>       Cc: Gerd Hoffmann <kraxel@redhat.com
> <mailto:kraxel@redhat.com>
> 	> <mailto:kraxel@redhat.com <mailto:kraxel@redhat.com> > >
> 	>       Cc: Dongwon Kim <dongwon.kim@intel.com
> <mailto:dongwon.kim@intel.com>
> 	> <mailto:dongwon.kim@intel.com
> <mailto:dongwon.kim@intel.com> > >
> 	>       Cc: Daniel Vetter <daniel.vetter@ffwll.ch
> <mailto:daniel.vetter@ffwll.ch>
> 	> <mailto:daniel.vetter@ffwll.ch <mailto:daniel.vetter@ffwll.ch> > >
> 	>       Cc: Christian Koenig <christian.koenig@amd.com
> <mailto:christian.koenig@amd.com>
> 	> <mailto:christian.koenig@amd.com
> <mailto:christian.koenig@amd.com> > >
> 	>       Cc: Dmitry Osipenko <dmitry.osipenko@collabora.com
> <mailto:dmitry.osipenko@collabora.com>
> 	> <mailto:dmitry.osipenko@collabora.com
> <mailto:dmitry.osipenko@collabora.com> > >
> 	>       Cc: Rob Clark <robdclark@chromium.org
> <mailto:robdclark@chromium.org>
> 	> <mailto:robdclark@chromium.org
> <mailto:robdclark@chromium.org> > >
> 	>       Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com
> <mailto:thomas.hellstrom@linux.intel.com>
> 	> <mailto:thomas.hellstrom@linux.intel.com
> <mailto:thomas.hellstrom@linux.intel.com> > >
> 	>       Cc: Oded Gabbay <ogabbay@kernel.org
> <mailto:ogabbay@kernel.org>
> 	> <mailto:ogabbay@kernel.org <mailto:ogabbay@kernel.org> > >
> 	>       Cc: Michal Wajdeczko <michal.wajdeczko@intel.com
> <mailto:michal.wajdeczko@intel.com>
> 	> <mailto:michal.wajdeczko@intel.com
> <mailto:michal.wajdeczko@intel.com> > >
> 	>       Cc: Michael Tretter <m.tretter@pengutronix.de
> <mailto:m.tretter@pengutronix.de>
> 	> <mailto:m.tretter@pengutronix.de
> <mailto:m.tretter@pengutronix.de> > >
> 	>
> 	>       Vivek Kasireddy (7):
> 	>         drm/virtio: Implement
> 	> VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING cmd
> 	>         drm/virtio: Add a helper to map and note the dma addrs and
> 	> lengths
> 	>         drm/virtio: Add helpers to initialize and free the imported
> object
> 	>         drm/virtio: Import prime buffers from other devices as guest
> blobs
> 	>         drm/virtio: Ensure that bo's backing store is valid while
> updating
> 	>           plane
> 	>         udmabuf/uapi: Add new ioctl to create a dmabuf from PCI bar
> 	> regions
> 	>         udmabuf: Implement UDMABUF_CREATE_LIST_FOR_PCIDEV
> ioctl
> 	>
> 	>        drivers/dma-buf/udmabuf.c              | 122 ++++++++++++++++--
> 	>        drivers/gpu/drm/virtio/virtgpu_drv.h   |   8 ++
> 	>        drivers/gpu/drm/virtio/virtgpu_plane.c |  56 ++++++++-
> 	>        drivers/gpu/drm/virtio/virtgpu_prime.c | 167
> 	> ++++++++++++++++++++++++-
> 	>        drivers/gpu/drm/virtio/virtgpu_vq.c    |  15 +++
> 	>        include/uapi/linux/udmabuf.h           |  11 +-
> 	>        6 files changed, 368 insertions(+), 11 deletions(-)
> 	>
> 	>       --
> 	>       2.43.0
> 	>
> 	>
> 
> 

Re: [RFC 0/7] drm/virtio: Import scanout buffers from other devices

[Gurchetan Singh]

[-- Attachment #1: Type: text/plain, Size: 20021 bytes --]

On Thu, May 30, 2024 at 12:21 AM Kasireddy, Vivek <vivek.kasireddy@intel.com>
wrote:

> Hi Gurchetan,
>
> >
> > On Fri, May 24, 2024 at 11:33 AM Kasireddy, Vivek
> > <vivek.kasireddy@intel.com <mailto:vivek.kasireddy@intel.com> > wrote:
> >
> >
> >       Hi,
> >
> >       Sorry, my previous reply got messed up as a result of HTML
> > formatting. This is
> >       a plain text version of the same reply.
> >
> >       >
> >       >
> >       >       Having virtio-gpu import scanout buffers (via prime) from
> other
> >       >       devices means that we'd be adding a head to headless GPUs
> > assigned
> >       >       to a Guest VM or additional heads to regular GPU devices
> that
> > are
> >       >       passthrough'd to the Guest. In these cases, the Guest
> > compositor
> >       >       can render into the scanout buffer using a primary GPU and
> has
> > the
> >       >       secondary GPU (virtio-gpu) import it for display purposes.
> >       >
> >       >       The main advantage with this is that the imported scanout
> > buffer can
> >       >       either be displayed locally on the Host (e.g, using Qemu +
> GTK
> > UI)
> >       >       or encoded and streamed to a remote client (e.g, Qemu +
> Spice
> > UI).
> >       >       Note that since Qemu uses udmabuf driver, there would be no
> >       > copies
> >       >       made of the scanout buffer as it is displayed. This should
> be
> >       >       possible even when it might reside in device memory such
> has
> >       > VRAM.
> >       >
> >       >       The specific use-case that can be supported with this
> series is
> > when
> >       >       running Weston or other guest compositors with "additional-
> > devices"
> >       >       feature (./weston --drm-device=card1 --additional-
> > devices=card0).
> >       >       More info about this feature can be found at:
> >       >       https://gitlab.freedesktop.org/wayland/weston/-
> >       > /merge_requests/736
> >       >
> >       >       In the above scenario, card1 could be a dGPU or an iGPU and
> > card0
> >       >       would be virtio-gpu in KMS only mode. However, the case
> > where this
> >       >       patch series could be particularly useful is when card1 is
> a GPU
> > VF
> >       >       that needs to share its scanout buffer (in a zero-copy
> way) with
> > the
> >       >       GPU PF on the Host. Or, it can also be useful when the
> scanout
> > buffer
> >       >       needs to be shared between any two GPU devices (assuming
> > one of
> >       > them
> >       >       is assigned to a Guest VM) as long as they are P2P DMA
> > compatible.
> >       >
> >       >
> >       >
> >       > Is passthrough iGPU-only or passthrough dGPU-only something you
> > intend to
> >       > use?
> >       Our main use-case involves passthrough’g a headless dGPU VF device
> > and sharing
> >       the Guest compositor’s scanout buffer with dGPU PF device on the
> > Host. Same goal for
> >       headless iGPU VF to iGPU PF device as well.
> >
> >
> >
> > Just to check my understanding: the same physical {i, d}GPU is
> partitioned
> > into the VF and PF, but the PF handles host-side display integration and
> > rendering?
> Yes, that is mostly right. In a nutshell, the same physical GPU is
> partitioned
> into one PF device and multiple VF devices. Only the PF device has access
> to
> the display hardware and can do KMS (on the Host). The VF devices are
> headless with no access to display hardware (cannot do KMS but can do
> render/
> encode/decode) and are generally assigned (or passthrough'd) to the Guest
> VMs.
> Some more details about this model can be found here:
>
> https://lore.kernel.org/dri-devel/20231110182231.1730-1-michal.wajdeczko@intel.com/
>
> >
> >
> >       However, using a combination of iGPU and dGPU where either of
> > them can be passthrough’d
> >       to the Guest is something I think can be supported with this patch
> > series as well.
> >
> >       >
> >       > If it's a dGPU + iGPU setup, then the way other people seem to
> do it
> > is a
> >       > "virtualized" iGPU (via virgl/gfxstream/take your pick) and pass-
> > through the
> >       > dGPU.
> >       >
> >       > For example, AMD seems to use virgl to allocate and import into
> > the dGPU.
> >       >
> >       > https://gitlab.freedesktop.org/mesa/mesa/-
> > /merge_requests/23896
> >       >
> >       > https://lore.kernel.org/all/20231221100016.4022353-1-
> >       > julia.zhang@amd.com/ <http://julia.zhang@amd.com/>
> >       >
> >       >
> >       > ChromeOS also uses that method (see crrev.com/c/3764931
> > <http://crrev.com/c/3764931>
> >       > <http://crrev.com/c/3764931> ) [cc: dGPU architect +Dominik Behr
> >       > <mailto:dbehr@google.com <mailto:dbehr@google.com> > ]
> >       >
> >       > So if iGPU + dGPU is the primary use case, you should be able to
> > use these
> >       > methods as well.  The model would "virtualized iGPU" +
> > passthrough dGPU,
> >       > not split SoCs.
> >       In our use-case, the goal is to have only one primary GPU
> > (passthrough’d iGPU/dGPU)
> >       do all the rendering (using native DRI drivers) for
> clients/compositor
> > and all the outputs
> >       and share the scanout buffers with the secondary GPU (virtio-gpu).
> > Since this is mostly
> >       how Mutter (and also Weston) work in a multi-GPU setup, I am not
> > sure if virgl is needed.
> >
> >
> >
> > I think you can probably use virgl with the PF and others probably will,
> but
> > supporting multiple methods in Linux is not unheard of.
> In our case, we have an alternative SR-IOV based GPU
> virtualization/partitioning
> model (as described above) where a Guest VM will have access to a
> hardware-accelerated
> GPU VF device for its rendering/encode/decode needs. So, in this
> situation, using
> virgl will become redundant and unnecessary.
>
> And, in this model, we intend to use virtio-gpu for KMS in the Guest VM
> (since the
> GPU VF device cannot do KMS) with the addition of this patchset. However,
> note that,
> since not all GPU SKUs/versions have the SRIOV capability, we plan on
> using virgl in
> those cases where it becomes necessary.
>
> >
> > Does your patchset need the Mesa kmsro patchset to function correctly?
> >
> > https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9592
> This patchset is an alternative proposal. So, KMSRO would not be needed.
> AFAICS, the above MR is mainly stalled because KMSRO uses dumb buffers
> which are not suitable for hardware-based rendering in all cases. And,
> KMSRO
> is not really helpful performance-wise with dGPUs, as it forces most
> buffers to
> be allocated from system memory.
>

Previously, it was recommended when exploring VDMABUF: "the /important/
thing is that the driver which exports the dma-buf (and thus handles the
mappings) must be aware of the virtualization so it can properly coordinate
things with the host side."

https://patchwork.kernel.org/project/linux-media/patch/20210203073517.1908882-3-vivek.kasireddy@intel.com/#23975915


So that's why the KMSRO approach was tried  (virtio-gpu dumb allocations,
not i915).  But as you point out, nobody uses dumb buffers for
hardware-based rendering.

So, if you are going with i915 allocates + virtio-gpu imports,  it should
be fine if you fixed all the issues with i915 allocates + VDMABUF imports.
It seems your fixes add complexity in VFIO and other places, but having
virtio-gpu 3d + virgl allocate adds complexity to Mesa-based allocation
paths (i915, amdgpu would all have to open virtio-gpu render node, and pick
a context type etc.).

I would just do virtio-gpu allocates, since it only requires user-space
patches and no extra ioctls, but that reflects my preferences.  If the
mm/VFIO/QEMU people are fine with your approach, I see nothing wrong with
merging it.

The one caveat is if someone uses a non-GTK/EGL host path, we'll have to
pin memory for the lifetime of the import, since knowing RESOURCE_FLUSH is
done is not sufficient.  But if you're only using it, it shouldn't be an
issue right now.


>
> >
> >
> > If so, I would try to get that reviewed first to meet DRM requirements
> > (https://dri.freedesktop.org/docs/drm/gpu/drm-uapi.html#open-source-
> > userspace-requirements).  You might explicitly call out the design
> decision
> > you're making: ("We can probably use virgl as the virtualized iGPU via
> PF, but
> > that adds unnecessary complexity b/c ______").
> As I described above, what we have is an alternative GPU virtualization
> scheme
> where virgl is not necessary if SRIOV capability is available. And, as
> mentioned
> earlier, I have tested this series with Mutter/Gnome-shell (upstream
> master)
> (plus one small patch:
> https://gitlab.gnome.org/GNOME/mutter/-/merge_requests/3745)
> and no other changes to any other userspace components on Host and Guest.
>
> >
> >
> >       And, doing it this way means that no other userspace components
> > need to be modified
> >       on both the Guest and the Host.
> >
> >       >
> >       >
> >       >
> >       >       As part of the import, the virtio-gpu driver shares the dma
> >       >       addresses and lengths with Qemu which then determines
> > whether
> >       > the
> >       >       memory region they belong to is owned by a PCI device or
> > whether it
> >       >       is part of the Guest's system ram. If it is the former, it
> identifies
> >       >       the devid (or bdf) and bar and provides this info (along
> with
> > offsets
> >       >       and sizes) to the udmabuf driver. In the latter case,
> instead of
> > the
> >       >       the devid and bar it provides the memfd. The udmabuf driver
> > then
> >       >       creates a dmabuf using this info that Qemu shares with
> Spice
> > for
> >       >       encode via Gstreamer.
> >       >
> >       >       Note that the virtio-gpu driver registers a move_notify()
> callback
> >       >       to track location changes associated with the scanout
> buffer and
> >       >       sends attach/detach backing cmds to Qemu when appropriate.
> > And,
> >       >       synchronization (that is, ensuring that Guest and Host are
> not
> >       >       using the scanout buffer at the same time) is ensured by
> > pinning/
> >       >       unpinning the dmabuf as part of plane update and using a
> fence
> >       >       in resource_flush cmd.
> >       >
> >       >
> >       > I'm not sure how QEMU's display paths work, but with crosvm if
> > you share
> >       > the guest-created dmabuf with the display, and the guest moves
> > the backing
> >       > pages, the only recourse is the destroy the surface and show a
> > black screen
> >       > to the user: not the best thing experience wise.
> >       Since Qemu GTK UI uses EGL, there is a blit done from the guest’s
> > scanout buffer onto an EGL
> >       backed buffer on the Host. So, this problem would not happen as of
> > now.
> >
> >
> >
> > The guest kernel doesn't know you're using the QEMU GTK UI + EGL host-
> > side.
> So, with blob=true, there is a dma fence in resource_flush() that gets
> associated
> with the Blit/Encode on the Host. This guest dma fence should eventually
> be signalled
> only when the Host is done using guest's scanout buffer.
>
> >
> > If somebody wants to use the virtio-gpu import mechanism with lower-level
> > Wayland-based display integration, then the problem would occur.
> Right, one way to address this issue is to prevent the Guest compositor
> from
> reusing the scanout buffer (until the Host is done) and forcing it to pick
> a new
> buffer (since Mesa GBM allows 4 backbuffers).
> I have tried this experiment with KMSRO and Wayland-based Qemu UI
> previously
> on iGPUs (and Weston) and noticed that the Guest FPS was getting halved:
>
> https://lore.kernel.org/qemu-devel/20210913222036.3193732-1-vivek.kasireddy@intel.com/
>
> and also discussed and proposed a solution which did not go anywhere:
>
> https://lore.kernel.org/dri-devel/20210913233529.3194401-1-vivek.kasireddy@intel.com/
>
> >
> > Perhaps, do that just to be safe unless you have performance concerns.
> If you meant pinning the imported scanout buffer in the Guest, then yes,
> that is something I am already doing in this patchset.
>
> >
> >
> >       >
> >       > Only amdgpu calls dma_buf_move_notfiy(..), and you're probably
> > testing on
> >       > Intel only, so you may not be hitting that code path anyways.
> >       I have tested with the Xe driver in the Guest which also calls
> > dma_buf_move_notfiy(). But
> >       note that for dGPUs, both Xe and amdgpu migrate the scanout buffer
> > from vram to system
> >       memory as part of export, because virtio-gpu is not P2P compatible.
> > However, I am hoping
> >       to relax this (p2p check against virtio-gpu) in Xe driver if it
> detects
> > that it is running in
> >       VF mode once the following patch series is merged:
> >       https://lore.kernel.org/dri-devel/20240422063602.3690124-1-
> > vivek.kasireddy@intel.com/
> >
> >       > I forgot the
> >       > exact reason, but apparently udmabuf may not work with amdgpu
> > displays
> >       > and it seems the virtualized iGPU + dGPU is the way to go for
> > amdgpu
> >       > anyways.
> >       I would really like to know why udmabuf would not work with
> > amdgpu?
> >
> >
> >
> > It's just a rumor I heard, but the idea is udmabuf would be imported into
> > AMDGPU_GEM_DOMAIN_CPU only.
> >
> > https://cgit.freedesktop.org/drm/drm-
> > misc/tree/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c#n333
> >
> > "AMDGPU_GEM_DOMAIN_CPU: System memory that is not GPU accessible.
> > Memory in this pool could be swapped out to disk if there is pressure."
> >
> > https://dri.freedesktop.org/docs/drm/gpu/amdgpu.html
> >
> >
> > Perhaps that limitation is artificial and unnecessary, and it may indeed
> work.
> > I don't think anybody has tried...
> Since udmabuf driver properly pins the backing pages (from memfd) for DMA,
> I don't see any reason why amdgpu would not be able to import.
>
> Thanks,
> Vivek
>
> >
> >
> >
> >
> >       > So I recommend just pinning the buffer for the lifetime of the
> >       > import for simplicity and correctness.
> >       Yeah, in this patch series, the dmabuf is indeed pinned, but only
> for a
> > short duration in the Guest –
> >       just until the Host is done using it (blit or encode).
> >
> >       Thanks,
> >       Vivek
> >
> >       >
> >       >
> >       >       This series is available at:
> >       >       https://gitlab.freedesktop.org/Vivek/drm-tip/-
> >       > /commits/virtgpu_import_rfc
> >       >
> >       >       along with additional patches for Qemu and Spice here:
> >       >       https://gitlab.freedesktop.org/Vivek/qemu/-
> >       > /commits/virtgpu_dmabuf_pcidev
> >       >       https://gitlab.freedesktop.org/Vivek/spice/-
> >       > /commits/encode_dmabuf_v4
> >       >
> >       >       Patchset overview:
> >       >
> >       >       Patch 1:   Implement
> >       > VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING cmd
> >       >       Patch 2-3: Helpers to initalize, import, free imported
> object
> >       >       Patch 4-5: Import and use buffers from other devices for
> > scanout
> >       >       Patch 6-7: Have udmabuf driver create dmabuf from PCI bars
> > for P2P
> >       > DMA
> >       >
> >       >       This series is tested using the following method:
> >       >       - Run Qemu with the following relevant options:
> >       >         qemu-system-x86_64 -m 4096m ....
> >       >         -device vfio-pci,host=0000:03:00.0
> >       >         -device virtio-
> > vga,max_outputs=1,blob=true,xres=1920,yres=1080
> >       >         -spice port=3001,gl=on,disable-ticketing=on,preferred-
> >       > codec=gstreamer:h264
> >       >         -object memory-backend-memfd,id=mem1,size=4096M
> >       >         -machine memory-backend=mem1 ...
> >       >       - Run upstream Weston with the following options in the
> Guest
> > VM:
> >       >         ./weston --drm-device=card1 --additional-devices=card0
> >       >
> >       >       where card1 is a DG2 dGPU (passthrough'd and using xe
> driver
> > in
> >       > Guest VM),
> >       >       card0 is virtio-gpu and the Host is using a RPL iGPU.
> >       >
> >       >       Cc: Gerd Hoffmann <kraxel@redhat.com
> > <mailto:kraxel@redhat.com>
> >       > <mailto:kraxel@redhat.com <mailto:kraxel@redhat.com> > >
> >       >       Cc: Dongwon Kim <dongwon.kim@intel.com
> > <mailto:dongwon.kim@intel.com>
> >       > <mailto:dongwon.kim@intel.com
> > <mailto:dongwon.kim@intel.com> > >
> >       >       Cc: Daniel Vetter <daniel.vetter@ffwll.ch
> > <mailto:daniel.vetter@ffwll.ch>
> >       > <mailto:daniel.vetter@ffwll.ch <mailto:daniel.vetter@ffwll.ch>
> > >
> >       >       Cc: Christian Koenig <christian.koenig@amd.com
> > <mailto:christian.koenig@amd.com>
> >       > <mailto:christian.koenig@amd.com
> > <mailto:christian.koenig@amd.com> > >
> >       >       Cc: Dmitry Osipenko <dmitry.osipenko@collabora.com
> > <mailto:dmitry.osipenko@collabora.com>
> >       > <mailto:dmitry.osipenko@collabora.com
> > <mailto:dmitry.osipenko@collabora.com> > >
> >       >       Cc: Rob Clark <robdclark@chromium.org
> > <mailto:robdclark@chromium.org>
> >       > <mailto:robdclark@chromium.org
> > <mailto:robdclark@chromium.org> > >
> >       >       Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com
> > <mailto:thomas.hellstrom@linux.intel.com>
> >       > <mailto:thomas.hellstrom@linux.intel.com
> > <mailto:thomas.hellstrom@linux.intel.com> > >
> >       >       Cc: Oded Gabbay <ogabbay@kernel.org
> > <mailto:ogabbay@kernel.org>
> >       > <mailto:ogabbay@kernel.org <mailto:ogabbay@kernel.org> > >
> >       >       Cc: Michal Wajdeczko <michal.wajdeczko@intel.com
> > <mailto:michal.wajdeczko@intel.com>
> >       > <mailto:michal.wajdeczko@intel.com
> > <mailto:michal.wajdeczko@intel.com> > >
> >       >       Cc: Michael Tretter <m.tretter@pengutronix.de
> > <mailto:m.tretter@pengutronix.de>
> >       > <mailto:m.tretter@pengutronix.de
> > <mailto:m.tretter@pengutronix.de> > >
> >       >
> >       >       Vivek Kasireddy (7):
> >       >         drm/virtio: Implement
> >       > VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING cmd
> >       >         drm/virtio: Add a helper to map and note the dma addrs
> and
> >       > lengths
> >       >         drm/virtio: Add helpers to initialize and free the
> imported
> > object
> >       >         drm/virtio: Import prime buffers from other devices as
> guest
> > blobs
> >       >         drm/virtio: Ensure that bo's backing store is valid while
> > updating
> >       >           plane
> >       >         udmabuf/uapi: Add new ioctl to create a dmabuf from PCI
> bar
> >       > regions
> >       >         udmabuf: Implement UDMABUF_CREATE_LIST_FOR_PCIDEV
> > ioctl
> >       >
> >       >        drivers/dma-buf/udmabuf.c              | 122
> ++++++++++++++++--
> >       >        drivers/gpu/drm/virtio/virtgpu_drv.h   |   8 ++
> >       >        drivers/gpu/drm/virtio/virtgpu_plane.c |  56 ++++++++-
> >       >        drivers/gpu/drm/virtio/virtgpu_prime.c | 167
> >       > ++++++++++++++++++++++++-
> >       >        drivers/gpu/drm/virtio/virtgpu_vq.c    |  15 +++
> >       >        include/uapi/linux/udmabuf.h           |  11 +-
> >       >        6 files changed, 368 insertions(+), 11 deletions(-)
> >       >
> >       >       --
> >       >       2.43.0
> >       >
> >       >
> >
> >
>
>

[-- Attachment #2: Type: text/html, Size: 29530 bytes --]

RE: [RFC 0/7] drm/virtio: Import scanout buffers from other devices

[Kasireddy, Vivek]

Hi Gurchetan,

> 
> On Thu, May 30, 2024 at 12:21 AM Kasireddy, Vivek
> <vivek.kasireddy@intel.com <mailto:vivek.kasireddy@intel.com> > wrote:
> 
> 
> 	Hi Gurchetan,
> 
> 	>
> 	> On Fri, May 24, 2024 at 11:33 AM Kasireddy, Vivek
> 	> <vivek.kasireddy@intel.com <mailto:vivek.kasireddy@intel.com>
> <mailto:vivek.kasireddy@intel.com <mailto:vivek.kasireddy@intel.com> > >
> wrote:
> 	>
> 	>
> 	>       Hi,
> 	>
> 	>       Sorry, my previous reply got messed up as a result of HTML
> 	> formatting. This is
> 	>       a plain text version of the same reply.
> 	>
> 	>       >
> 	>       >
> 	>       >       Having virtio-gpu import scanout buffers (via prime) from
> other
> 	>       >       devices means that we'd be adding a head to headless
> GPUs
> 	> assigned
> 	>       >       to a Guest VM or additional heads to regular GPU devices
> that
> 	> are
> 	>       >       passthrough'd to the Guest. In these cases, the Guest
> 	> compositor
> 	>       >       can render into the scanout buffer using a primary GPU
> and has
> 	> the
> 	>       >       secondary GPU (virtio-gpu) import it for display purposes.
> 	>       >
> 	>       >       The main advantage with this is that the imported scanout
> 	> buffer can
> 	>       >       either be displayed locally on the Host (e.g, using Qemu +
> GTK
> 	> UI)
> 	>       >       or encoded and streamed to a remote client (e.g, Qemu +
> Spice
> 	> UI).
> 	>       >       Note that since Qemu uses udmabuf driver, there would
> be no
> 	>       > copies
> 	>       >       made of the scanout buffer as it is displayed. This should
> be
> 	>       >       possible even when it might reside in device memory such
> has
> 	>       > VRAM.
> 	>       >
> 	>       >       The specific use-case that can be supported with this series
> is
> 	> when
> 	>       >       running Weston or other guest compositors with
> "additional-
> 	> devices"
> 	>       >       feature (./weston --drm-device=card1 --additional-
> 	> devices=card0).
> 	>       >       More info about this feature can be found at:
> 	>       >       https://gitlab.freedesktop.org/wayland/weston/-
> 	>       > /merge_requests/736
> 	>       >
> 	>       >       In the above scenario, card1 could be a dGPU or an iGPU
> and
> 	> card0
> 	>       >       would be virtio-gpu in KMS only mode. However, the case
> 	> where this
> 	>       >       patch series could be particularly useful is when card1 is a
> GPU
> 	> VF
> 	>       >       that needs to share its scanout buffer (in a zero-copy way)
> with
> 	> the
> 	>       >       GPU PF on the Host. Or, it can also be useful when the
> scanout
> 	> buffer
> 	>       >       needs to be shared between any two GPU devices
> (assuming
> 	> one of
> 	>       > them
> 	>       >       is assigned to a Guest VM) as long as they are P2P DMA
> 	> compatible.
> 	>       >
> 	>       >
> 	>       >
> 	>       > Is passthrough iGPU-only or passthrough dGPU-only
> something you
> > intend to
> >       > use?
> >       Our main use-case involves passthrough’g a headless dGPU VF
> device
> > and sharing
> >       the Guest compositor’s scanout buffer with dGPU PF device on
> the
> > Host. Same goal for
> >       headless iGPU VF to iGPU PF device as well.
> >
> >
> >
> > Just to check my understanding: the same physical {i, d}GPU is
> partitioned
> > into the VF and PF, but the PF handles host-side display integration
> and
> > rendering?
> Yes, that is mostly right. In a nutshell, the same physical GPU is
> partitioned
> into one PF device and multiple VF devices. Only the PF device has
> access to
> the display hardware and can do KMS (on the Host). The VF devices
> are
> headless with no access to display hardware (cannot do KMS but can
> do render/
> encode/decode) and are generally assigned (or passthrough'd) to the
> Guest VMs.
> Some more details about this model can be found here:
> https://lore.kernel.org/dri-devel/20231110182231.1730-1-
> michal.wajdeczko@intel.com/
> 
> >
> >
> >       However, using a combination of iGPU and dGPU where either
> of
> > them can be passthrough’d
> >       to the Guest is something I think can be supported with this
> patch
> > series as well.
> >
> > >
> > > If it's a dGPU + iGPU setup, then the way other people seem to
> do it
> > is a
> > > "virtualized" iGPU (via virgl/gfxstream/take your pick) and
> pass-
> > through the
> > > dGPU.
> > >
> > > For example, AMD seems to use virgl to allocate and import
> into
> > the dGPU.
> > >
> > > https://gitlab.freedesktop.org/mesa/mesa/-
> > /merge_requests/23896
> > >
> > > https://lore.kernel.org/all/20231221100016.4022353-1-
> > > julia.zhang@amd.com/ <http://julia.zhang@amd.com/>
> <http://julia.zhang@amd.com/>
> > >
> > >
> > > ChromeOS also uses that method (see crrev.com/c/3764931
> <http://crrev.com/c/3764931>
> > <http://crrev.com/c/3764931>
> > > <http://crrev.com/c/3764931> ) [cc: dGPU architect +Dominik
> Behr
> >> <mailto:dbehr@google.com <mailto:dbehr@google.com>
> <mailto:dbehr@google.com <mailto:dbehr@google.com> > > ]
> > >
> > > So if iGPU + dGPU is the primary use case, you should be able
> to
> > use these
> > > methods as well.  The model would "virtualized iGPU" +
> > passthrough dGPU,
> > > not split SoCs.
> >       In our use-case, the goal is to have only one primary GPU
> > (passthrough’d iGPU/dGPU)
> >       do all the rendering (using native DRI drivers) for
> clients/compositor
> > and all the outputs
> > and share the scanout buffers with the secondary GPU (virtio-
> gpu).
> > Since this is mostly
> >       how Mutter (and also Weston) work in a multi-GPU setup, I am
> not
> > sure if virgl is needed.
> >
> >
> >
> > I think you can probably use virgl with the PF and others probably
> will, but
> > supporting multiple methods in Linux is not unheard of.
> In our case, we have an alternative SR-IOV based GPU
> virtualization/partitioning
> model (as described above) where a Guest VM will have access to a
> hardware-accelerated
> GPU VF device for its rendering/encode/decode needs. So, in this
> situation, using
> virgl will become redundant and unnecessary.
> 
> And, in this model, we intend to use virtio-gpu for KMS in the Guest
> VM (since the
> GPU VF device cannot do KMS) with the addition of this patchset.
> However, note that,
> since not all GPU SKUs/versions have the SRIOV capability, we plan
> on using virgl in
> those cases where it becomes necessary.
> 
> >
> > Does your patchset need the Mesa kmsro patchset to function
> correctly?
> >
> > https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9592
> >This patchset is an alternative proposal. So, KMSRO would not be
> >needed.
> >AFAICS, the above MR is mainly stalled because KMSRO uses dumb
> >buffers
> >which are not suitable for hardware-based rendering in all cases.
> >And, KMSRO
> >is not really helpful performance-wise with dGPUs, as it forces most
> >buffers to
> >be allocated from system memory.
> 
> Previously, it was recommended when exploring VDMABUF: "the
> /important/ thing is that the driver which exports the dma-buf (and thus
> handles the mappings) must be aware of the virtualization so it can properly
> coordinate things with the host side."
> 
> https://patchwork.kernel.org/project/linux-media/patch/20210203073517.1908882-3-
> vivek.kasireddy@intel.com/#23975915
Yeah, I remember that. It would be nice to have Gerd weigh in again because
this design is slightly different as it uses DMA addresses (and not pages) to make
it possible to share buffers allocated by Guest in both system memory and VRAM.
And, as mentioned earlier, with Qemu GTK UI (since it uses EGL), we Blit the
dmabuf to a Host allocated buffer managed by EGL.

But I do recognize that if we were to share the dmabuf directly with the Host 
compositor (like you do with CrosVM + Wayland UI), then yes, the Guest
compositor (or probably the exporter) needs to be aware of this virtualization.

> 
> So that's why the KMSRO approach was tried  (virtio-gpu dumb allocations,
> not i915).  But as you point out, nobody uses dumb buffers for hardware-
> based rendering.
Right, we have been using KMSRO + dumb buffers all this while but as we start
working more with dGPUs we realized that they are not great performance-wise.
So, one of my goals with this series (and also including the VFIO patch series) is to
ensure that (with one dGPU VF assigned to the Guest) the Guest allocated framebuffer
stays in VRAM (and does not migrated to Guest's system memory) as it is shared with
the dGPU PF on the Host.

> 
> So, if you are going with i915 allocates + virtio-gpu imports,  it should be fine
> if you fixed all the issues with i915 allocates + VDMABUF imports.  It seems
> your fixes add complexity in VFIO and other places,
For iGPUs, only this series is needed; and, the VFIO patch series is needed if
we are doing dGPU to dGPU buffer sharing (P2P DMA). And, it appears, the
VFIO patches are anyway needed for RDMA/SPDK use-cases as well.

> but having virtio-gpu 3d +
> virgl allocate adds complexity to Mesa-based allocation paths (i915, amdgpu
> would all have to open virtio-gpu render node, and pick a context type etc.).
> 
> I would just do virtio-gpu allocates, since it only requires user-space patches
> and no extra ioctls, but that reflects my preferences. 
This patch series is also not adding any extra ioctls; and no uapi changes are needed.
The only big change that needs to happen is on the Guest compositor side but both
Mutter and Weston already have the concept of primary and secondary GPUs when
dealing with multi-GPU scenarios, so they are mostly ok. 

> If the mm/VFIO/QEMU
> people are fine with your approach, I see nothing wrong with merging it.
Sima has said they are OK with the design, but they specifically asked people working
on virtio-gpu to weigh in. 

> 
> The one caveat is if someone uses a non-GTK/EGL host path, we'll have to pin
> memory for the lifetime of the import, since knowing RESOURCE_FLUSH is
Yeah, as discussed earlier, non-EGL host paths would probably need additional
signaling mechanism to achieve full frame-rate, when sharing the dmabuf directly
with the Host compositor.

> done is not sufficient.  But if you're only using it, it shouldn't be an issue right
> now.
To ensure that I am not regressing virgl or other use-cases, I am currently limiting
this feature to only cases where blob=true and virgl=false for now. I'll send a v2 soon
with some fixes related to cleanup of imported BOs.

Thanks,
Vivek

> 
> 
> 
> 	>
> 	>
> 	> If so, I would try to get that reviewed first to meet DRM
> requirements
> 	> (https://dri.freedesktop.org/docs/drm/gpu/drm-uapi.html#open-
> source-
> 	> userspace-requirements).  You might explicitly call out the design
> decision
> 	> you're making: ("We can probably use virgl as the virtualized iGPU
> via PF, but
> 	> that adds unnecessary complexity b/c ______").
> 	As I described above, what we have is an alternative GPU
> virtualization scheme
> 	where virgl is not necessary if SRIOV capability is available. And, as
> mentioned
> 	earlier, I have tested this series with Mutter/Gnome-shell (upstream
> master)
> 	(plus one small patch: https://gitlab.gnome.org/GNOME/mutter/-
> /merge_requests/3745)
> 	and no other changes to any other userspace components on Host
> and Guest.
> 
> 	>
> 	>
> 	>       And, doing it this way means that no other userspace
> components
> 	> need to be modified
> 	>       on both the Guest and the Host.
> 	>
> 	>       >
> 	>       >
> 	>       >
> 	>       >       As part of the import, the virtio-gpu driver shares the dma
> 	>       >       addresses and lengths with Qemu which then determines
> 	> whether
> 	>       > the
> 	>       >       memory region they belong to is owned by a PCI device or
> 	> whether it
> 	>       >       is part of the Guest's system ram. If it is the former, it
> identifies
> 	>       >       the devid (or bdf) and bar and provides this info (along
> with
> 	> offsets
> 	>       >       and sizes) to the udmabuf driver. In the latter case, instead
> of
> 	> the
> 	>       >       the devid and bar it provides the memfd. The udmabuf
> driver
> 	> then
> 	>       >       creates a dmabuf using this info that Qemu shares with
> Spice
> 	> for
> 	>       >       encode via Gstreamer.
> 	>       >
> 	>       >       Note that the virtio-gpu driver registers a move_notify()
> callback
> 	>       >       to track location changes associated with the scanout
> buffer and
> 	>       >       sends attach/detach backing cmds to Qemu when
> appropriate.
> 	> And,
> 	>       >       synchronization (that is, ensuring that Guest and Host are
> not
> 	>       >       using the scanout buffer at the same time) is ensured by
> 	> pinning/
> 	>       >       unpinning the dmabuf as part of plane update and using a
> fence
> 	>       >       in resource_flush cmd.
> 	>       >
> 	>       >
> 	>       > I'm not sure how QEMU's display paths work, but with crosvm
> if
> 	> you share
> 	>       > the guest-created dmabuf with the display, and the guest
> moves
> 	> the backing
> 	>       > pages, the only recourse is the destroy the surface and show a
> 	> black screen
> 	>       > to the user: not the best thing experience wise.
> 	>       Since Qemu GTK UI uses EGL, there is a blit done from the
> guest’s
> 	> scanout buffer onto an EGL
> 	>       backed buffer on the Host. So, this problem would not happen
> as of
> 	> now.
> 	>
> 	>
> 	>
> 	> The guest kernel doesn't know you're using the QEMU GTK UI + EGL
> host-
> 	> side.
> 	So, with blob=true, there is a dma fence in resource_flush() that gets
> associated
> 	with the Blit/Encode on the Host. This guest dma fence should
> eventually be signalled
> 	only when the Host is done using guest's scanout buffer.
> 
> 	>
> 	> If somebody wants to use the virtio-gpu import mechanism with
> lower-level
> 	> Wayland-based display integration, then the problem would occur.
> 	Right, one way to address this issue is to prevent the Guest
> compositor from
> 	reusing the scanout buffer (until the Host is done) and forcing it to
> pick a new
> 	buffer (since Mesa GBM allows 4 backbuffers).
> 	I have tried this experiment with KMSRO and Wayland-based Qemu
> UI previously
> 	on iGPUs (and Weston) and noticed that the Guest FPS was getting
> halved:
> 	https://lore.kernel.org/qemu-devel/20210913222036.3193732-1-
> vivek.kasireddy@intel.com/
> 
> 	and also discussed and proposed a solution which did not go
> anywhere:
> 	https://lore.kernel.org/dri-devel/20210913233529.3194401-1-
> vivek.kasireddy@intel.com/
> 
> 	>
> 	> Perhaps, do that just to be safe unless you have performance
> concerns.
> 	If you meant pinning the imported scanout buffer in the Guest, then
> yes,
> 	that is something I am already doing in this patchset.
> 
> 	>
> 	>
> 	>       >
> 	>       > Only amdgpu calls dma_buf_move_notfiy(..), and you're
> probably
> 	> testing on
> 	>       > Intel only, so you may not be hitting that code path anyways.
> 	>       I have tested with the Xe driver in the Guest which also calls
> 	> dma_buf_move_notfiy(). But
> 	>       note that for dGPUs, both Xe and amdgpu migrate the scanout
> buffer
> 	> from vram to system
> 	>       memory as part of export, because virtio-gpu is not P2P
> compatible.
> 	> However, I am hoping
> 	>       to relax this (p2p check against virtio-gpu) in Xe driver if it
> detects
> 	> that it is running in
> 	>       VF mode once the following patch series is merged:
> 	>       https://lore.kernel.org/dri-devel/20240422063602.3690124-1-
> 	> vivek.kasireddy@intel.com/ <http://vivek.kasireddy@intel.com/>
> 	>
> 	>       > I forgot the
> 	>       > exact reason, but apparently udmabuf may not work with
> amdgpu
> 	> displays
> 	>       > and it seems the virtualized iGPU + dGPU is the way to go for
> 	> amdgpu
> 	>       > anyways.
> 	>       I would really like to know why udmabuf would not work with
> 	> amdgpu?
> 	>
> 	>
> 	>
> 	> It's just a rumor I heard, but the idea is udmabuf would be
> imported into
> 	> AMDGPU_GEM_DOMAIN_CPU only.
> 	>
> 	> https://cgit.freedesktop.org/drm/drm-
> 	>
> misc/tree/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c#n333
> 	>
> 	> "AMDGPU_GEM_DOMAIN_CPU: System memory that is not GPU
> accessible.
> 	> Memory in this pool could be swapped out to disk if there is
> pressure."
> 	>
> 	> https://dri.freedesktop.org/docs/drm/gpu/amdgpu.html
> 	>
> 	>
> 	> Perhaps that limitation is artificial and unnecessary, and it may
> indeed work.
> 	> I don't think anybody has tried...
> 	Since udmabuf driver properly pins the backing pages (from memfd)
> for DMA,
> 	I don't see any reason why amdgpu would not be able to import.
> 
> 	Thanks,
> 	Vivek
> 
> 	>
> 	>
> 	>
> 	>
> 	>       > So I recommend just pinning the buffer for the lifetime of the
> 	>       > import for simplicity and correctness.
> 	>       Yeah, in this patch series, the dmabuf is indeed pinned, but only
> for a
> 	> short duration in the Guest –
> 	>       just until the Host is done using it (blit or encode).
> 	>
> 	>       Thanks,
> 	>       Vivek
> 	>
> 	>       >
> 	>       >
> 	>       >       This series is available at:
> 	>       >       https://gitlab.freedesktop.org/Vivek/drm-tip/-
> 	>       > /commits/virtgpu_import_rfc
> 	>       >
> 	>       >       along with additional patches for Qemu and Spice here:
> 	>       >       https://gitlab.freedesktop.org/Vivek/qemu/-
> 	>       > /commits/virtgpu_dmabuf_pcidev
> 	>       >       https://gitlab.freedesktop.org/Vivek/spice/-
> 	>       > /commits/encode_dmabuf_v4
> 	>       >
> 	>       >       Patchset overview:
> 	>       >
> 	>       >       Patch 1:   Implement
> 	>       > VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING cmd
> 	>       >       Patch 2-3: Helpers to initalize, import, free imported object
> 	>       >       Patch 4-5: Import and use buffers from other devices for
> 	> scanout
> 	>       >       Patch 6-7: Have udmabuf driver create dmabuf from PCI
> bars
> 	> for P2P
> 	>       > DMA
> 	>       >
> 	>       >       This series is tested using the following method:
> 	>       >       - Run Qemu with the following relevant options:
> 	>       >         qemu-system-x86_64 -m 4096m ....
> 	>       >         -device vfio-pci,host=0000:03:00.0
> 	>       >         -device virtio-
> 	> vga,max_outputs=1,blob=true,xres=1920,yres=1080
> 	>       >         -spice port=3001,gl=on,disable-ticketing=on,preferred-
> 	>       > codec=gstreamer:h264
> 	>       >         -object memory-backend-memfd,id=mem1,size=4096M
> 	>       >         -machine memory-backend=mem1 ...
> 	>       >       - Run upstream Weston with the following options in the
> Guest
> 	> VM:
> 	>       >         ./weston --drm-device=card1 --additional-devices=card0
> 	>       >
> 	>       >       where card1 is a DG2 dGPU (passthrough'd and using xe
> driver
> 	> in
> 	>       > Guest VM),
> 	>       >       card0 is virtio-gpu and the Host is using a RPL iGPU.
> 	>       >
> 	>       >       Cc: Gerd Hoffmann <kraxel@redhat.com
> <mailto:kraxel@redhat.com>
> 	> <mailto:kraxel@redhat.com <mailto:kraxel@redhat.com> >
> 	>       > <mailto:kraxel@redhat.com <mailto:kraxel@redhat.com>
> <mailto:kraxel@redhat.com <mailto:kraxel@redhat.com> > > >
> 	>       >       Cc: Dongwon Kim <dongwon.kim@intel.com
> <mailto:dongwon.kim@intel.com>
> 	> <mailto:dongwon.kim@intel.com
> <mailto:dongwon.kim@intel.com> >
> 	>       > <mailto:dongwon.kim@intel.com
> <mailto:dongwon.kim@intel.com>
> 	> <mailto:dongwon.kim@intel.com
> <mailto:dongwon.kim@intel.com> > > >
> 	>       >       Cc: Daniel Vetter <daniel.vetter@ffwll.ch
> <mailto:daniel.vetter@ffwll.ch>
> 	> <mailto:daniel.vetter@ffwll.ch <mailto:daniel.vetter@ffwll.ch> >
> 	>       > <mailto:daniel.vetter@ffwll.ch
> <mailto:daniel.vetter@ffwll.ch>  <mailto:daniel.vetter@ffwll.ch
> <mailto:daniel.vetter@ffwll.ch> > > >
> 	>       >       Cc: Christian Koenig <christian.koenig@amd.com
> <mailto:christian.koenig@amd.com>
> 	> <mailto:christian.koenig@amd.com
> <mailto:christian.koenig@amd.com> >
> 	>       > <mailto:christian.koenig@amd.com
> <mailto:christian.koenig@amd.com>
> 	> <mailto:christian.koenig@amd.com
> <mailto:christian.koenig@amd.com> > > >
> 	>       >       Cc: Dmitry Osipenko <dmitry.osipenko@collabora.com
> <mailto:dmitry.osipenko@collabora.com>
> 	> <mailto:dmitry.osipenko@collabora.com
> <mailto:dmitry.osipenko@collabora.com> >
> 	>       > <mailto:dmitry.osipenko@collabora.com
> <mailto:dmitry.osipenko@collabora.com>
> 	> <mailto:dmitry.osipenko@collabora.com
> <mailto:dmitry.osipenko@collabora.com> > > >
> 	>       >       Cc: Rob Clark <robdclark@chromium.org
> <mailto:robdclark@chromium.org>
> 	> <mailto:robdclark@chromium.org
> <mailto:robdclark@chromium.org> >
> 	>       > <mailto:robdclark@chromium.org
> <mailto:robdclark@chromium.org>
> 	> <mailto:robdclark@chromium.org
> <mailto:robdclark@chromium.org> > > >
> 	>       >       Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com
> <mailto:thomas.hellstrom@linux.intel.com>
> 	> <mailto:thomas.hellstrom@linux.intel.com
> <mailto:thomas.hellstrom@linux.intel.com> >
> 	>       > <mailto:thomas.hellstrom@linux.intel.com
> <mailto:thomas.hellstrom@linux.intel.com>
> 	> <mailto:thomas.hellstrom@linux.intel.com
> <mailto:thomas.hellstrom@linux.intel.com> > > >
> 	>       >       Cc: Oded Gabbay <ogabbay@kernel.org
> <mailto:ogabbay@kernel.org>
> 	> <mailto:ogabbay@kernel.org <mailto:ogabbay@kernel.org> >
> 	>       > <mailto:ogabbay@kernel.org <mailto:ogabbay@kernel.org>
> <mailto:ogabbay@kernel.org <mailto:ogabbay@kernel.org> > > >
> 	>       >       Cc: Michal Wajdeczko <michal.wajdeczko@intel.com
> <mailto:michal.wajdeczko@intel.com>
> 	> <mailto:michal.wajdeczko@intel.com
> <mailto:michal.wajdeczko@intel.com> >
> 	>       > <mailto:michal.wajdeczko@intel.com
> <mailto:michal.wajdeczko@intel.com>
> 	> <mailto:michal.wajdeczko@intel.com
> <mailto:michal.wajdeczko@intel.com> > > >
> 	>       >       Cc: Michael Tretter <m.tretter@pengutronix.de
> <mailto:m.tretter@pengutronix.de>
> 	> <mailto:m.tretter@pengutronix.de
> <mailto:m.tretter@pengutronix.de> >
> 	>       > <mailto:m.tretter@pengutronix.de
> <mailto:m.tretter@pengutronix.de>
> 	> <mailto:m.tretter@pengutronix.de
> <mailto:m.tretter@pengutronix.de> > > >
> 	>       >
> 	>       >       Vivek Kasireddy (7):
> 	>       >         drm/virtio: Implement
> 	>       > VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING cmd
> 	>       >         drm/virtio: Add a helper to map and note the dma addrs
> and
> 	>       > lengths
> 	>       >         drm/virtio: Add helpers to initialize and free the imported
> 	> object
> 	>       >         drm/virtio: Import prime buffers from other devices as
> guest
> 	> blobs
> 	>       >         drm/virtio: Ensure that bo's backing store is valid while
> 	> updating
> 	>       >           plane
> 	>       >         udmabuf/uapi: Add new ioctl to create a dmabuf from PCI
> bar
> 	>       > regions
> 	>       >         udmabuf: Implement
> UDMABUF_CREATE_LIST_FOR_PCIDEV
> 	> ioctl
> 	>       >
> 	>       >        drivers/dma-buf/udmabuf.c              | 122
> ++++++++++++++++--
> 	>       >        drivers/gpu/drm/virtio/virtgpu_drv.h   |   8 ++
> 	>       >        drivers/gpu/drm/virtio/virtgpu_plane.c |  56 ++++++++-
> 	>       >        drivers/gpu/drm/virtio/virtgpu_prime.c | 167
> 	>       > ++++++++++++++++++++++++-
> 	>       >        drivers/gpu/drm/virtio/virtgpu_vq.c    |  15 +++
> 	>       >        include/uapi/linux/udmabuf.h           |  11 +-
> 	>       >        6 files changed, 368 insertions(+), 11 deletions(-)
> 	>       >
> 	>       >       --
> 	>       >       2.43.0
> 	>       >
> 	>       >
> 	>
> 	>
> 
>