[PATCH v7 0/3] Virtio-GPU S4 support

[PATCH v7 0/3] Virtio-GPU S4 support

[dongwon.kim]

From: Dongwon Kim <dongwon.kim@intel.com>

This patch series implements functions for .freeze and .restore hooks for
virtio-gpu driver as well as pm-notifier to handle object restoration in
S4(hiberation) case.

First patch adds `virtgpu_freeze` and `virtgpu_restore` functions.
These functions handle the deletion of virtio queues before suspension and
their recreation during the restoration process.

Second patch implements a mechanism for restoring `virtio_gpu_object` instances.
This is necessary because the host (QEMU) deletes all associated resources during
the virtio-gpu reset, which occurs as part of hiberation/resume process.

Third patch adds pm-notifier to the driver that handles resubmission of virtio-gpu
objects to the QEMU once the guest resumes from hibenation.

These changes ensure that the virtio-gpu driver can properly handle hibernation
scenarios without resource loss.

v2: 10ms sleep is added in virtgpu_freeze to avoid the situation
    the driver is locked up during resumption.

v3: Plain 10ms delay (v2) is replaced with wait calls which wait until
    the virtio queue is empty.
    (Dmitry Osipenko)

v4: New version of patchset only covers S4 case because loss of resources in S3
    case can be avoided by skipping virtio-gpu-reset in QEMU (hw/display/virtio-gpu.c).
    To skip virtio-gpu-reset (soft-reset), virtio-gpu-pci device should be attached to
    PCIE bus AND a PCIE option, 'x-pcie-pm-no-soft-reset' should added and set to 'true'.
    (e.g. -device virtio-gpu-pci,bus=port,x-pcie-pm-no-soft-reset=true)

v5: Remove virtio_gpu_object from the restore list before freeing the object
    to prevent an use-after-free situation.
    (Nirmoy Das)

    Protect restore list operations with a spinlock
    (Nirmoy Das)

    Move restore list node into virtio_gpu_bo struct to reduce memory usage
    (Dmitry Osipenko)

    Remove unused header - drm_atomic_helper.h
    (Dmitry Osipenko)

v6: Include object backed by imported dmabuf
    (Dmitry Osipenko)

    Not storing virgl objects in the restore_list as virgl 3D objects are not
    recoverable.
    (Dmitry Osipenko)

    Change the name 'list',a node in restore_list to 'restore_node'
    (Nirmoy Das)

    Use mutex instead of spinlock when updating restore_list
    (Nirmoy Das)

    Initialize restore_node when virtio_gpu_object is created - this is to
    check if the node is in the list with 'list_empty' before removing it.

    Restoring objects in the PM notifier is too late, as virtio-gpu
    message communication begins in virtgpu_restore once virtqueues
    are re-established. To address this, a 'hibernation' flag is set
    during the PM_HIBERNATION_PREPARE phase in the notifier. This flag
    is then used in virtgpu_restore to detect if the system is resuming
    from S4, allowing objects to be recovered immediately after virtqueues
    are reconfigured.

v7: Add a helper, virtio_gpu_add_object_to_restore_list
    (Dmitry Osipenko)

    Unreference all objects before hibernation so they can be removed
    on the host side, since they will be fully restored anyway. This
    prevents the situation where host-side hibernation fails (leaving
    all associated resources still alive) while the virtio-gpu driver
    still attempts to restore those objects.
    (Dmitry Osipenko)

Dongwon Kim (3):
  drm/virtio: Freeze and restore hooks to support suspend and resume
  drm/virtio: Add support for saving and restoring virtio_gpu_objects
  drm/virtio: Add PM notifier to restore objects after hibernation

 drivers/gpu/drm/virtio/virtgpu_drv.c    | 74 +++++++++++++++++++++-
 drivers/gpu/drm/virtio/virtgpu_drv.h    | 23 ++++++-
 drivers/gpu/drm/virtio/virtgpu_kms.c    | 54 ++++++++++++++--
 drivers/gpu/drm/virtio/virtgpu_object.c | 83 ++++++++++++++++++++++++-
 drivers/gpu/drm/virtio/virtgpu_prime.c  | 43 ++++++++++++-
 drivers/gpu/drm/virtio/virtgpu_vq.c     | 13 +++-
 drivers/gpu/drm/virtio/virtgpu_vram.c   |  4 +-
 7 files changed, 280 insertions(+), 14 deletions(-)

-- 
2.34.1

[PATCH v7 1/3] drm/virtio: Freeze and restore hooks to support suspend and resume

[dongwon.kim]

From: Dongwon Kim <dongwon.kim@intel.com>

virtio device needs to delete before VM suspend happens
then reinitialize all virtqueues again upon resume

v2: 10ms sleep was added in virtgpu_freeze to avoid the situation
    the driver is locked up during resumption.

v3: Plain 10ms delay was replaced with wait calls which wait until
    the virtio queue is empty.
    (Dmitry Osipenko)

v4: Change wait_event to wait_event_timeout to prevent permanent wait
    (Nirmoy Das)

Suggested-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Tested-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Cc: Vivek Kasireddy <vivek.kasireddy@intel.com>
Cc: Nirmoy Das <nirmoyd@nvidia.com>
Signed-off-by: Dongwon Kim <dongwon.kim@intel.com>
---
 drivers/gpu/drm/virtio/virtgpu_drv.c | 62 +++++++++++++++++++++++++++-
 drivers/gpu/drm/virtio/virtgpu_drv.h |  1 +
 drivers/gpu/drm/virtio/virtgpu_kms.c | 23 ++++++++---
 3 files changed, 79 insertions(+), 7 deletions(-)

diff --git a/drivers/gpu/drm/virtio/virtgpu_drv.c b/drivers/gpu/drm/virtio/virtgpu_drv.c
index 71c6ccad4b99..039fb8f18ead 100644
--- a/drivers/gpu/drm/virtio/virtgpu_drv.c
+++ b/drivers/gpu/drm/virtio/virtgpu_drv.c
@@ -163,6 +163,62 @@ static unsigned int features[] = {
 	VIRTIO_GPU_F_RESOURCE_BLOB,
 	VIRTIO_GPU_F_CONTEXT_INIT,
 };
+
+#ifdef CONFIG_PM_SLEEP
+static int virtgpu_freeze(struct virtio_device *vdev)
+{
+	struct drm_device *dev = vdev->priv;
+	struct virtio_gpu_device *vgdev = dev->dev_private;
+	int error;
+
+	error = drm_mode_config_helper_suspend(dev);
+	if (error) {
+		DRM_ERROR("suspend error %d\n", error);
+		return error;
+	}
+
+	flush_work(&vgdev->obj_free_work);
+	flush_work(&vgdev->ctrlq.dequeue_work);
+	flush_work(&vgdev->cursorq.dequeue_work);
+	flush_work(&vgdev->config_changed_work);
+
+	wait_event_timeout(vgdev->ctrlq.ack_queue,
+		vgdev->ctrlq.vq->num_free == vgdev->ctrlq.vq->num_max,
+		5 * HZ);
+
+	wait_event_timeout(vgdev->cursorq.ack_queue,
+		vgdev->cursorq.vq->num_free == vgdev->cursorq.vq->num_max,
+		5 * HZ);
+
+	vdev->config->del_vqs(vdev);
+
+	return 0;
+}
+
+static int virtgpu_restore(struct virtio_device *vdev)
+{
+	struct drm_device *dev = vdev->priv;
+	struct virtio_gpu_device *vgdev = dev->dev_private;
+	int error;
+
+	error = virtio_gpu_find_vqs(vgdev);
+	if (error) {
+		DRM_ERROR("failed to find virt queues\n");
+		return error;
+	}
+
+	virtio_device_ready(vdev);
+
+	error = drm_mode_config_helper_resume(dev);
+	if (error) {
+		DRM_ERROR("resume error %d\n", error);
+		return error;
+	}
+
+	return 0;
+}
+#endif
+
 static struct virtio_driver virtio_gpu_driver = {
 	.feature_table = features,
 	.feature_table_size = ARRAY_SIZE(features),
@@ -171,7 +227,11 @@ static struct virtio_driver virtio_gpu_driver = {
 	.probe = virtio_gpu_probe,
 	.remove = virtio_gpu_remove,
 	.shutdown = virtio_gpu_shutdown,
-	.config_changed = virtio_gpu_config_changed
+	.config_changed = virtio_gpu_config_changed,
+#ifdef CONFIG_PM_SLEEP
+	.freeze = virtgpu_freeze,
+	.restore = virtgpu_restore,
+#endif
 };
 
 static int __init virtio_gpu_driver_init(void)
diff --git a/drivers/gpu/drm/virtio/virtgpu_drv.h b/drivers/gpu/drm/virtio/virtgpu_drv.h
index f17660a71a3e..1279f998c8e0 100644
--- a/drivers/gpu/drm/virtio/virtgpu_drv.h
+++ b/drivers/gpu/drm/virtio/virtgpu_drv.h
@@ -300,6 +300,7 @@ void virtio_gpu_deinit(struct drm_device *dev);
 void virtio_gpu_release(struct drm_device *dev);
 int virtio_gpu_driver_open(struct drm_device *dev, struct drm_file *file);
 void virtio_gpu_driver_postclose(struct drm_device *dev, struct drm_file *file);
+int virtio_gpu_find_vqs(struct virtio_gpu_device *vgdev);
 
 /* virtgpu_gem.c */
 int virtio_gpu_gem_object_open(struct drm_gem_object *obj,
diff --git a/drivers/gpu/drm/virtio/virtgpu_kms.c b/drivers/gpu/drm/virtio/virtgpu_kms.c
index 1c15cbf326b7..cbebe19c3fb3 100644
--- a/drivers/gpu/drm/virtio/virtgpu_kms.c
+++ b/drivers/gpu/drm/virtio/virtgpu_kms.c
@@ -114,15 +114,28 @@ static void virtio_gpu_get_capsets(struct virtio_gpu_device *vgdev,
 	vgdev->num_capsets = num_capsets;
 }
 
-int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev)
+int virtio_gpu_find_vqs(struct virtio_gpu_device *vgdev)
 {
 	struct virtqueue_info vqs_info[] = {
 		{ "control", virtio_gpu_ctrl_ack },
 		{ "cursor", virtio_gpu_cursor_ack },
 	};
-	struct virtio_gpu_device *vgdev;
-	/* this will expand later */
 	struct virtqueue *vqs[2];
+	int ret;
+
+	ret = virtio_find_vqs(vgdev->vdev, 2, vqs, vqs_info, NULL);
+	if (ret)
+		return ret;
+
+	vgdev->ctrlq.vq = vqs[0];
+	vgdev->cursorq.vq = vqs[1];
+
+	return 0;
+}
+
+int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev)
+{
+	struct virtio_gpu_device *vgdev;
 	u32 num_scanouts, num_capsets;
 	int ret = 0;
 
@@ -206,13 +219,11 @@ int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev)
 	DRM_INFO("features: %ccontext_init\n",
 		 vgdev->has_context_init ? '+' : '-');
 
-	ret = virtio_find_vqs(vgdev->vdev, 2, vqs, vqs_info, NULL);
+	ret = virtio_gpu_find_vqs(vgdev);
 	if (ret) {
 		DRM_ERROR("failed to find virt queues\n");
 		goto err_vqs;
 	}
-	vgdev->ctrlq.vq = vqs[0];
-	vgdev->cursorq.vq = vqs[1];
 	ret = virtio_gpu_alloc_vbufs(vgdev);
 	if (ret) {
 		DRM_ERROR("failed to alloc vbufs\n");
-- 
2.34.1

[PATCH v7 2/3] drm/virtio: Add support for saving and restoring virtio_gpu_objects

[dongwon.kim]

From: Dongwon Kim <dongwon.kim@intel.com>

When the host KVM/QEMU resumes from hibernation, it loses all graphics
resources previously submitted by the guest OS, as the QEMU process is
terminated during the suspend-resume cycle. This leads to invalid resource
errors when the guest OS attempts to interact with the host using those
resources after resumption.

To resolve this, the virtio-gpu driver now tracks all active virtio_gpu_objects
and provides a mechanism to restore them by re-submitting the objects to QEMU
when needed (e.g., during resume from hibernation).

v2: - Attach backing is done if bo->attached was set before

v3: - Restoration is no longer triggered via .restore; instead, it is handled
      by a PM notifier only during hibernation.

v4: - Remove virtio_gpu_object from the restore list before freeing the object
      to prevent an use-after-free situation.
      (Nirmoy Das)

    - Protect restore list operations with a spinlock
      (Nirmoy Das)

    - Initialize ret with 0 in virtio_gpu_object_restore_all
      (Nirmoy Das)

    - Move restore list node into virtio_gpu_bo struct to reduce memory usage
      (Dmitry Osipenko)

v5: - Include object backed by imported dmabuf
      (Dmitry Osipenko)

    - Not storing virgl objects in the restore_list as virgl 3D objects are not
      recoverable.
      (Dmitry Osipenko)

    - Change the name 'list',a node in restore_list to 'restore_node'
      (Nirmoy Das)

    - Use mutex instead of spinlock when updating restore_list
      (Nirmoy Das)

    - Initialize restore_node when virtio_gpu_object is created - this is to
      determine whether the object should be removed from the restore_list with
      'list_empty' function when it is time to free the object as not all objects
      will be added to the list.

v6: - Add a helper, virtio_gpu_add_object_to_restore_list
      (Dmitry Osipenko)

Cc: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Cc: Vivek Kasireddy <vivek.kasireddy@intel.com>
Cc: Nirmoy Das <nirmoyd@nvidia.com>
Signed-off-by: Dongwon Kim <dongwon.kim@intel.com>
---
 drivers/gpu/drm/virtio/virtgpu_drv.h    | 14 ++++++
 drivers/gpu/drm/virtio/virtgpu_kms.c    |  3 ++
 drivers/gpu/drm/virtio/virtgpu_object.c | 66 +++++++++++++++++++++++++
 drivers/gpu/drm/virtio/virtgpu_prime.c  | 41 +++++++++++++++
 drivers/gpu/drm/virtio/virtgpu_vram.c   |  2 +
 5 files changed, 126 insertions(+)

diff --git a/drivers/gpu/drm/virtio/virtgpu_drv.h b/drivers/gpu/drm/virtio/virtgpu_drv.h
index 1279f998c8e0..1632d92ee346 100644
--- a/drivers/gpu/drm/virtio/virtgpu_drv.h
+++ b/drivers/gpu/drm/virtio/virtgpu_drv.h
@@ -98,6 +98,10 @@ struct virtio_gpu_object {
 
 	int uuid_state;
 	uuid_t uuid;
+
+	/* for restoration of objects after hibernation */
+	struct virtio_gpu_object_params params;
+	struct list_head restore_node;
 };
 #define gem_to_virtio_gpu_obj(gobj) \
 	container_of((gobj), struct virtio_gpu_object, base.base)
@@ -265,6 +269,8 @@ struct virtio_gpu_device {
 	struct work_struct obj_free_work;
 	spinlock_t obj_free_lock;
 	struct list_head obj_free_list;
+	struct mutex obj_restore_lock;
+	struct list_head obj_restore_list;
 
 	struct virtio_gpu_drv_capset *capsets;
 	uint32_t num_capsets;
@@ -479,6 +485,12 @@ bool virtio_gpu_is_shmem(struct virtio_gpu_object *bo);
 
 int virtio_gpu_resource_id_get(struct virtio_gpu_device *vgdev,
 			       uint32_t *resid);
+
+void virtio_gpu_add_object_to_restore_list(struct virtio_gpu_device *vgdev,
+					   struct virtio_gpu_object *bo);
+
+int virtio_gpu_object_restore_all(struct virtio_gpu_device *vgdev);
+
 /* virtgpu_prime.c */
 int virtio_gpu_resource_assign_uuid(struct virtio_gpu_device *vgdev,
 				    struct virtio_gpu_object *bo);
@@ -493,6 +505,8 @@ int virtgpu_dma_buf_import_sgt(struct virtio_gpu_mem_entry **ents,
 			       unsigned int *nents,
 			       struct virtio_gpu_object *bo,
 			       struct dma_buf_attachment *attach);
+int virtgpu_dma_buf_obj_resubmit(struct virtio_gpu_device *vgdev,
+				 struct virtio_gpu_object *bo);
 
 /* virtgpu_debugfs.c */
 void virtio_gpu_debugfs_init(struct drm_minor *minor);
diff --git a/drivers/gpu/drm/virtio/virtgpu_kms.c b/drivers/gpu/drm/virtio/virtgpu_kms.c
index cbebe19c3fb3..8ad79de70d85 100644
--- a/drivers/gpu/drm/virtio/virtgpu_kms.c
+++ b/drivers/gpu/drm/virtio/virtgpu_kms.c
@@ -170,6 +170,8 @@ int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev)
 		  virtio_gpu_array_put_free_work);
 	INIT_LIST_HEAD(&vgdev->obj_free_list);
 	spin_lock_init(&vgdev->obj_free_lock);
+	INIT_LIST_HEAD(&vgdev->obj_restore_list);
+	mutex_init(&vgdev->obj_restore_lock);
 
 #ifdef __LITTLE_ENDIAN
 	if (virtio_has_feature(vgdev->vdev, VIRTIO_GPU_F_VIRGL))
@@ -298,6 +300,7 @@ void virtio_gpu_deinit(struct drm_device *dev)
 	flush_work(&vgdev->config_changed_work);
 	virtio_reset_device(vgdev->vdev);
 	vgdev->vdev->config->del_vqs(vgdev->vdev);
+	mutex_destroy(&vgdev->obj_restore_lock);
 }
 
 void virtio_gpu_release(struct drm_device *dev)
diff --git a/drivers/gpu/drm/virtio/virtgpu_object.c b/drivers/gpu/drm/virtio/virtgpu_object.c
index e6363c887500..8e8a8ec4a361 100644
--- a/drivers/gpu/drm/virtio/virtgpu_object.c
+++ b/drivers/gpu/drm/virtio/virtgpu_object.c
@@ -65,6 +65,12 @@ void virtio_gpu_cleanup_object(struct virtio_gpu_object *bo)
 {
 	struct virtio_gpu_device *vgdev = bo->base.base.dev->dev_private;
 
+	if (!list_empty(&bo->restore_node)) {
+		mutex_lock(&vgdev->obj_restore_lock);
+		list_del(&bo->restore_node);
+		mutex_unlock(&vgdev->obj_restore_lock);
+	}
+
 	virtio_gpu_resource_id_put(vgdev, bo->hw_res_handle);
 	if (virtio_gpu_is_shmem(bo)) {
 		drm_gem_shmem_free(&bo->base);
@@ -220,6 +226,8 @@ int virtio_gpu_object_create(struct virtio_gpu_device *vgdev,
 		return PTR_ERR(shmem_obj);
 	bo = gem_to_virtio_gpu_obj(&shmem_obj->base);
 
+	INIT_LIST_HEAD(&bo->restore_node);
+
 	ret = virtio_gpu_resource_id_get(vgdev, &bo->hw_res_handle);
 	if (ret < 0)
 		goto err_free_gem;
@@ -258,6 +266,12 @@ int virtio_gpu_object_create(struct virtio_gpu_device *vgdev,
 		virtio_gpu_object_attach(vgdev, bo, ents, nents);
 	}
 
+	if (!params->virgl) {
+		/* store non-virgl object with its param to the restore list */
+		bo->params = *params;
+		virtio_gpu_add_object_to_restore_list(vgdev, bo);
+	}
+
 	*bo_ptr = bo;
 	return 0;
 
@@ -271,3 +285,55 @@ int virtio_gpu_object_create(struct virtio_gpu_device *vgdev,
 	drm_gem_shmem_free(shmem_obj);
 	return ret;
 }
+
+void virtio_gpu_add_object_to_restore_list(struct virtio_gpu_device *vgdev,
+					   struct virtio_gpu_object *bo)
+{
+	mutex_lock(&vgdev->obj_restore_lock);
+	list_add_tail(&bo->restore_node, &vgdev->obj_restore_list);
+	mutex_unlock(&vgdev->obj_restore_lock);
+}
+
+int virtio_gpu_object_restore_all(struct virtio_gpu_device *vgdev)
+{
+	struct virtio_gpu_object *bo, *tmp;
+	struct virtio_gpu_mem_entry *ents;
+	unsigned int nents;
+	int ret = 0;
+
+	mutex_lock(&vgdev->obj_restore_lock);
+	list_for_each_entry_safe(bo, tmp, &vgdev->obj_restore_list,
+				 restore_node) {
+		if (drm_gem_is_imported(&bo->base.base)) {
+			ret = virtgpu_dma_buf_obj_resubmit(vgdev, bo);
+			if (ret)
+				break;
+
+			continue;
+		}
+
+		if (bo->params.blob || bo->attached) {
+			ret = virtio_gpu_object_shmem_init(vgdev, bo, &ents,
+							   &nents);
+			if (ret)
+				break;
+		}
+
+		if (bo->params.blob) {
+			virtio_gpu_cmd_resource_create_blob(vgdev, bo,
+							    &bo->params,
+							    ents, nents);
+		} else {
+			virtio_gpu_cmd_create_resource(vgdev, bo, &bo->params,
+						       NULL, NULL);
+			if (bo->attached) {
+				bo->attached = false;
+				virtio_gpu_object_attach(vgdev, bo, ents,
+							 nents);
+			}
+		}
+	}
+	mutex_unlock(&vgdev->obj_restore_lock);
+
+	return ret;
+}
diff --git a/drivers/gpu/drm/virtio/virtgpu_prime.c b/drivers/gpu/drm/virtio/virtgpu_prime.c
index ce49282198cb..088eaf000a9a 100644
--- a/drivers/gpu/drm/virtio/virtgpu_prime.c
+++ b/drivers/gpu/drm/virtio/virtgpu_prime.c
@@ -264,6 +264,12 @@ static int virtgpu_dma_buf_init_obj(struct drm_device *dev,
 	dma_buf_unpin(attach);
 	dma_resv_unlock(resv);
 
+	/* store the dmabuf imported object with its params to
+	 * the restore list
+	 */
+	bo->params = params;
+	virtio_gpu_add_object_to_restore_list(vgdev, bo);
+
 	return 0;
 
 err_import:
@@ -274,6 +280,39 @@ static int virtgpu_dma_buf_init_obj(struct drm_device *dev,
 	return ret;
 }
 
+int virtgpu_dma_buf_obj_resubmit(struct virtio_gpu_device *vgdev,
+				 struct virtio_gpu_object *bo)
+{
+	struct virtio_gpu_mem_entry *ents;
+	struct scatterlist *sl;
+	int i;
+
+	if (!bo->sgt) {
+		DRM_ERROR("no sgt bound to virtio_gpu_object\n");
+		return -ENOMEM;
+	}
+
+	ents = kvmalloc_array(bo->sgt->nents,
+			      sizeof(struct virtio_gpu_mem_entry),
+			      GFP_KERNEL);
+	if (!ents) {
+		DRM_ERROR("failed to allocate ent list\n");
+		return -ENOMEM;
+	}
+
+	for_each_sgtable_dma_sg(bo->sgt, sl, i) {
+		ents[i].addr = cpu_to_le64(sg_dma_address(sl));
+		ents[i].length = cpu_to_le32(sg_dma_len(sl));
+		ents[i].padding = 0;
+	}
+
+	virtio_gpu_cmd_resource_create_blob(vgdev, bo, &bo->params,
+					    ents, bo->sgt->nents);
+
+	return 0;
+}
+
+
 static const struct drm_gem_object_funcs virtgpu_gem_dma_buf_funcs = {
 	.free = virtgpu_dma_buf_free_obj,
 };
@@ -319,6 +358,8 @@ struct drm_gem_object *virtgpu_gem_prime_import(struct drm_device *dev,
 	if (!bo)
 		return ERR_PTR(-ENOMEM);
 
+	INIT_LIST_HEAD(&bo->restore_node);
+
 	obj = &bo->base.base;
 	obj->resv = buf->resv;
 	obj->funcs = &virtgpu_gem_dma_buf_funcs;
diff --git a/drivers/gpu/drm/virtio/virtgpu_vram.c b/drivers/gpu/drm/virtio/virtgpu_vram.c
index 5ad3b7c6f73c..dffba04849f5 100644
--- a/drivers/gpu/drm/virtio/virtgpu_vram.c
+++ b/drivers/gpu/drm/virtio/virtgpu_vram.c
@@ -200,6 +200,8 @@ int virtio_gpu_vram_create(struct virtio_gpu_device *vgdev,
 	obj = &vram->base.base.base;
 	obj->funcs = &virtio_gpu_vram_funcs;
 
+	INIT_LIST_HEAD(&vram->base.restore_node);
+
 	params->size = PAGE_ALIGN(params->size);
 	drm_gem_private_object_init(vgdev->ddev, obj, params->size);
 
-- 
2.34.1

[PATCH v7 3/3] drm/virtio: Add PM notifier to restore objects after hibernation

[dongwon.kim]

From: Dongwon Kim <dongwon.kim@intel.com>

Register a PM notifier in virtio-gpu to handle suspend/hibernate
events. On PM_POST_HIBERNATION, restore all GPU objects so that the
driver can properly recover after resume.

v2: Remove unused header - drm_atomic_helper.h
    (Dmitry Osipenko)

v3: Objects for virgl usecase can't be recovered after resume so
    blocking S4 when virgl is enabled
    (Dmitry Osipenko)

v4: Restoring objects in the PM notifier is too late, as virtio-gpu
    message communication begins in virtgpu_restore once virtqueues
    are re-established. To address this, a 'hibernation' flag is set
    during the PM_HIBERNATION_PREPARE phase in the notifier. This flag
    is then used in virtgpu_restore to detect if the system is resuming
    from S4, allowing objects to be recovered immediately after virtqueues
    are reconfigured.

v5: Unreference all objects before hibernation so they can be removed
    on the host side, since they will be fully restored anyway. This
    prevents the situation where host-side hibernation fails (leaving
    all associated resources still alive) while the virtio-gpu driver
    still attempts to restore those objects.
    (Dmitry Osipenko)

Suggested-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Cc: Vivek Kasireddy <vivek.kasireddy@intel.com>
Signed-off-by: Dongwon Kim <dongwon.kim@intel.com>
---
 drivers/gpu/drm/virtio/virtgpu_drv.c    | 12 +++++++++++
 drivers/gpu/drm/virtio/virtgpu_drv.h    |  8 ++++++-
 drivers/gpu/drm/virtio/virtgpu_kms.c    | 28 +++++++++++++++++++++++++
 drivers/gpu/drm/virtio/virtgpu_object.c | 17 ++++++++++++++-
 drivers/gpu/drm/virtio/virtgpu_prime.c  |  2 +-
 drivers/gpu/drm/virtio/virtgpu_vq.c     | 13 +++++++++---
 drivers/gpu/drm/virtio/virtgpu_vram.c   |  2 +-
 7 files changed, 75 insertions(+), 7 deletions(-)

diff --git a/drivers/gpu/drm/virtio/virtgpu_drv.c b/drivers/gpu/drm/virtio/virtgpu_drv.c
index 039fb8f18ead..268f45f9a573 100644
--- a/drivers/gpu/drm/virtio/virtgpu_drv.c
+++ b/drivers/gpu/drm/virtio/virtgpu_drv.c
@@ -177,6 +177,9 @@ static int virtgpu_freeze(struct virtio_device *vdev)
 		return error;
 	}
 
+	if (vgdev->hibernation)
+		virtio_gpu_object_unref_all(vgdev);
+
 	flush_work(&vgdev->obj_free_work);
 	flush_work(&vgdev->ctrlq.dequeue_work);
 	flush_work(&vgdev->cursorq.dequeue_work);
@@ -209,6 +212,15 @@ static int virtgpu_restore(struct virtio_device *vdev)
 
 	virtio_device_ready(vdev);
 
+	if (vgdev->hibernation) {
+		vgdev->hibernation = false;
+		error = virtio_gpu_object_restore_all(vgdev);
+		if (error) {
+			DRM_ERROR("Failed to recover virtio-gpu objects\n");
+			return error;
+		}
+	}
+
 	error = drm_mode_config_helper_resume(dev);
 	if (error) {
 		DRM_ERROR("resume error %d\n", error);
diff --git a/drivers/gpu/drm/virtio/virtgpu_drv.h b/drivers/gpu/drm/virtio/virtgpu_drv.h
index 1632d92ee346..ffddaaf9bc76 100644
--- a/drivers/gpu/drm/virtio/virtgpu_drv.h
+++ b/drivers/gpu/drm/virtio/virtgpu_drv.h
@@ -261,6 +261,7 @@ struct virtio_gpu_device {
 	bool has_resource_blob;
 	bool has_host_visible;
 	bool has_context_init;
+	bool hibernation;
 	struct virtio_shm_region host_visible_region;
 	struct drm_mm host_visible_mm;
 
@@ -277,6 +278,8 @@ struct virtio_gpu_device {
 	uint64_t capset_id_mask;
 	struct list_head cap_cache;
 
+	struct notifier_block pm_nb;
+
 	/* protects uuid state when exporting */
 	spinlock_t resource_export_lock;
 	/* protects map state and host_visible_mm */
@@ -341,7 +344,8 @@ void virtio_gpu_cmd_create_resource(struct virtio_gpu_device *vgdev,
 				    struct virtio_gpu_object_array *objs,
 				    struct virtio_gpu_fence *fence);
 void virtio_gpu_cmd_unref_resource(struct virtio_gpu_device *vgdev,
-				   struct virtio_gpu_object *bo);
+				   struct virtio_gpu_object *bo,
+				   int no_cb);
 int virtio_gpu_panic_cmd_transfer_to_host_2d(struct virtio_gpu_device *vgdev,
 					     uint64_t offset,
 					     uint32_t width, uint32_t height,
@@ -491,6 +495,8 @@ void virtio_gpu_add_object_to_restore_list(struct virtio_gpu_device *vgdev,
 
 int virtio_gpu_object_restore_all(struct virtio_gpu_device *vgdev);
 
+void virtio_gpu_object_unref_all(struct virtio_gpu_device *vgdev);
+
 /* virtgpu_prime.c */
 int virtio_gpu_resource_assign_uuid(struct virtio_gpu_device *vgdev,
 				    struct virtio_gpu_object *bo);
diff --git a/drivers/gpu/drm/virtio/virtgpu_kms.c b/drivers/gpu/drm/virtio/virtgpu_kms.c
index 8ad79de70d85..cc8aad1b72cd 100644
--- a/drivers/gpu/drm/virtio/virtgpu_kms.c
+++ b/drivers/gpu/drm/virtio/virtgpu_kms.c
@@ -26,6 +26,8 @@
 #include <linux/virtio.h>
 #include <linux/virtio_config.h>
 #include <linux/virtio_ring.h>
+#include <linux/suspend.h>
+#include <linux/pm_runtime.h>
 
 #include <drm/drm_file.h>
 #include <drm/drm_managed.h>
@@ -133,6 +135,25 @@ int virtio_gpu_find_vqs(struct virtio_gpu_device *vgdev)
 	return 0;
 }
 
+static int virtio_gpu_pm_notifier(struct notifier_block *nb, unsigned long mode,
+				  void *data)
+{
+	struct virtio_gpu_device *vgdev = container_of(nb,
+						struct virtio_gpu_device,
+						pm_nb);
+
+	if (mode == PM_HIBERNATION_PREPARE) {
+		if (vgdev->has_virgl_3d) {
+			DRM_ERROR("S4 not allowed when VIRGL is enabled\n");
+			return notifier_from_errno(-EPERM);
+		}
+
+		vgdev->hibernation = true;
+	}
+
+	return NOTIFY_DONE;
+}
+
 int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev)
 {
 	struct virtio_gpu_device *vgdev;
@@ -269,6 +290,12 @@ int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev)
 		wait_event_timeout(vgdev->resp_wq, !vgdev->display_info_pending,
 				   5 * HZ);
 	}
+
+	vgdev->pm_nb.notifier_call = virtio_gpu_pm_notifier;
+	ret = register_pm_notifier(&vgdev->pm_nb);
+	if (ret)
+		goto err_scanouts;
+
 	return 0;
 
 err_scanouts:
@@ -301,6 +328,7 @@ void virtio_gpu_deinit(struct drm_device *dev)
 	virtio_reset_device(vgdev->vdev);
 	vgdev->vdev->config->del_vqs(vgdev->vdev);
 	mutex_destroy(&vgdev->obj_restore_lock);
+	unregister_pm_notifier(&vgdev->pm_nb);
 }
 
 void virtio_gpu_release(struct drm_device *dev)
diff --git a/drivers/gpu/drm/virtio/virtgpu_object.c b/drivers/gpu/drm/virtio/virtgpu_object.c
index 8e8a8ec4a361..7c27c2fcd7ed 100644
--- a/drivers/gpu/drm/virtio/virtgpu_object.c
+++ b/drivers/gpu/drm/virtio/virtgpu_object.c
@@ -98,7 +98,7 @@ static void virtio_gpu_free_object(struct drm_gem_object *obj)
 	struct virtio_gpu_device *vgdev = bo->base.base.dev->dev_private;
 
 	if (bo->created) {
-		virtio_gpu_cmd_unref_resource(vgdev, bo);
+		virtio_gpu_cmd_unref_resource(vgdev, bo, false);
 		virtio_gpu_notify(vgdev);
 		/* completion handler calls virtio_gpu_cleanup_object() */
 		return;
@@ -337,3 +337,18 @@ int virtio_gpu_object_restore_all(struct virtio_gpu_device *vgdev)
 
 	return ret;
 }
+
+void virtio_gpu_object_unref_all(struct virtio_gpu_device *vgdev)
+{
+	struct virtio_gpu_object *bo, *tmp;
+
+	mutex_lock(&vgdev->obj_restore_lock);
+	list_for_each_entry_safe(bo, tmp, &vgdev->obj_restore_list,
+				 restore_node)
+		if (bo->created) {
+			virtio_gpu_cmd_unref_resource(vgdev, bo, true);
+			virtio_gpu_notify(vgdev);
+		}
+
+	mutex_unlock(&vgdev->obj_restore_lock);
+}
diff --git a/drivers/gpu/drm/virtio/virtgpu_prime.c b/drivers/gpu/drm/virtio/virtgpu_prime.c
index 088eaf000a9a..69f2186d0908 100644
--- a/drivers/gpu/drm/virtio/virtgpu_prime.c
+++ b/drivers/gpu/drm/virtio/virtgpu_prime.c
@@ -218,7 +218,7 @@ static void virtgpu_dma_buf_free_obj(struct drm_gem_object *obj)
 	}
 
 	if (bo->created) {
-		virtio_gpu_cmd_unref_resource(vgdev, bo);
+		virtio_gpu_cmd_unref_resource(vgdev, bo, false);
 		virtio_gpu_notify(vgdev);
 		return;
 	}
diff --git a/drivers/gpu/drm/virtio/virtgpu_vq.c b/drivers/gpu/drm/virtio/virtgpu_vq.c
index 8181b22b9b46..fdd0166584bc 100644
--- a/drivers/gpu/drm/virtio/virtgpu_vq.c
+++ b/drivers/gpu/drm/virtio/virtgpu_vq.c
@@ -625,14 +625,21 @@ static void virtio_gpu_cmd_unref_cb(struct virtio_gpu_device *vgdev,
 }
 
 void virtio_gpu_cmd_unref_resource(struct virtio_gpu_device *vgdev,
-				   struct virtio_gpu_object *bo)
+				   struct virtio_gpu_object *bo,
+				   int no_cb)
 {
 	struct virtio_gpu_resource_unref *cmd_p;
 	struct virtio_gpu_vbuffer *vbuf;
 	int ret;
 
-	cmd_p = virtio_gpu_alloc_cmd_cb(vgdev, &vbuf, sizeof(*cmd_p),
-					virtio_gpu_cmd_unref_cb);
+	if (no_cb) {
+		cmd_p = virtio_gpu_alloc_cmd_cb(vgdev, &vbuf, sizeof(*cmd_p),
+						NULL);
+	} else {
+		cmd_p = virtio_gpu_alloc_cmd_cb(vgdev, &vbuf, sizeof(*cmd_p),
+						virtio_gpu_cmd_unref_cb);
+	}
+
 	memset(cmd_p, 0, sizeof(*cmd_p));
 
 	cmd_p->hdr.type = cpu_to_le32(VIRTIO_GPU_CMD_RESOURCE_UNREF);
diff --git a/drivers/gpu/drm/virtio/virtgpu_vram.c b/drivers/gpu/drm/virtio/virtgpu_vram.c
index dffba04849f5..5ec75bf03d0f 100644
--- a/drivers/gpu/drm/virtio/virtgpu_vram.c
+++ b/drivers/gpu/drm/virtio/virtgpu_vram.c
@@ -18,7 +18,7 @@ static void virtio_gpu_vram_free(struct drm_gem_object *obj)
 		if (unmap)
 			virtio_gpu_cmd_unmap(vgdev, bo);
 
-		virtio_gpu_cmd_unref_resource(vgdev, bo);
+		virtio_gpu_cmd_unref_resource(vgdev, bo, false);
 		virtio_gpu_notify(vgdev);
 		return;
 	}
-- 
2.34.1

Re: [PATCH v7 2/3] drm/virtio: Add support for saving and restoring virtio_gpu_objects

[kernel test robot]

Hi,

kernel test robot noticed the following build errors:

[auto build test ERROR on drm-misc/drm-misc-next]
[also build test ERROR on drm/drm-next daeinki-drm-exynos/exynos-drm-next drm-i915/for-linux-next drm-i915/for-linux-next-fixes drm-tip/drm-tip linus/master v6.19-rc4 next-20260109]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/dongwon-kim-intel-com/drm-virtio-Freeze-and-restore-hooks-to-support-suspend-and-resume/20260108-024124
base:   https://gitlab.freedesktop.org/drm/misc/kernel.git drm-misc-next
patch link:    https://lore.kernel.org/r/20260107182745.229481-3-dongwon.kim%40intel.com
patch subject: [PATCH v7 2/3] drm/virtio: Add support for saving and restoring virtio_gpu_objects
config: x86_64-rhel-9.4 (https://download.01.org/0day-ci/archive/20260109/202601091715.AaqfDsnD-lkp@intel.com/config)
compiler: gcc-14 (Debian 14.2.0-19) 14.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260109/202601091715.AaqfDsnD-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202601091715.AaqfDsnD-lkp@intel.com/

All errors (new ones prefixed by >>):

   drivers/gpu/drm/virtio/virtgpu_prime.c: In function 'virtgpu_dma_buf_obj_resubmit':
>> drivers/gpu/drm/virtio/virtgpu_prime.c:291:17: error: implicit declaration of function 'DRM_ERROR'; did you mean 'pud_ERROR'? [-Wimplicit-function-declaration]
     291 |                 DRM_ERROR("no sgt bound to virtio_gpu_object\n");
         |                 ^~~~~~~~~
         |                 pud_ERROR


vim +291 drivers/gpu/drm/virtio/virtgpu_prime.c

   282	
   283	int virtgpu_dma_buf_obj_resubmit(struct virtio_gpu_device *vgdev,
   284					 struct virtio_gpu_object *bo)
   285	{
   286		struct virtio_gpu_mem_entry *ents;
   287		struct scatterlist *sl;
   288		int i;
   289	
   290		if (!bo->sgt) {
 > 291			DRM_ERROR("no sgt bound to virtio_gpu_object\n");
   292			return -ENOMEM;
   293		}
   294	
   295		ents = kvmalloc_array(bo->sgt->nents,
   296				      sizeof(struct virtio_gpu_mem_entry),
   297				      GFP_KERNEL);
   298		if (!ents) {
   299			DRM_ERROR("failed to allocate ent list\n");
   300			return -ENOMEM;
   301		}
   302	
   303		for_each_sgtable_dma_sg(bo->sgt, sl, i) {
   304			ents[i].addr = cpu_to_le64(sg_dma_address(sl));
   305			ents[i].length = cpu_to_le32(sg_dma_len(sl));
   306			ents[i].padding = 0;
   307		}
   308	
   309		virtio_gpu_cmd_resource_create_blob(vgdev, bo, &bo->params,
   310						    ents, bo->sgt->nents);
   311	
   312		return 0;
   313	}
   314	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH v7 2/3] drm/virtio: Add support for saving and restoring virtio_gpu_objects

[kernel test robot]

Hi,

kernel test robot noticed the following build errors:

[auto build test ERROR on drm-misc/drm-misc-next]
[also build test ERROR on drm/drm-next daeinki-drm-exynos/exynos-drm-next drm-i915/for-linux-next drm-i915/for-linux-next-fixes drm-tip/drm-tip linus/master v6.19-rc4 next-20260109]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/dongwon-kim-intel-com/drm-virtio-Freeze-and-restore-hooks-to-support-suspend-and-resume/20260108-024124
base:   https://gitlab.freedesktop.org/drm/misc/kernel.git drm-misc-next
patch link:    https://lore.kernel.org/r/20260107182745.229481-3-dongwon.kim%40intel.com
patch subject: [PATCH v7 2/3] drm/virtio: Add support for saving and restoring virtio_gpu_objects
config: hexagon-allmodconfig (https://download.01.org/0day-ci/archive/20260112/202601121153.7CddhDzp-lkp@intel.com/config)
compiler: clang version 17.0.6 (https://github.com/llvm/llvm-project 6009708b4367171ccdbf4b5905cb6a803753fe18)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260112/202601121153.7CddhDzp-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202601121153.7CddhDzp-lkp@intel.com/

All errors (new ones prefixed by >>):

>> drivers/gpu/drm/virtio/virtgpu_prime.c:291:3: error: call to undeclared function 'DRM_ERROR'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
     291 |                 DRM_ERROR("no sgt bound to virtio_gpu_object\n");
         |                 ^
   drivers/gpu/drm/virtio/virtgpu_prime.c:299:3: error: call to undeclared function 'DRM_ERROR'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
     299 |                 DRM_ERROR("failed to allocate ent list\n");
         |                 ^
   2 errors generated.


vim +/DRM_ERROR +291 drivers/gpu/drm/virtio/virtgpu_prime.c

   282	
   283	int virtgpu_dma_buf_obj_resubmit(struct virtio_gpu_device *vgdev,
   284					 struct virtio_gpu_object *bo)
   285	{
   286		struct virtio_gpu_mem_entry *ents;
   287		struct scatterlist *sl;
   288		int i;
   289	
   290		if (!bo->sgt) {
 > 291			DRM_ERROR("no sgt bound to virtio_gpu_object\n");
   292			return -ENOMEM;
   293		}
   294	
   295		ents = kvmalloc_array(bo->sgt->nents,
   296				      sizeof(struct virtio_gpu_mem_entry),
   297				      GFP_KERNEL);
   298		if (!ents) {
   299			DRM_ERROR("failed to allocate ent list\n");
   300			return -ENOMEM;
   301		}
   302	
   303		for_each_sgtable_dma_sg(bo->sgt, sl, i) {
   304			ents[i].addr = cpu_to_le64(sg_dma_address(sl));
   305			ents[i].length = cpu_to_le32(sg_dma_len(sl));
   306			ents[i].padding = 0;
   307		}
   308	
   309		virtio_gpu_cmd_resource_create_blob(vgdev, bo, &bo->params,
   310						    ents, bo->sgt->nents);
   311	
   312		return 0;
   313	}
   314	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH v7 2/3] drm/virtio: Add support for saving and restoring virtio_gpu_objects

[kernel test robot]

Hi,

kernel test robot noticed the following build errors:

[auto build test ERROR on drm-misc/drm-misc-next]
[also build test ERROR on drm/drm-next daeinki-drm-exynos/exynos-drm-next drm-i915/for-linux-next drm-i915/for-linux-next-fixes drm-tip/drm-tip linus/master v6.19-rc4 next-20260109]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/dongwon-kim-intel-com/drm-virtio-Freeze-and-restore-hooks-to-support-suspend-and-resume/20260108-024124
base:   https://gitlab.freedesktop.org/drm/misc/kernel.git drm-misc-next
patch link:    https://lore.kernel.org/r/20260107182745.229481-3-dongwon.kim%40intel.com
patch subject: [PATCH v7 2/3] drm/virtio: Add support for saving and restoring virtio_gpu_objects
config: m68k-allmodconfig (https://download.01.org/0day-ci/archive/20260112/202601121246.Q3BHNIsw-lkp@intel.com/config)
compiler: m68k-linux-gcc (GCC) 15.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260112/202601121246.Q3BHNIsw-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202601121246.Q3BHNIsw-lkp@intel.com/

All errors (new ones prefixed by >>):

   drivers/gpu/drm/virtio/virtgpu_prime.c: In function 'virtgpu_dma_buf_obj_resubmit':
>> drivers/gpu/drm/virtio/virtgpu_prime.c:291:17: error: implicit declaration of function 'DRM_ERROR'; did you mean 'DRM_IOR'? [-Wimplicit-function-declaration]
     291 |                 DRM_ERROR("no sgt bound to virtio_gpu_object\n");
         |                 ^~~~~~~~~
         |                 DRM_IOR


vim +291 drivers/gpu/drm/virtio/virtgpu_prime.c

   282	
   283	int virtgpu_dma_buf_obj_resubmit(struct virtio_gpu_device *vgdev,
   284					 struct virtio_gpu_object *bo)
   285	{
   286		struct virtio_gpu_mem_entry *ents;
   287		struct scatterlist *sl;
   288		int i;
   289	
   290		if (!bo->sgt) {
 > 291			DRM_ERROR("no sgt bound to virtio_gpu_object\n");
   292			return -ENOMEM;
   293		}
   294	
   295		ents = kvmalloc_array(bo->sgt->nents,
   296				      sizeof(struct virtio_gpu_mem_entry),
   297				      GFP_KERNEL);
   298		if (!ents) {
   299			DRM_ERROR("failed to allocate ent list\n");
   300			return -ENOMEM;
   301		}
   302	
   303		for_each_sgtable_dma_sg(bo->sgt, sl, i) {
   304			ents[i].addr = cpu_to_le64(sg_dma_address(sl));
   305			ents[i].length = cpu_to_le32(sg_dma_len(sl));
   306			ents[i].padding = 0;
   307		}
   308	
   309		virtio_gpu_cmd_resource_create_blob(vgdev, bo, &bo->params,
   310						    ents, bo->sgt->nents);
   311	
   312		return 0;
   313	}
   314	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH v7 0/3] Virtio-GPU S4 support

[Dmitry Osipenko]

On 1/7/26 21:27, dongwon.kim@intel.com wrote:
> From: Dongwon Kim <dongwon.kim@intel.com>
> 
> This patch series implements functions for .freeze and .restore hooks for
> virtio-gpu driver as well as pm-notifier to handle object restoration in
> S4(hiberation) case.
> 
> First patch adds `virtgpu_freeze` and `virtgpu_restore` functions.
> These functions handle the deletion of virtio queues before suspension and
> their recreation during the restoration process.
> 
> Second patch implements a mechanism for restoring `virtio_gpu_object` instances.
> This is necessary because the host (QEMU) deletes all associated resources during
> the virtio-gpu reset, which occurs as part of hiberation/resume process.
> 
> Third patch adds pm-notifier to the driver that handles resubmission of virtio-gpu
> objects to the QEMU once the guest resumes from hibenation.
> 
> These changes ensure that the virtio-gpu driver can properly handle hibernation
> scenarios without resource loss.
> 
> v2: 10ms sleep is added in virtgpu_freeze to avoid the situation
>     the driver is locked up during resumption.
> 
> v3: Plain 10ms delay (v2) is replaced with wait calls which wait until
>     the virtio queue is empty.
>     (Dmitry Osipenko)
> 
> v4: New version of patchset only covers S4 case because loss of resources in S3
>     case can be avoided by skipping virtio-gpu-reset in QEMU (hw/display/virtio-gpu.c).
>     To skip virtio-gpu-reset (soft-reset), virtio-gpu-pci device should be attached to
>     PCIE bus AND a PCIE option, 'x-pcie-pm-no-soft-reset' should added and set to 'true'.
>     (e.g. -device virtio-gpu-pci,bus=port,x-pcie-pm-no-soft-reset=true)
> 
> v5: Remove virtio_gpu_object from the restore list before freeing the object
>     to prevent an use-after-free situation.
>     (Nirmoy Das)
> 
>     Protect restore list operations with a spinlock
>     (Nirmoy Das)
> 
>     Move restore list node into virtio_gpu_bo struct to reduce memory usage
>     (Dmitry Osipenko)
> 
>     Remove unused header - drm_atomic_helper.h
>     (Dmitry Osipenko)
> 
> v6: Include object backed by imported dmabuf
>     (Dmitry Osipenko)
> 
>     Not storing virgl objects in the restore_list as virgl 3D objects are not
>     recoverable.
>     (Dmitry Osipenko)
> 
>     Change the name 'list',a node in restore_list to 'restore_node'
>     (Nirmoy Das)
> 
>     Use mutex instead of spinlock when updating restore_list
>     (Nirmoy Das)
> 
>     Initialize restore_node when virtio_gpu_object is created - this is to
>     check if the node is in the list with 'list_empty' before removing it.
> 
>     Restoring objects in the PM notifier is too late, as virtio-gpu
>     message communication begins in virtgpu_restore once virtqueues
>     are re-established. To address this, a 'hibernation' flag is set
>     during the PM_HIBERNATION_PREPARE phase in the notifier. This flag
>     is then used in virtgpu_restore to detect if the system is resuming
>     from S4, allowing objects to be recovered immediately after virtqueues
>     are reconfigured.
> 
> v7: Add a helper, virtio_gpu_add_object_to_restore_list
>     (Dmitry Osipenko)
> 
>     Unreference all objects before hibernation so they can be removed
>     on the host side, since they will be fully restored anyway. This
>     prevents the situation where host-side hibernation fails (leaving
>     all associated resources still alive) while the virtio-gpu driver
>     still attempts to restore those objects.
>     (Dmitry Osipenko)
> 
> Dongwon Kim (3):
>   drm/virtio: Freeze and restore hooks to support suspend and resume
>   drm/virtio: Add support for saving and restoring virtio_gpu_objects
>   drm/virtio: Add PM notifier to restore objects after hibernation
> 
>  drivers/gpu/drm/virtio/virtgpu_drv.c    | 74 +++++++++++++++++++++-
>  drivers/gpu/drm/virtio/virtgpu_drv.h    | 23 ++++++-
>  drivers/gpu/drm/virtio/virtgpu_kms.c    | 54 ++++++++++++++--
>  drivers/gpu/drm/virtio/virtgpu_object.c | 83 ++++++++++++++++++++++++-
>  drivers/gpu/drm/virtio/virtgpu_prime.c  | 43 ++++++++++++-
>  drivers/gpu/drm/virtio/virtgpu_vq.c     | 13 +++-
>  drivers/gpu/drm/virtio/virtgpu_vram.c   |  4 +-
>  7 files changed, 280 insertions(+), 14 deletions(-)
> 

Hello Kim,

Want let you know that I've seen the patches, but didn't have enough
time to review and test them. Will try to do it sooner. Will leave
couple comments for now.

Meanwhile there is a kernel bot bug report.

-- 
Best regards,
Dmitry

Re: [PATCH v7 1/3] drm/virtio: Freeze and restore hooks to support suspend and resume

[Dmitry Osipenko]

On 1/7/26 21:27, dongwon.kim@intel.com wrote:
> +#ifdef CONFIG_PM_SLEEP
> +static int virtgpu_freeze(struct virtio_device *vdev)
> +{
> +	struct drm_device *dev = vdev->priv;
> +	struct virtio_gpu_device *vgdev = dev->dev_private;
> +	int error;
> +
> +	error = drm_mode_config_helper_suspend(dev);
> +	if (error) {
> +		DRM_ERROR("suspend error %d\n", error);

Nit: better to use DRM_DEV_ERROR everywhere to know which device
produces the error

-- 
Best regards,
Dmitry

Re: [PATCH v7 3/3] drm/virtio: Add PM notifier to restore objects after hibernation

[Dmitry Osipenko]

On 1/7/26 21:27, dongwon.kim@intel.com wrote:
> --- a/drivers/gpu/drm/virtio/virtgpu_drv.c
> +++ b/drivers/gpu/drm/virtio/virtgpu_drv.c
> @@ -177,6 +177,9 @@ static int virtgpu_freeze(struct virtio_device *vdev)
>  		return error;
>  	}
>  
> +	if (vgdev->hibernation)
> +		virtio_gpu_object_unref_all(vgdev);

Nit: function name doesn't reflect well what it's doing, I'd name it as
`virtio_gpu_unref_stored_objs`

-- 
Best regards,
Dmitry

Re: [PATCH v7 3/3] drm/virtio: Add PM notifier to restore objects after hibernation

[Dmitry Osipenko]

On 1/7/26 21:27, dongwon.kim@intel.com wrote:
> +++ b/drivers/gpu/drm/virtio/virtgpu_drv.h
> @@ -261,6 +261,7 @@ struct virtio_gpu_device {
>  	bool has_resource_blob;
>  	bool has_host_visible;
>  	bool has_context_init;
> +	bool hibernation;

Nit: bool hibernated;

-- 
Best regards,
Dmitry

Re: [PATCH v7 2/3] drm/virtio: Add support for saving and restoring virtio_gpu_objects

[Dmitry Osipenko]

On 1/7/26 21:27, dongwon.kim@intel.com wrote:
> diff --git a/drivers/gpu/drm/virtio/virtgpu_object.c b/drivers/gpu/drm/virtio/virtgpu_object.c
> index e6363c887500..8e8a8ec4a361 100644
> --- a/drivers/gpu/drm/virtio/virtgpu_object.c
> +++ b/drivers/gpu/drm/virtio/virtgpu_object.c
> @@ -65,6 +65,12 @@ void virtio_gpu_cleanup_object(struct virtio_gpu_object *bo)
>  {
>  	struct virtio_gpu_device *vgdev = bo->base.base.dev->dev_private;
>  
> +	if (!list_empty(&bo->restore_node)) {
> +		mutex_lock(&vgdev->obj_restore_lock);
> +		list_del(&bo->restore_node);
> +		mutex_unlock(&vgdev->obj_restore_lock);
> +	}

This looks fragile to me, I'd use list_del_init() unconditionally.

Also, looks wrong to delete object's node from the list by the
virtio_gpu_cleanup_object() that is invoked from cmd-completion
callback. You calling virtio_gpu_object_unref_all() that walks up
obj_restore_list while obj already may be in process of async unref'ing?

-- 
Best regards,
Dmitry

RE: [PATCH v7 0/3] Virtio-GPU S4 support

[Kim, Dongwon]

Hi Dmitry,

> -----Original Message-----
> From: Dmitry Osipenko <dmitry.osipenko@collabora.com>
> Sent: Wednesday, February 25, 2026 8:06 AM
> To: Kim, Dongwon <dongwon.kim@intel.com>; dri-devel@lists.freedesktop.org;
> airlied@redhat.com; kraxel@redhat.com
> Cc: nirmoyd@nvidia.com; Kasireddy, Vivek <vivek.kasireddy@intel.com>
> Subject: Re: [PATCH v7 0/3] Virtio-GPU S4 support
> 
> On 1/7/26 21:27, dongwon.kim@intel.com wrote:
> > From: Dongwon Kim <dongwon.kim@intel.com>
> >
> > This patch series implements functions for .freeze and .restore hooks
> > for virtio-gpu driver as well as pm-notifier to handle object
> > restoration in
> > S4(hiberation) case.
> >
> > First patch adds `virtgpu_freeze` and `virtgpu_restore` functions.
> > These functions handle the deletion of virtio queues before suspension
> > and their recreation during the restoration process.
> >
> > Second patch implements a mechanism for restoring `virtio_gpu_object`
> instances.
> > This is necessary because the host (QEMU) deletes all associated
> > resources during the virtio-gpu reset, which occurs as part of
> hiberation/resume process.
> >
> > Third patch adds pm-notifier to the driver that handles resubmission
> > of virtio-gpu objects to the QEMU once the guest resumes from hibenation.
> >
> > These changes ensure that the virtio-gpu driver can properly handle
> > hibernation scenarios without resource loss.
> >
> > v2: 10ms sleep is added in virtgpu_freeze to avoid the situation
> >     the driver is locked up during resumption.
> >
> > v3: Plain 10ms delay (v2) is replaced with wait calls which wait until
> >     the virtio queue is empty.
> >     (Dmitry Osipenko)
> >
> > v4: New version of patchset only covers S4 case because loss of resources in
> S3
> >     case can be avoided by skipping virtio-gpu-reset in QEMU
> (hw/display/virtio-gpu.c).
> >     To skip virtio-gpu-reset (soft-reset), virtio-gpu-pci device should be
> attached to
> >     PCIE bus AND a PCIE option, 'x-pcie-pm-no-soft-reset' should added and set
> to 'true'.
> >     (e.g. -device
> > virtio-gpu-pci,bus=port,x-pcie-pm-no-soft-reset=true)
> >
> > v5: Remove virtio_gpu_object from the restore list before freeing the object
> >     to prevent an use-after-free situation.
> >     (Nirmoy Das)
> >
> >     Protect restore list operations with a spinlock
> >     (Nirmoy Das)
> >
> >     Move restore list node into virtio_gpu_bo struct to reduce memory usage
> >     (Dmitry Osipenko)
> >
> >     Remove unused header - drm_atomic_helper.h
> >     (Dmitry Osipenko)
> >
> > v6: Include object backed by imported dmabuf
> >     (Dmitry Osipenko)
> >
> >     Not storing virgl objects in the restore_list as virgl 3D objects are not
> >     recoverable.
> >     (Dmitry Osipenko)
> >
> >     Change the name 'list',a node in restore_list to 'restore_node'
> >     (Nirmoy Das)
> >
> >     Use mutex instead of spinlock when updating restore_list
> >     (Nirmoy Das)
> >
> >     Initialize restore_node when virtio_gpu_object is created - this is to
> >     check if the node is in the list with 'list_empty' before removing it.
> >
> >     Restoring objects in the PM notifier is too late, as virtio-gpu
> >     message communication begins in virtgpu_restore once virtqueues
> >     are re-established. To address this, a 'hibernation' flag is set
> >     during the PM_HIBERNATION_PREPARE phase in the notifier. This flag
> >     is then used in virtgpu_restore to detect if the system is resuming
> >     from S4, allowing objects to be recovered immediately after virtqueues
> >     are reconfigured.
> >
> > v7: Add a helper, virtio_gpu_add_object_to_restore_list
> >     (Dmitry Osipenko)
> >
> >     Unreference all objects before hibernation so they can be removed
> >     on the host side, since they will be fully restored anyway. This
> >     prevents the situation where host-side hibernation fails (leaving
> >     all associated resources still alive) while the virtio-gpu driver
> >     still attempts to restore those objects.
> >     (Dmitry Osipenko)
> >
> > Dongwon Kim (3):
> >   drm/virtio: Freeze and restore hooks to support suspend and resume
> >   drm/virtio: Add support for saving and restoring virtio_gpu_objects
> >   drm/virtio: Add PM notifier to restore objects after hibernation
> >
> >  drivers/gpu/drm/virtio/virtgpu_drv.c    | 74 +++++++++++++++++++++-
> >  drivers/gpu/drm/virtio/virtgpu_drv.h    | 23 ++++++-
> >  drivers/gpu/drm/virtio/virtgpu_kms.c    | 54 ++++++++++++++--
> >  drivers/gpu/drm/virtio/virtgpu_object.c | 83
> > ++++++++++++++++++++++++-  drivers/gpu/drm/virtio/virtgpu_prime.c  | 43
> ++++++++++++-
> >  drivers/gpu/drm/virtio/virtgpu_vq.c     | 13 +++-
> >  drivers/gpu/drm/virtio/virtgpu_vram.c   |  4 +-
> >  7 files changed, 280 insertions(+), 14 deletions(-)
> >
> 
> Hello Kim,
> 
> Want let you know that I've seen the patches, but didn't have enough time to
> review and test them. Will try to do it sooner. Will leave couple comments for
> now.
> 
> Meanwhile there is a kernel bot bug report.

Thanks for the comment. No problem. I will wait for your test results then create V8 based on
your feedback including what you just left for individual patches (function/variable
names..etc.)

I thought I got an error report for the previous version but I will double check.

> 
> --
> Best regards,
> Dmitry

Re: [PATCH v7 0/3] Virtio-GPU S4 support

[Dmitry Osipenko]

On 2/25/26 20:06, Kim, Dongwon wrote:
> Hi Dmitry,
> 
>> -----Original Message-----
>> From: Dmitry Osipenko <dmitry.osipenko@collabora.com>
>> Sent: Wednesday, February 25, 2026 8:06 AM
>> To: Kim, Dongwon <dongwon.kim@intel.com>; dri-devel@lists.freedesktop.org;
>> airlied@redhat.com; kraxel@redhat.com
>> Cc: nirmoyd@nvidia.com; Kasireddy, Vivek <vivek.kasireddy@intel.com>
>> Subject: Re: [PATCH v7 0/3] Virtio-GPU S4 support
>>
>> On 1/7/26 21:27, dongwon.kim@intel.com wrote:
>>> From: Dongwon Kim <dongwon.kim@intel.com>
>>>
>>> This patch series implements functions for .freeze and .restore hooks
>>> for virtio-gpu driver as well as pm-notifier to handle object
>>> restoration in
>>> S4(hiberation) case.
>>>
>>> First patch adds `virtgpu_freeze` and `virtgpu_restore` functions.
>>> These functions handle the deletion of virtio queues before suspension
>>> and their recreation during the restoration process.
>>>
>>> Second patch implements a mechanism for restoring `virtio_gpu_object`
>> instances.
>>> This is necessary because the host (QEMU) deletes all associated
>>> resources during the virtio-gpu reset, which occurs as part of
>> hiberation/resume process.
>>>
>>> Third patch adds pm-notifier to the driver that handles resubmission
>>> of virtio-gpu objects to the QEMU once the guest resumes from hibenation.
>>>
>>> These changes ensure that the virtio-gpu driver can properly handle
>>> hibernation scenarios without resource loss.
>>>
>>> v2: 10ms sleep is added in virtgpu_freeze to avoid the situation
>>>     the driver is locked up during resumption.
>>>
>>> v3: Plain 10ms delay (v2) is replaced with wait calls which wait until
>>>     the virtio queue is empty.
>>>     (Dmitry Osipenko)
>>>
>>> v4: New version of patchset only covers S4 case because loss of resources in
>> S3
>>>     case can be avoided by skipping virtio-gpu-reset in QEMU
>> (hw/display/virtio-gpu.c).
>>>     To skip virtio-gpu-reset (soft-reset), virtio-gpu-pci device should be
>> attached to
>>>     PCIE bus AND a PCIE option, 'x-pcie-pm-no-soft-reset' should added and set
>> to 'true'.
>>>     (e.g. -device
>>> virtio-gpu-pci,bus=port,x-pcie-pm-no-soft-reset=true)
>>>
>>> v5: Remove virtio_gpu_object from the restore list before freeing the object
>>>     to prevent an use-after-free situation.
>>>     (Nirmoy Das)
>>>
>>>     Protect restore list operations with a spinlock
>>>     (Nirmoy Das)
>>>
>>>     Move restore list node into virtio_gpu_bo struct to reduce memory usage
>>>     (Dmitry Osipenko)
>>>
>>>     Remove unused header - drm_atomic_helper.h
>>>     (Dmitry Osipenko)
>>>
>>> v6: Include object backed by imported dmabuf
>>>     (Dmitry Osipenko)
>>>
>>>     Not storing virgl objects in the restore_list as virgl 3D objects are not
>>>     recoverable.
>>>     (Dmitry Osipenko)
>>>
>>>     Change the name 'list',a node in restore_list to 'restore_node'
>>>     (Nirmoy Das)
>>>
>>>     Use mutex instead of spinlock when updating restore_list
>>>     (Nirmoy Das)
>>>
>>>     Initialize restore_node when virtio_gpu_object is created - this is to
>>>     check if the node is in the list with 'list_empty' before removing it.
>>>
>>>     Restoring objects in the PM notifier is too late, as virtio-gpu
>>>     message communication begins in virtgpu_restore once virtqueues
>>>     are re-established. To address this, a 'hibernation' flag is set
>>>     during the PM_HIBERNATION_PREPARE phase in the notifier. This flag
>>>     is then used in virtgpu_restore to detect if the system is resuming
>>>     from S4, allowing objects to be recovered immediately after virtqueues
>>>     are reconfigured.
>>>
>>> v7: Add a helper, virtio_gpu_add_object_to_restore_list
>>>     (Dmitry Osipenko)
>>>
>>>     Unreference all objects before hibernation so they can be removed
>>>     on the host side, since they will be fully restored anyway. This
>>>     prevents the situation where host-side hibernation fails (leaving
>>>     all associated resources still alive) while the virtio-gpu driver
>>>     still attempts to restore those objects.
>>>     (Dmitry Osipenko)
>>>
>>> Dongwon Kim (3):
>>>   drm/virtio: Freeze and restore hooks to support suspend and resume
>>>   drm/virtio: Add support for saving and restoring virtio_gpu_objects
>>>   drm/virtio: Add PM notifier to restore objects after hibernation
>>>
>>>  drivers/gpu/drm/virtio/virtgpu_drv.c    | 74 +++++++++++++++++++++-
>>>  drivers/gpu/drm/virtio/virtgpu_drv.h    | 23 ++++++-
>>>  drivers/gpu/drm/virtio/virtgpu_kms.c    | 54 ++++++++++++++--
>>>  drivers/gpu/drm/virtio/virtgpu_object.c | 83
>>> ++++++++++++++++++++++++-  drivers/gpu/drm/virtio/virtgpu_prime.c  | 43
>> ++++++++++++-
>>>  drivers/gpu/drm/virtio/virtgpu_vq.c     | 13 +++-
>>>  drivers/gpu/drm/virtio/virtgpu_vram.c   |  4 +-
>>>  7 files changed, 280 insertions(+), 14 deletions(-)
>>>
>>
>> Hello Kim,
>>
>> Want let you know that I've seen the patches, but didn't have enough time to
>> review and test them. Will try to do it sooner. Will leave couple comments for
>> now.
>>
>> Meanwhile there is a kernel bot bug report.
> 
> Thanks for the comment. No problem. I will wait for your test results then create V8 based on
> your feedback including what you just left for individual patches (function/variable
> names..etc.)
> 
> I thought I got an error report for the previous version but I will double check.

Thanks for your patience and sending v8, will test it. There other
patches in my queue, hopefully it will get untangled sooner.

-- 
Best regards,
Dmitry