[PATCH v3 0/3] Introduce DRM device wedged event

[PATCH v3 0/3] Introduce DRM device wedged event

[Raag Jadav]

This series introduces device wedged event in DRM subsystem and uses
it in xe and i915 drivers. Detailed description in commit message.

This was earlier attempted as xe specific uevent in v1 and v2.
https://patchwork.freedesktop.org/series/136909/

v2: Change authorship to Himal (Aravind)
    Add uevent for all device wedged cases (Aravind)
v3: Generic re-implementation in DRM subsystem (Lucas)

Himal Prasad Ghimiray (1):
  drm/xe: Use device wedged event

Raag Jadav (2):
  drm: Introduce device wedged event
  drm/i915: Use device wedged event

 drivers/gpu/drm/drm_drv.c             | 21 +++++++++++++++++++++
 drivers/gpu/drm/i915/gt/intel_reset.c |  2 ++
 drivers/gpu/drm/xe/xe_device.c        |  8 ++++++--
 include/drm/drm_drv.h                 |  1 +
 4 files changed, 30 insertions(+), 2 deletions(-)

-- 
2.34.1

[PATCH v3 1/3] drm: Introduce device wedged event

[Raag Jadav]

Introduce device wedged event, which will notify userspace of wedged
(hanged/unusable) state of the DRM device through a uevent. This is
useful especially in cases where the device is in unrecoverable state
and requires userspace intervention for recovery.

Purpose of this implementation is to be vendor agnostic. Userspace
consumers (sysadmin) can define udev rules to parse this event and
take respective action to recover the device.

Consumer expectations:
----------------------
1) Unbind driver
2) Reset bus device
3) Re-bind driver

Signed-off-by: Raag Jadav <raag.jadav@intel.com>
---
 drivers/gpu/drm/drm_drv.c | 21 +++++++++++++++++++++
 include/drm/drm_drv.h     |  1 +
 2 files changed, 22 insertions(+)

diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c
index 93543071a500..dc55cc237d89 100644
--- a/drivers/gpu/drm/drm_drv.c
+++ b/drivers/gpu/drm/drm_drv.c
@@ -499,6 +499,27 @@ void drm_dev_unplug(struct drm_device *dev)
 }
 EXPORT_SYMBOL(drm_dev_unplug);
 
+/**
+ * drm_dev_wedged - declare DRM device as wedged
+ * @dev: DRM device
+ *
+ * This declares a DRM device specified by @dev as wedged (hanged/unusable)
+ * and generates a uevent for it, on the basis of which, userspace may take
+ * respective action to recover the device.
+ * Currently we only set WEDGED=1 in the uevent environment, but this can
+ * be expanded in the future.
+ */
+void drm_dev_wedged(struct drm_device *dev)
+{
+	char *event_string = "WEDGED=1";
+	char *envp[] = { event_string, NULL };
+
+	DRM_INFO("%s: device wedged, generating uevent\n", dev_name(dev->dev));
+
+	kobject_uevent_env(&dev->primary->kdev->kobj, KOBJ_CHANGE, envp);
+}
+EXPORT_SYMBOL(drm_dev_wedged);
+
 /*
  * DRM internal mount
  * We want to be able to allocate our own "struct address_space" to control
diff --git a/include/drm/drm_drv.h b/include/drm/drm_drv.h
index cd37936c3926..a0b2d1435b86 100644
--- a/include/drm/drm_drv.h
+++ b/include/drm/drm_drv.h
@@ -489,6 +489,7 @@ void drm_put_dev(struct drm_device *dev);
 bool drm_dev_enter(struct drm_device *dev, int *idx);
 void drm_dev_exit(int idx);
 void drm_dev_unplug(struct drm_device *dev);
+void drm_dev_wedged(struct drm_device *dev);
 
 /**
  * drm_dev_is_unplugged - is a DRM device unplugged
-- 
2.34.1

Re: [PATCH v3 1/3] drm: Introduce device wedged event

[Jani Nikula]

On Mon, 02 Sep 2024, Raag Jadav <raag.jadav@intel.com> wrote:
> Introduce device wedged event, which will notify userspace of wedged
> (hanged/unusable) state of the DRM device through a uevent. This is
> useful especially in cases where the device is in unrecoverable state
> and requires userspace intervention for recovery.
>
> Purpose of this implementation is to be vendor agnostic. Userspace
> consumers (sysadmin) can define udev rules to parse this event and
> take respective action to recover the device.
>
> Consumer expectations:
> ----------------------
> 1) Unbind driver
> 2) Reset bus device
> 3) Re-bind driver
>
> Signed-off-by: Raag Jadav <raag.jadav@intel.com>
> ---
>  drivers/gpu/drm/drm_drv.c | 21 +++++++++++++++++++++
>  include/drm/drm_drv.h     |  1 +
>  2 files changed, 22 insertions(+)
>
> diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c
> index 93543071a500..dc55cc237d89 100644
> --- a/drivers/gpu/drm/drm_drv.c
> +++ b/drivers/gpu/drm/drm_drv.c
> @@ -499,6 +499,27 @@ void drm_dev_unplug(struct drm_device *dev)
>  }
>  EXPORT_SYMBOL(drm_dev_unplug);
>  
> +/**
> + * drm_dev_wedged - declare DRM device as wedged
> + * @dev: DRM device
> + *
> + * This declares a DRM device specified by @dev as wedged (hanged/unusable)
> + * and generates a uevent for it, on the basis of which, userspace may take
> + * respective action to recover the device.
> + * Currently we only set WEDGED=1 in the uevent environment, but this can
> + * be expanded in the future.
> + */
> +void drm_dev_wedged(struct drm_device *dev)
> +{
> +	char *event_string = "WEDGED=1";
> +	char *envp[] = { event_string, NULL };
> +
> +	DRM_INFO("%s: device wedged, generating uevent\n", dev_name(dev->dev));

drm_info() please, and you can drop that handrolled dev_name().

BR,
Jani.

> +
> +	kobject_uevent_env(&dev->primary->kdev->kobj, KOBJ_CHANGE, envp);
> +}
> +EXPORT_SYMBOL(drm_dev_wedged);
> +
>  /*
>   * DRM internal mount
>   * We want to be able to allocate our own "struct address_space" to control
> diff --git a/include/drm/drm_drv.h b/include/drm/drm_drv.h
> index cd37936c3926..a0b2d1435b86 100644
> --- a/include/drm/drm_drv.h
> +++ b/include/drm/drm_drv.h
> @@ -489,6 +489,7 @@ void drm_put_dev(struct drm_device *dev);
>  bool drm_dev_enter(struct drm_device *dev, int *idx);
>  void drm_dev_exit(int idx);
>  void drm_dev_unplug(struct drm_device *dev);
> +void drm_dev_wedged(struct drm_device *dev);
>  
>  /**
>   * drm_dev_is_unplugged - is a DRM device unplugged

-- 
Jani Nikula, Intel

Re: [PATCH v3 1/3] drm: Introduce device wedged event

[Aravind Iddamsetty]

On 02/09/24 13:18, Raag Jadav wrote:
> Introduce device wedged event, which will notify userspace of wedged
> (hanged/unusable) state of the DRM device through a uevent. This is
> useful especially in cases where the device is in unrecoverable state
> and requires userspace intervention for recovery.
>
> Purpose of this implementation is to be vendor agnostic. Userspace
> consumers (sysadmin) can define udev rules to parse this event and
> take respective action to recover the device.
>
> Consumer expectations:
> ----------------------
> 1) Unbind driver
> 2) Reset bus device
> 3) Re-bind driver
>
> Signed-off-by: Raag Jadav <raag.jadav@intel.com>
> ---
>  drivers/gpu/drm/drm_drv.c | 21 +++++++++++++++++++++
>  include/drm/drm_drv.h     |  1 +
>  2 files changed, 22 insertions(+)
>
> diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c
> index 93543071a500..dc55cc237d89 100644
> --- a/drivers/gpu/drm/drm_drv.c
> +++ b/drivers/gpu/drm/drm_drv.c
> @@ -499,6 +499,27 @@ void drm_dev_unplug(struct drm_device *dev)
>  }
>  EXPORT_SYMBOL(drm_dev_unplug);
>  
> +/**
> + * drm_dev_wedged - declare DRM device as wedged
> + * @dev: DRM device
> + *
> + * This declares a DRM device specified by @dev as wedged (hanged/unusable)
this doesn't seem to set any drm state as wedged, it is just sending an
uevent. you might need to correct the above statement.

Thanks,
Aravind.
> + * and generates a uevent for it, on the basis of which, userspace may take
> + * respective action to recover the device.
> + * Currently we only set WEDGED=1 in the uevent environment, but this can
> + * be expanded in the future.
> + */
> +void drm_dev_wedged(struct drm_device *dev)
> +{
> +	char *event_string = "WEDGED=1";
> +	char *envp[] = { event_string, NULL };
> +
> +	DRM_INFO("%s: device wedged, generating uevent\n", dev_name(dev->dev));
> +
> +	kobject_uevent_env(&dev->primary->kdev->kobj, KOBJ_CHANGE, envp);
> +}
> +EXPORT_SYMBOL(drm_dev_wedged);
> +
>  /*
>   * DRM internal mount
>   * We want to be able to allocate our own "struct address_space" to control
> diff --git a/include/drm/drm_drv.h b/include/drm/drm_drv.h
> index cd37936c3926..a0b2d1435b86 100644
> --- a/include/drm/drm_drv.h
> +++ b/include/drm/drm_drv.h
> @@ -489,6 +489,7 @@ void drm_put_dev(struct drm_device *dev);
>  bool drm_dev_enter(struct drm_device *dev, int *idx);
>  void drm_dev_exit(int idx);
>  void drm_dev_unplug(struct drm_device *dev);
> +void drm_dev_wedged(struct drm_device *dev);
>  
>  /**
>   * drm_dev_is_unplugged - is a DRM device unplugged

Re: [PATCH v3 1/3] drm: Introduce device wedged event

[Raag Jadav]

On Mon, Sep 02, 2024 at 02:44:21PM +0530, Aravind Iddamsetty wrote:
> 
> On 02/09/24 13:18, Raag Jadav wrote:
> > Introduce device wedged event, which will notify userspace of wedged
> > (hanged/unusable) state of the DRM device through a uevent. This is
> > useful especially in cases where the device is in unrecoverable state
> > and requires userspace intervention for recovery.
> >
> > Purpose of this implementation is to be vendor agnostic. Userspace
> > consumers (sysadmin) can define udev rules to parse this event and
> > take respective action to recover the device.
> >
> > Consumer expectations:
> > ----------------------
> > 1) Unbind driver
> > 2) Reset bus device
> > 3) Re-bind driver
> >
> > Signed-off-by: Raag Jadav <raag.jadav@intel.com>
> > ---
> >  drivers/gpu/drm/drm_drv.c | 21 +++++++++++++++++++++
> >  include/drm/drm_drv.h     |  1 +
> >  2 files changed, 22 insertions(+)
> >
> > diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c
> > index 93543071a500..dc55cc237d89 100644
> > --- a/drivers/gpu/drm/drm_drv.c
> > +++ b/drivers/gpu/drm/drm_drv.c
> > @@ -499,6 +499,27 @@ void drm_dev_unplug(struct drm_device *dev)
> >  }
> >  EXPORT_SYMBOL(drm_dev_unplug);
> >  
> > +/**
> > + * drm_dev_wedged - declare DRM device as wedged
> > + * @dev: DRM device
> > + *
> > + * This declares a DRM device specified by @dev as wedged (hanged/unusable)
> this doesn't seem to set any drm state as wedged, it is just sending an
> uevent. you might need to correct the above statement.

On a second thought, perhaps this warrants any action on drm_device?

Raag

[PATCH v3 2/3] drm/xe: Use device wedged event

[Raag Jadav]

From: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>

This was previously attempted as xe specific reset uevent but dropped
in commit 77a0d4d1cea2 ("drm/xe/uapi: Remove reset uevent for now")
as part of refactoring.

Now that we have device wedged event supported by DRM core, make use
of it. With this in place userspace will be notified of wedged device,
on the basis of which, userspace may take respective action to recover
the device.

$ udevadm monitor --property --kernel
monitor will print the received events for:
KERNEL - the kernel uevent

KERNEL[307.420340] change   /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:01.0/0000:03:00.0/drm/card0 (drm)
ACTION=change
DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:01.0/0000:03:00.0/drm/card0
SUBSYSTEM=drm
WEDGED=1
DEVNAME=/dev/dri/card0
DEVTYPE=drm_minor
SEQNUM=5106
MAJOR=226
MINOR=0

Signed-off-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Co-developed-by: Raag Jadav <raag.jadav@intel.com>
Signed-off-by: Raag Jadav <raag.jadav@intel.com>
---
 drivers/gpu/drm/xe/xe_device.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c
index 1a0d7fdd094b..82b1583ada30 100644
--- a/drivers/gpu/drm/xe/xe_device.c
+++ b/drivers/gpu/drm/xe/xe_device.c
@@ -959,11 +959,12 @@ static void xe_device_wedged_fini(struct drm_device *drm, void *arg)
  * xe_device_declare_wedged - Declare device wedged
  * @xe: xe device instance
  *
- * This is a final state that can only be cleared with a mudule
+ * This is a final state that can only be cleared with a module
  * re-probe (unbind + bind).
  * In this state every IOCTL will be blocked so the GT cannot be used.
  * In general it will be called upon any critical error such as gt reset
- * failure or guc loading failure.
+ * failure or guc loading failure. Userspace will be notified of this state
+ * by a DRM uevent.
  * If xe.wedged module parameter is set to 2, this function will be called
  * on every single execution timeout (a.k.a. GPU hang) right after devcoredump
  * snapshot capture. In this mode, GT reset won't be attempted so the state of
@@ -993,6 +994,9 @@ void xe_device_declare_wedged(struct xe_device *xe)
 			"IOCTLs and executions are blocked. Only a rebind may clear the failure\n"
 			"Please file a _new_ bug report at https://gitlab.freedesktop.org/drm/xe/kernel/issues/new\n",
 			dev_name(xe->drm.dev));
+
+		/* Notify userspace of wedged device */
+		drm_dev_wedged(&xe->drm);
 	}
 
 	for_each_gt(gt, xe, id)
-- 
2.34.1

Re: [PATCH v3 2/3] drm/xe: Use device wedged event

[Aravind Iddamsetty]

On 02/09/24 13:18, Raag Jadav wrote:

This patch looks entirely new from what was sent earlier
so you could send it as a fresh patch.

Thanks,
Aravind,
> From: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
>
> This was previously attempted as xe specific reset uevent but dropped
> in commit 77a0d4d1cea2 ("drm/xe/uapi: Remove reset uevent for now")
> as part of refactoring.
>
> Now that we have device wedged event supported by DRM core, make use
> of it. With this in place userspace will be notified of wedged device,
> on the basis of which, userspace may take respective action to recover
> the device.
>
> $ udevadm monitor --property --kernel
> monitor will print the received events for:
> KERNEL - the kernel uevent
>
> KERNEL[307.420340] change   /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:01.0/0000:03:00.0/drm/card0 (drm)
> ACTION=change
> DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:01.0/0000:03:00.0/drm/card0
> SUBSYSTEM=drm
> WEDGED=1
> DEVNAME=/dev/dri/card0
> DEVTYPE=drm_minor
> SEQNUM=5106
> MAJOR=226
> MINOR=0
>
> Signed-off-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
> Co-developed-by: Raag Jadav <raag.jadav@intel.com>
> Signed-off-by: Raag Jadav <raag.jadav@intel.com>
> ---
>  drivers/gpu/drm/xe/xe_device.c | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c
> index 1a0d7fdd094b..82b1583ada30 100644
> --- a/drivers/gpu/drm/xe/xe_device.c
> +++ b/drivers/gpu/drm/xe/xe_device.c
> @@ -959,11 +959,12 @@ static void xe_device_wedged_fini(struct drm_device *drm, void *arg)
>   * xe_device_declare_wedged - Declare device wedged
>   * @xe: xe device instance
>   *
> - * This is a final state that can only be cleared with a mudule
> + * This is a final state that can only be cleared with a module
>   * re-probe (unbind + bind).
>   * In this state every IOCTL will be blocked so the GT cannot be used.
>   * In general it will be called upon any critical error such as gt reset
> - * failure or guc loading failure.
> + * failure or guc loading failure. Userspace will be notified of this state
> + * by a DRM uevent.
>   * If xe.wedged module parameter is set to 2, this function will be called
>   * on every single execution timeout (a.k.a. GPU hang) right after devcoredump
>   * snapshot capture. In this mode, GT reset won't be attempted so the state of
> @@ -993,6 +994,9 @@ void xe_device_declare_wedged(struct xe_device *xe)
>  			"IOCTLs and executions are blocked. Only a rebind may clear the failure\n"
>  			"Please file a _new_ bug report at https://gitlab.freedesktop.org/drm/xe/kernel/issues/new\n",
>  			dev_name(xe->drm.dev));
> +
> +		/* Notify userspace of wedged device */
> +		drm_dev_wedged(&xe->drm);
>  	}
>  
>  	for_each_gt(gt, xe, id)

[PATCH v3 3/3] drm/i915: Use device wedged event

[Raag Jadav]

Now that we have device wedged event supported by DRM core, make use
of it. With this in place, userspace will be notified of wedged device
on gt reset failure.

Signed-off-by: Raag Jadav <raag.jadav@intel.com>
---
 drivers/gpu/drm/i915/gt/intel_reset.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/gpu/drm/i915/gt/intel_reset.c b/drivers/gpu/drm/i915/gt/intel_reset.c
index 735cd23a43c6..60d09ec536c4 100644
--- a/drivers/gpu/drm/i915/gt/intel_reset.c
+++ b/drivers/gpu/drm/i915/gt/intel_reset.c
@@ -1409,6 +1409,8 @@ static void intel_gt_reset_global(struct intel_gt *gt,
 
 	if (!test_bit(I915_WEDGED, &gt->reset.flags))
 		kobject_uevent_env(kobj, KOBJ_CHANGE, reset_done_event);
+	else
+		drm_dev_wedged(&gt->i915->drm);
 }
 
 /**
-- 
2.34.1

Re: [PATCH v3 3/3] drm/i915: Use device wedged event

[Aravind Iddamsetty]

On 02/09/24 13:18, Raag Jadav wrote:
> Now that we have device wedged event supported by DRM core, make use
> of it. With this in place, userspace will be notified of wedged device
> on gt reset failure.
>
> Signed-off-by: Raag Jadav <raag.jadav@intel.com>
> ---
>  drivers/gpu/drm/i915/gt/intel_reset.c | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/drivers/gpu/drm/i915/gt/intel_reset.c b/drivers/gpu/drm/i915/gt/intel_reset.c
> index 735cd23a43c6..60d09ec536c4 100644
> --- a/drivers/gpu/drm/i915/gt/intel_reset.c
> +++ b/drivers/gpu/drm/i915/gt/intel_reset.c
> @@ -1409,6 +1409,8 @@ static void intel_gt_reset_global(struct intel_gt *gt,
>  
>  	if (!test_bit(I915_WEDGED, &gt->reset.flags))
>  		kobject_uevent_env(kobj, KOBJ_CHANGE, reset_done_event);
> +	else
> +		drm_dev_wedged(&gt->i915->drm);
>  }
rather than intel_gt_reset_global, __intel_get_set_wedged looks to be
an appropriate place where actually the device is declared wedged and
that would cover all call sites too.

Thanks,
Aravind.
>  
>  /**

Re: [PATCH v3 3/3] drm/i915: Use device wedged event

[Raag Jadav]

On Mon, Sep 02, 2024 at 02:22:21PM +0530, Aravind Iddamsetty wrote:
> 
> On 02/09/24 13:18, Raag Jadav wrote:
> > Now that we have device wedged event supported by DRM core, make use
> > of it. With this in place, userspace will be notified of wedged device
> > on gt reset failure.
> >
> > Signed-off-by: Raag Jadav <raag.jadav@intel.com>
> > ---
> >  drivers/gpu/drm/i915/gt/intel_reset.c | 2 ++
> >  1 file changed, 2 insertions(+)
> >
> > diff --git a/drivers/gpu/drm/i915/gt/intel_reset.c b/drivers/gpu/drm/i915/gt/intel_reset.c
> > index 735cd23a43c6..60d09ec536c4 100644
> > --- a/drivers/gpu/drm/i915/gt/intel_reset.c
> > +++ b/drivers/gpu/drm/i915/gt/intel_reset.c
> > @@ -1409,6 +1409,8 @@ static void intel_gt_reset_global(struct intel_gt *gt,
> >  
> >  	if (!test_bit(I915_WEDGED, &gt->reset.flags))
> >  		kobject_uevent_env(kobj, KOBJ_CHANGE, reset_done_event);
> > +	else
> > +		drm_dev_wedged(&gt->i915->drm);
> >  }
> rather than intel_gt_reset_global, __intel_get_set_wedged looks to be
> an appropriate place where actually the device is declared wedged and
> that would cover all call sites too.

Which is why it may be the appropriate place IMHO.
We'd want to make sure the device is _really_ unrecoverable when we
choose to send the event.

Raag