[PATCH v9 0/4] Introduce DRM device wedged event

[PATCH v9 0/4] Introduce DRM device wedged event

[Raag Jadav]

This series introduces device wedged event in DRM subsystem and uses
it in xe and i915 drivers. Detailed description in commit message.

This was earlier attempted as xe specific uevent in v1 and v2.
https://patchwork.freedesktop.org/series/136909/

Similar work by André Almeida.
https://lore.kernel.org/dri-devel/20221125175203.52481-1-andrealmeid@igalia.com/

v2: Change authorship to Himal (Aravind)
    Add uevent for all device wedged cases (Aravind)

v3: Generic re-implementation in DRM subsystem (Lucas)

v4: s/drm_dev_wedged/drm_dev_wedged_event
    Use drm_info() (Jani)
    Kernel doc adjustment (Aravind)
    Change authorship to Raag (Aravind)

v5: Send recovery method with uevent (Lina)
    Expose supported recovery methods via sysfs (Lucas)

v6: Access wedge_recovery_opts[] using helper function (Jani)
    Use snprintf() (Jani)

v7: Convert recovery helpers into regular functions (Andy, Jani)
    Aesthetic adjustments (Andy)
    Handle invalid method cases
    Add documentation to drm-uapi.rst (Sima)

v8: Drop sysfs and allow sending multiple methods with uevent (Lucas, Michal)
    Improve documentation (Christian, Rodrigo)
    static_assert() globally (Andy)

v9: Document prerequisites section (Christian)
    Provide 'none' method for reset cases (Christian)
    Provide recovery opts using switch cases

Raag Jadav (4):
  drm: Introduce device wedged event
  drm/doc: Document device wedged event
  drm/xe: Use device wedged event
  drm/i915: Use device wedged event

 Documentation/gpu/drm-uapi.rst        | 102 +++++++++++++++++++++++++-
 drivers/gpu/drm/drm_drv.c             |  63 ++++++++++++++++
 drivers/gpu/drm/i915/gt/intel_reset.c |   3 +
 drivers/gpu/drm/xe/xe_device.c        |   9 ++-
 include/drm/drm_device.h              |   8 ++
 include/drm/drm_drv.h                 |   1 +
 6 files changed, 181 insertions(+), 5 deletions(-)

-- 
2.34.1

[PATCH v9 1/4] drm: Introduce device wedged event

[Raag Jadav]

Introduce device wedged event, which notifies userspace of 'wedged'
(hanged/unusable) state of the DRM device through a uevent. This is
useful especially in cases where the device is no longer operating as
expected and has become unrecoverable from driver context. Purpose of
this implementation is to provide drivers a generic way to recover with
the help of userspace intervention without taking any drastic measures
in the driver.

A 'wedged' device is basically a dead device that needs attention. The
uevent is the notification that is sent to userspace along with a hint
about what could possibly be attempted to recover the device and bring
it back to usable state. Different drivers may have different ideas of
a 'wedged' device depending on their hardware implementation, and hence
the vendor agnostic nature of the event. It is up to the drivers to
decide when they see the need for recovery and how they want to recover
from the available methods.

Prerequisites
-------------

The driver, before opting for recovery, needs to make sure that the
'wedged' device doesn't harm the system as a whole by taking care of the
prerequisites. Necessary actions must include disabling DMA to system
memory as well as any communication channels with other devices. Further,
the driver must ensure that all dma_fences are signalled and any device
state that the core kernel might depend on are cleaned up. Once the event
is sent, the device must be kept in 'wedged' state until the recovery is
performed. New accesses to the device (IOCTLs) should be blocked,
preferably with an error code that resembles the type of failure the
device has encountered. This will signify the reason for wegeding which
can be reported to the application if needed.

Recovery
--------

Current implementation defines three recovery methods, out of which,
drivers can use any one, multiple or none. Method(s) of choice will be
sent in the uevent environment as ``WEDGED=<method1>[,<method2>]`` in
order of less to more side-effects. If driver is unsure about recovery
or method is unknown (like soft/hard reboot, firmware flashing, hardware
replacement or any other procedure which can't be attempted on the fly),
``WEDGED=unknown`` will be sent instead.

Userspace consumers can parse this event and attempt recovery as per the
following expectations.

    =============== ================================
    Recovery method Consumer expectations
    =============== ================================
    none            optional telemetry collection
    rebind          unbind + bind driver
    bus-reset       unbind + reset bus device + bind
    unknown         admin/user policy
    =============== ================================

The only exception to this is ``WEDGED=none``, which signifies that the
device was temporarily 'wedged' at some point but was able to recover
using device specific methods like reset. No explicit action is expected
from userspace consumers in this case, but they can still take additional
steps like gathering telemetry information (devcoredump, syslog). This is
useful because the first hang is usually the most critical one which can
result in consequential hangs or complete wedging.

Example
-------

Udev rule::

    SUBSYSTEM=="drm", ENV{WEDGED}=="rebind", DEVPATH=="*/drm/card[0-9]",
    RUN+="/path/to/rebind.sh $env{DEVPATH}"

Recovery script::

    #!/bin/sh

    DEVPATH=$(readlink -f /sys/$1/device)
    DEVICE=$(basename $DEVPATH)
    DRIVER=$(readlink -f $DEVPATH/driver)

    echo -n $DEVICE > $DRIVER/unbind
    sleep 1
    echo -n $DEVICE > $DRIVER/bind

Customization
-------------

Although basic recovery is possible with a simple script, admin/users can
define custom policies around recovery action. For example, if the driver
supports multiple recovery methods, consumers can opt for the suitable one
based on policy definition. Consumers can also choose to have the device
available for debugging or additional data collection before performing
the recovery. This is useful especially when the driver is unsure about
recovery or method is unknown.

v4: s/drm_dev_wedged/drm_dev_wedged_event
    Use drm_info() (Jani)
    Kernel doc adjustment (Aravind)
v5: Send recovery method with uevent (Lina)
v6: Access wedge_recovery_opts[] using helper function (Jani)
    Use snprintf() (Jani)
v7: Convert recovery helpers into regular functions (Andy, Jani)
    Aesthetic adjustments (Andy)
    Handle invalid method cases
v8: Allow sending multiple methods with uevent (Lucas, Michal)
    static_assert() globally (Andy)
v9: Provide 'none' method for reset cases (Christian)
    Provide recovery opts using switch cases

Signed-off-by: Raag Jadav <raag.jadav@intel.com>
---
 drivers/gpu/drm/drm_drv.c | 63 +++++++++++++++++++++++++++++++++++++++
 include/drm/drm_device.h  |  8 +++++
 include/drm/drm_drv.h     |  1 +
 3 files changed, 72 insertions(+)

diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c
index c2c172eb25df..115e1d1c80ea 100644
--- a/drivers/gpu/drm/drm_drv.c
+++ b/drivers/gpu/drm/drm_drv.c
@@ -26,6 +26,7 @@
  * DEALINGS IN THE SOFTWARE.
  */
 
+#include <linux/bitops.h>
 #include <linux/debugfs.h>
 #include <linux/fs.h>
 #include <linux/module.h>
@@ -33,6 +34,7 @@
 #include <linux/mount.h>
 #include <linux/pseudo_fs.h>
 #include <linux/slab.h>
+#include <linux/sprintf.h>
 #include <linux/srcu.h>
 #include <linux/xarray.h>
 
@@ -497,6 +499,67 @@ void drm_dev_unplug(struct drm_device *dev)
 }
 EXPORT_SYMBOL(drm_dev_unplug);
 
+/*
+ * Available recovery methods for wedged device. To be sent along with device
+ * wedged uevent.
+ */
+static const char *drm_get_wedge_recovery(unsigned int opt)
+{
+	switch (BIT(opt)) {
+	case DRM_WEDGE_RECOVERY_NONE:
+		return "none";
+	case DRM_WEDGE_RECOVERY_REBIND:
+		return "rebind";
+	case DRM_WEDGE_RECOVERY_BUS_RESET:
+		return "bus-reset";
+	default:
+		return NULL;
+	}
+}
+
+/**
+ * drm_dev_wedged_event - generate a device wedged uevent
+ * @dev: DRM device
+ * @method: method(s) to be used for recovery
+ *
+ * This generates a device wedged uevent for the DRM device specified by @dev.
+ * Recovery @method\(s) of choice will be sent in the uevent environment as
+ * ``WEDGED=<method1>[,<method2>]`` in order of less to more side-effects.
+ * If caller is unsure about recovery or @method is unknown (0),
+ * ``WEDGED=unknown`` will be sent instead.
+ *
+ * Returns: 0 on success, negative error code otherwise.
+ */
+int drm_dev_wedged_event(struct drm_device *dev, unsigned long method)
+{
+	const char *recovery = NULL;
+	unsigned int len, opt;
+	/* Event string length up to 28+ characters with available methods */
+	char event_string[32];
+	char *envp[] = { event_string, NULL };
+
+	len = scnprintf(event_string, sizeof(event_string), "%s", "WEDGED=");
+
+	for_each_set_bit(opt, &method, BITS_PER_TYPE(method)) {
+		recovery = drm_get_wedge_recovery(opt);
+		if (drm_WARN(dev, !recovery, "device wedged, invalid recovery method %u\n", opt))
+			break;
+
+		len += scnprintf(event_string + len, sizeof(event_string), "%s,", recovery);
+	}
+
+	if (recovery)
+		/* Get rid of trailing comma */
+		event_string[len - 1] = '\0';
+	else
+		/* Caller is unsure about recovery, do the best we can at this point. */
+		snprintf(event_string, sizeof(event_string), "%s", "WEDGED=unknown");
+
+	drm_info(dev, "device wedged, needs recovery\n");
+	return kobject_uevent_env(&dev->primary->kdev->kobj, KOBJ_CHANGE, envp);
+}
+EXPORT_SYMBOL(drm_dev_wedged_event);
+
 /*
  * DRM internal mount
  * We want to be able to allocate our own "struct address_space" to control
diff --git a/include/drm/drm_device.h b/include/drm/drm_device.h
index c91f87b5242d..6ea54a578cda 100644
--- a/include/drm/drm_device.h
+++ b/include/drm/drm_device.h
@@ -21,6 +21,14 @@ struct inode;
 struct pci_dev;
 struct pci_controller;
 
+/*
+ * Recovery methods for wedged device in order of less to more side-effects.
+ * To be used with drm_dev_wedged_event() as recovery @method. Callers can
+ * use any one, multiple (or'd) or none depending on their needs.
+ */
+#define DRM_WEDGE_RECOVERY_NONE		BIT(0)	/* optional telemetry collection */
+#define DRM_WEDGE_RECOVERY_REBIND	BIT(1)	/* unbind + bind driver */
+#define DRM_WEDGE_RECOVERY_BUS_RESET	BIT(2)	/* unbind + reset bus device + bind */
 
 /**
  * enum switch_power_state - power state of drm device
diff --git a/include/drm/drm_drv.h b/include/drm/drm_drv.h
index 1bbbcb8e2d23..f41a82839e28 100644
--- a/include/drm/drm_drv.h
+++ b/include/drm/drm_drv.h
@@ -479,6 +479,7 @@ void drm_put_dev(struct drm_device *dev);
 bool drm_dev_enter(struct drm_device *dev, int *idx);
 void drm_dev_exit(int idx);
 void drm_dev_unplug(struct drm_device *dev);
+int drm_dev_wedged_event(struct drm_device *dev, unsigned long method);
 
 /**
  * drm_dev_is_unplugged - is a DRM device unplugged
-- 
2.34.1

Re: [PATCH v9 1/4] drm: Introduce device wedged event

[Aravind Iddamsetty]

On 15/11/24 10:37, Raag Jadav wrote:
> Introduce device wedged event, which notifies userspace of 'wedged'
> (hanged/unusable) state of the DRM device through a uevent. This is
> useful especially in cases where the device is no longer operating as
> expected and has become unrecoverable from driver context. Purpose of
> this implementation is to provide drivers a generic way to recover with
> the help of userspace intervention without taking any drastic measures
> in the driver.
>
> A 'wedged' device is basically a dead device that needs attention. The
> uevent is the notification that is sent to userspace along with a hint
> about what could possibly be attempted to recover the device and bring
> it back to usable state. Different drivers may have different ideas of
> a 'wedged' device depending on their hardware implementation, and hence
> the vendor agnostic nature of the event. It is up to the drivers to
> decide when they see the need for recovery and how they want to recover
> from the available methods.
>
> Prerequisites
> -------------
>
> The driver, before opting for recovery, needs to make sure that the
> 'wedged' device doesn't harm the system as a whole by taking care of the
> prerequisites. Necessary actions must include disabling DMA to system
> memory as well as any communication channels with other devices. Further,
> the driver must ensure that all dma_fences are signalled and any device
> state that the core kernel might depend on are cleaned up. Once the event
> is sent, the device must be kept in 'wedged' state until the recovery is
> performed. New accesses to the device (IOCTLs) should be blocked,
> preferably with an error code that resembles the type of failure the
> device has encountered. This will signify the reason for wegeding which
> can be reported to the application if needed.

should we even drop the mmaps we created?

Thanks,
Aravind.
>
> Recovery
> --------
>
> Current implementation defines three recovery methods, out of which,
> drivers can use any one, multiple or none. Method(s) of choice will be
> sent in the uevent environment as ``WEDGED=<method1>[,<method2>]`` in
> order of less to more side-effects. If driver is unsure about recovery
> or method is unknown (like soft/hard reboot, firmware flashing, hardware
> replacement or any other procedure which can't be attempted on the fly),
> ``WEDGED=unknown`` will be sent instead.
>
> Userspace consumers can parse this event and attempt recovery as per the
> following expectations.
>
>     =============== ================================
>     Recovery method Consumer expectations
>     =============== ================================
>     none            optional telemetry collection
>     rebind          unbind + bind driver
>     bus-reset       unbind + reset bus device + bind
>     unknown         admin/user policy
>     =============== ================================
>
> The only exception to this is ``WEDGED=none``, which signifies that the
> device was temporarily 'wedged' at some point but was able to recover
> using device specific methods like reset. No explicit action is expected
> from userspace consumers in this case, but they can still take additional
> steps like gathering telemetry information (devcoredump, syslog). This is
> useful because the first hang is usually the most critical one which can
> result in consequential hangs or complete wedging.
>
> Example
> -------
>
> Udev rule::
>
>     SUBSYSTEM=="drm", ENV{WEDGED}=="rebind", DEVPATH=="*/drm/card[0-9]",
>     RUN+="/path/to/rebind.sh $env{DEVPATH}"
>
> Recovery script::
>
>     #!/bin/sh
>
>     DEVPATH=$(readlink -f /sys/$1/device)
>     DEVICE=$(basename $DEVPATH)
>     DRIVER=$(readlink -f $DEVPATH/driver)
>
>     echo -n $DEVICE > $DRIVER/unbind
>     sleep 1
>     echo -n $DEVICE > $DRIVER/bind
>
> Customization
> -------------
>
> Although basic recovery is possible with a simple script, admin/users can
> define custom policies around recovery action. For example, if the driver
> supports multiple recovery methods, consumers can opt for the suitable one
> based on policy definition. Consumers can also choose to have the device
> available for debugging or additional data collection before performing
> the recovery. This is useful especially when the driver is unsure about
> recovery or method is unknown.
>
> v4: s/drm_dev_wedged/drm_dev_wedged_event
>     Use drm_info() (Jani)
>     Kernel doc adjustment (Aravind)
> v5: Send recovery method with uevent (Lina)
> v6: Access wedge_recovery_opts[] using helper function (Jani)
>     Use snprintf() (Jani)
> v7: Convert recovery helpers into regular functions (Andy, Jani)
>     Aesthetic adjustments (Andy)
>     Handle invalid method cases
> v8: Allow sending multiple methods with uevent (Lucas, Michal)
>     static_assert() globally (Andy)
> v9: Provide 'none' method for reset cases (Christian)
>     Provide recovery opts using switch cases
>
> Signed-off-by: Raag Jadav <raag.jadav@intel.com>
> ---
>  drivers/gpu/drm/drm_drv.c | 63 +++++++++++++++++++++++++++++++++++++++
>  include/drm/drm_device.h  |  8 +++++
>  include/drm/drm_drv.h     |  1 +
>  3 files changed, 72 insertions(+)
>
> diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c
> index c2c172eb25df..115e1d1c80ea 100644
> --- a/drivers/gpu/drm/drm_drv.c
> +++ b/drivers/gpu/drm/drm_drv.c
> @@ -26,6 +26,7 @@
>   * DEALINGS IN THE SOFTWARE.
>   */
>  
> +#include <linux/bitops.h>
>  #include <linux/debugfs.h>
>  #include <linux/fs.h>
>  #include <linux/module.h>
> @@ -33,6 +34,7 @@
>  #include <linux/mount.h>
>  #include <linux/pseudo_fs.h>
>  #include <linux/slab.h>
> +#include <linux/sprintf.h>
>  #include <linux/srcu.h>
>  #include <linux/xarray.h>
>  
> @@ -497,6 +499,67 @@ void drm_dev_unplug(struct drm_device *dev)
>  }
>  EXPORT_SYMBOL(drm_dev_unplug);
>  
> +/*
> + * Available recovery methods for wedged device. To be sent along with device
> + * wedged uevent.
> + */
> +static const char *drm_get_wedge_recovery(unsigned int opt)
> +{
> +	switch (BIT(opt)) {
> +	case DRM_WEDGE_RECOVERY_NONE:
> +		return "none";
> +	case DRM_WEDGE_RECOVERY_REBIND:
> +		return "rebind";
> +	case DRM_WEDGE_RECOVERY_BUS_RESET:
> +		return "bus-reset";
> +	default:
> +		return NULL;
> +	}
> +}
> +
> +/**
> + * drm_dev_wedged_event - generate a device wedged uevent
> + * @dev: DRM device
> + * @method: method(s) to be used for recovery
> + *
> + * This generates a device wedged uevent for the DRM device specified by @dev.
> + * Recovery @method\(s) of choice will be sent in the uevent environment as
> + * ``WEDGED=<method1>[,<method2>]`` in order of less to more side-effects.
> + * If caller is unsure about recovery or @method is unknown (0),
> + * ``WEDGED=unknown`` will be sent instead.
> + *
> + * Returns: 0 on success, negative error code otherwise.
> + */
> +int drm_dev_wedged_event(struct drm_device *dev, unsigned long method)
> +{
> +	const char *recovery = NULL;
> +	unsigned int len, opt;
> +	/* Event string length up to 28+ characters with available methods */
> +	char event_string[32];
> +	char *envp[] = { event_string, NULL };
> +
> +	len = scnprintf(event_string, sizeof(event_string), "%s", "WEDGED=");
> +
> +	for_each_set_bit(opt, &method, BITS_PER_TYPE(method)) {
> +		recovery = drm_get_wedge_recovery(opt);
> +		if (drm_WARN(dev, !recovery, "device wedged, invalid recovery method %u\n", opt))
> +			break;
> +
> +		len += scnprintf(event_string + len, sizeof(event_string), "%s,", recovery);
> +	}
> +
> +	if (recovery)
> +		/* Get rid of trailing comma */
> +		event_string[len - 1] = '\0';
> +	else
> +		/* Caller is unsure about recovery, do the best we can at this point. */
> +		snprintf(event_string, sizeof(event_string), "%s", "WEDGED=unknown");
> +
> +	drm_info(dev, "device wedged, needs recovery\n");
> +	return kobject_uevent_env(&dev->primary->kdev->kobj, KOBJ_CHANGE, envp);
> +}
> +EXPORT_SYMBOL(drm_dev_wedged_event);
> +
>  /*
>   * DRM internal mount
>   * We want to be able to allocate our own "struct address_space" to control
> diff --git a/include/drm/drm_device.h b/include/drm/drm_device.h
> index c91f87b5242d..6ea54a578cda 100644
> --- a/include/drm/drm_device.h
> +++ b/include/drm/drm_device.h
> @@ -21,6 +21,14 @@ struct inode;
>  struct pci_dev;
>  struct pci_controller;
>  
> +/*
> + * Recovery methods for wedged device in order of less to more side-effects.
> + * To be used with drm_dev_wedged_event() as recovery @method. Callers can
> + * use any one, multiple (or'd) or none depending on their needs.
> + */
> +#define DRM_WEDGE_RECOVERY_NONE		BIT(0)	/* optional telemetry collection */
> +#define DRM_WEDGE_RECOVERY_REBIND	BIT(1)	/* unbind + bind driver */
> +#define DRM_WEDGE_RECOVERY_BUS_RESET	BIT(2)	/* unbind + reset bus device + bind */
>  
>  /**
>   * enum switch_power_state - power state of drm device
> diff --git a/include/drm/drm_drv.h b/include/drm/drm_drv.h
> index 1bbbcb8e2d23..f41a82839e28 100644
> --- a/include/drm/drm_drv.h
> +++ b/include/drm/drm_drv.h
> @@ -479,6 +479,7 @@ void drm_put_dev(struct drm_device *dev);
>  bool drm_dev_enter(struct drm_device *dev, int *idx);
>  void drm_dev_exit(int idx);
>  void drm_dev_unplug(struct drm_device *dev);
> +int drm_dev_wedged_event(struct drm_device *dev, unsigned long method);
>  
>  /**
>   * drm_dev_is_unplugged - is a DRM device unplugged

Re: [PATCH v9 1/4] drm: Introduce device wedged event

[Raag Jadav]

On Mon, Nov 18, 2024 at 08:26:37PM +0530, Aravind Iddamsetty wrote:
> On 15/11/24 10:37, Raag Jadav wrote:
> > Introduce device wedged event, which notifies userspace of 'wedged'
> > (hanged/unusable) state of the DRM device through a uevent. This is
> > useful especially in cases where the device is no longer operating as
> > expected and has become unrecoverable from driver context. Purpose of
> > this implementation is to provide drivers a generic way to recover with
> > the help of userspace intervention without taking any drastic measures
> > in the driver.
> >
> > A 'wedged' device is basically a dead device that needs attention. The
> > uevent is the notification that is sent to userspace along with a hint
> > about what could possibly be attempted to recover the device and bring
> > it back to usable state. Different drivers may have different ideas of
> > a 'wedged' device depending on their hardware implementation, and hence
> > the vendor agnostic nature of the event. It is up to the drivers to
> > decide when they see the need for recovery and how they want to recover
> > from the available methods.
> >
> > Prerequisites
> > -------------
> >
> > The driver, before opting for recovery, needs to make sure that the
> > 'wedged' device doesn't harm the system as a whole by taking care of the
> > prerequisites. Necessary actions must include disabling DMA to system
> > memory as well as any communication channels with other devices. Further,
> > the driver must ensure that all dma_fences are signalled and any device
> > state that the core kernel might depend on are cleaned up. Once the event
> > is sent, the device must be kept in 'wedged' state until the recovery is
> > performed. New accesses to the device (IOCTLs) should be blocked,
> > preferably with an error code that resembles the type of failure the
> > device has encountered. This will signify the reason for wegeding which
> > can be reported to the application if needed.
> 
> should we even drop the mmaps we created?

Whatever is required for a clean recovery, yes.

Although how would this play out? Do we risk loosing display?
Or any other possible side-effects?

Raag

Re: [PATCH v9 1/4] drm: Introduce device wedged event

[Christian König]

Am 22.11.24 um 08:07 schrieb Raag Jadav:
> On Mon, Nov 18, 2024 at 08:26:37PM +0530, Aravind Iddamsetty wrote:
>> On 15/11/24 10:37, Raag Jadav wrote:
>>> Introduce device wedged event, which notifies userspace of 'wedged'
>>> (hanged/unusable) state of the DRM device through a uevent. This is
>>> useful especially in cases where the device is no longer operating as
>>> expected and has become unrecoverable from driver context. Purpose of
>>> this implementation is to provide drivers a generic way to recover with
>>> the help of userspace intervention without taking any drastic measures
>>> in the driver.
>>>
>>> A 'wedged' device is basically a dead device that needs attention. The
>>> uevent is the notification that is sent to userspace along with a hint
>>> about what could possibly be attempted to recover the device and bring
>>> it back to usable state. Different drivers may have different ideas of
>>> a 'wedged' device depending on their hardware implementation, and hence
>>> the vendor agnostic nature of the event. It is up to the drivers to
>>> decide when they see the need for recovery and how they want to recover
>>> from the available methods.
>>>
>>> Prerequisites
>>> -------------
>>>
>>> The driver, before opting for recovery, needs to make sure that the
>>> 'wedged' device doesn't harm the system as a whole by taking care of the
>>> prerequisites. Necessary actions must include disabling DMA to system
>>> memory as well as any communication channels with other devices. Further,
>>> the driver must ensure that all dma_fences are signalled and any device
>>> state that the core kernel might depend on are cleaned up. Once the event
>>> is sent, the device must be kept in 'wedged' state until the recovery is
>>> performed. New accesses to the device (IOCTLs) should be blocked,
>>> preferably with an error code that resembles the type of failure the
>>> device has encountered. This will signify the reason for wegeding which
>>> can be reported to the application if needed.
>> should we even drop the mmaps we created?
> Whatever is required for a clean recovery, yes.
>
> Although how would this play out? Do we risk loosing display?
> Or any other possible side-effects?

Before sending a wedge event all DMA transfers of the device have to be 
blocked.

So yes, all display, mmap() and file descriptor connections you had with 
the device would need to be re-created.

Regards,
Christian.

>
> Raag

Re: [PATCH v9 1/4] drm: Introduce device wedged event

[Raag Jadav]

On Fri, Nov 22, 2024 at 11:09:32AM +0100, Christian König wrote:
> Am 22.11.24 um 08:07 schrieb Raag Jadav:
> > On Mon, Nov 18, 2024 at 08:26:37PM +0530, Aravind Iddamsetty wrote:
> > > On 15/11/24 10:37, Raag Jadav wrote:
> > > > Introduce device wedged event, which notifies userspace of 'wedged'
> > > > (hanged/unusable) state of the DRM device through a uevent. This is
> > > > useful especially in cases where the device is no longer operating as
> > > > expected and has become unrecoverable from driver context. Purpose of
> > > > this implementation is to provide drivers a generic way to recover with
> > > > the help of userspace intervention without taking any drastic measures
> > > > in the driver.
> > > > 
> > > > A 'wedged' device is basically a dead device that needs attention. The
> > > > uevent is the notification that is sent to userspace along with a hint
> > > > about what could possibly be attempted to recover the device and bring
> > > > it back to usable state. Different drivers may have different ideas of
> > > > a 'wedged' device depending on their hardware implementation, and hence
> > > > the vendor agnostic nature of the event. It is up to the drivers to
> > > > decide when they see the need for recovery and how they want to recover
> > > > from the available methods.
> > > > 
> > > > Prerequisites
> > > > -------------
> > > > 
> > > > The driver, before opting for recovery, needs to make sure that the
> > > > 'wedged' device doesn't harm the system as a whole by taking care of the
> > > > prerequisites. Necessary actions must include disabling DMA to system
> > > > memory as well as any communication channels with other devices. Further,
> > > > the driver must ensure that all dma_fences are signalled and any device
> > > > state that the core kernel might depend on are cleaned up. Once the event
> > > > is sent, the device must be kept in 'wedged' state until the recovery is
> > > > performed. New accesses to the device (IOCTLs) should be blocked,
> > > > preferably with an error code that resembles the type of failure the
> > > > device has encountered. This will signify the reason for wegeding which
> > > > can be reported to the application if needed.
> > > should we even drop the mmaps we created?
> > Whatever is required for a clean recovery, yes.
> > 
> > Although how would this play out? Do we risk loosing display?
> > Or any other possible side-effects?
> 
> Before sending a wedge event all DMA transfers of the device have to be
> blocked.
> 
> So yes, all display, mmap() and file descriptor connections you had with the
> device would need to be re-created.

Does it mean we'd have to rely on userspace to unmap()?

Raag

Re: [PATCH v9 1/4] drm: Introduce device wedged event

[Aravind Iddamsetty]

On 22/11/24 21:32, Raag Jadav wrote:
> On Fri, Nov 22, 2024 at 11:09:32AM +0100, Christian König wrote:
>> Am 22.11.24 um 08:07 schrieb Raag Jadav:
>>> On Mon, Nov 18, 2024 at 08:26:37PM +0530, Aravind Iddamsetty wrote:
>>>> On 15/11/24 10:37, Raag Jadav wrote:
>>>>> Introduce device wedged event, which notifies userspace of 'wedged'
>>>>> (hanged/unusable) state of the DRM device through a uevent. This is
>>>>> useful especially in cases where the device is no longer operating as
>>>>> expected and has become unrecoverable from driver context. Purpose of
>>>>> this implementation is to provide drivers a generic way to recover with
>>>>> the help of userspace intervention without taking any drastic measures
>>>>> in the driver.
>>>>>
>>>>> A 'wedged' device is basically a dead device that needs attention. The
>>>>> uevent is the notification that is sent to userspace along with a hint
>>>>> about what could possibly be attempted to recover the device and bring
>>>>> it back to usable state. Different drivers may have different ideas of
>>>>> a 'wedged' device depending on their hardware implementation, and hence
>>>>> the vendor agnostic nature of the event. It is up to the drivers to
>>>>> decide when they see the need for recovery and how they want to recover
>>>>> from the available methods.
>>>>>
>>>>> Prerequisites
>>>>> -------------
>>>>>
>>>>> The driver, before opting for recovery, needs to make sure that the
>>>>> 'wedged' device doesn't harm the system as a whole by taking care of the
>>>>> prerequisites. Necessary actions must include disabling DMA to system
>>>>> memory as well as any communication channels with other devices. Further,
>>>>> the driver must ensure that all dma_fences are signalled and any device
>>>>> state that the core kernel might depend on are cleaned up. Once the event
>>>>> is sent, the device must be kept in 'wedged' state until the recovery is
>>>>> performed. New accesses to the device (IOCTLs) should be blocked,
>>>>> preferably with an error code that resembles the type of failure the
>>>>> device has encountered. This will signify the reason for wegeding which
>>>>> can be reported to the application if needed.
>>>> should we even drop the mmaps we created?
>>> Whatever is required for a clean recovery, yes.
>>>
>>> Although how would this play out? Do we risk loosing display?
>>> Or any other possible side-effects?
>> Before sending a wedge event all DMA transfers of the device have to be
>> blocked.
>>
>> So yes, all display, mmap() and file descriptor connections you had with the
>> device would need to be re-created.
> Does it mean we'd have to rely on userspace to unmap()?


I'm not sure of display, but at least all user mappings can be destroyed
using drm_vma_node_unmap.

Thanks,
Aravind.
>
> Raag

Re: [PATCH v9 1/4] drm: Introduce device wedged event

[Christian König]

Am 25.11.24 um 06:56 schrieb Aravind Iddamsetty:
> On 22/11/24 21:32, Raag Jadav wrote:
>> On Fri, Nov 22, 2024 at 11:09:32AM +0100, Christian König wrote:
>>> Am 22.11.24 um 08:07 schrieb Raag Jadav:
>>>> On Mon, Nov 18, 2024 at 08:26:37PM +0530, Aravind Iddamsetty wrote:
>>>>> On 15/11/24 10:37, Raag Jadav wrote:
>>>>>> Introduce device wedged event, which notifies userspace of 'wedged'
>>>>>> (hanged/unusable) state of the DRM device through a uevent. This is
>>>>>> useful especially in cases where the device is no longer operating as
>>>>>> expected and has become unrecoverable from driver context. Purpose of
>>>>>> this implementation is to provide drivers a generic way to recover with
>>>>>> the help of userspace intervention without taking any drastic measures
>>>>>> in the driver.
>>>>>>
>>>>>> A 'wedged' device is basically a dead device that needs attention. The
>>>>>> uevent is the notification that is sent to userspace along with a hint
>>>>>> about what could possibly be attempted to recover the device and bring
>>>>>> it back to usable state. Different drivers may have different ideas of
>>>>>> a 'wedged' device depending on their hardware implementation, and hence
>>>>>> the vendor agnostic nature of the event. It is up to the drivers to
>>>>>> decide when they see the need for recovery and how they want to recover
>>>>>> from the available methods.
>>>>>>
>>>>>> Prerequisites
>>>>>> -------------
>>>>>>
>>>>>> The driver, before opting for recovery, needs to make sure that the
>>>>>> 'wedged' device doesn't harm the system as a whole by taking care of the
>>>>>> prerequisites. Necessary actions must include disabling DMA to system
>>>>>> memory as well as any communication channels with other devices. Further,
>>>>>> the driver must ensure that all dma_fences are signalled and any device
>>>>>> state that the core kernel might depend on are cleaned up. Once the event
>>>>>> is sent, the device must be kept in 'wedged' state until the recovery is
>>>>>> performed. New accesses to the device (IOCTLs) should be blocked,
>>>>>> preferably with an error code that resembles the type of failure the
>>>>>> device has encountered. This will signify the reason for wegeding which
>>>>>> can be reported to the application if needed.
>>>>> should we even drop the mmaps we created?
>>>> Whatever is required for a clean recovery, yes.
>>>>
>>>> Although how would this play out? Do we risk loosing display?
>>>> Or any other possible side-effects?
>>> Before sending a wedge event all DMA transfers of the device have to be
>>> blocked.
>>>
>>> So yes, all display, mmap() and file descriptor connections you had with the
>>> device would need to be re-created.
>> Does it mean we'd have to rely on userspace to unmap()?
>
> I'm not sure of display, but at least all user mappings can be destroyed
> using drm_vma_node_unmap.

That is not really correct. The mappings are not destroy, they are 
invalidated.

On access a page fault is generated and TTM should redirect the access 
to the dummy page.

The userspace application still has to unmap the VMA to destroy it.

Regards,
Christian.

>
> Thanks,
> Aravind.
>> Raag

Re: [PATCH v9 1/4] drm: Introduce device wedged event

[Christian König]

Am 22.11.24 um 17:02 schrieb Raag Jadav:
> On Fri, Nov 22, 2024 at 11:09:32AM +0100, Christian König wrote:
>> Am 22.11.24 um 08:07 schrieb Raag Jadav:
>>> On Mon, Nov 18, 2024 at 08:26:37PM +0530, Aravind Iddamsetty wrote:
>>>> On 15/11/24 10:37, Raag Jadav wrote:
>>>>> Introduce device wedged event, which notifies userspace of 'wedged'
>>>>> (hanged/unusable) state of the DRM device through a uevent. This is
>>>>> useful especially in cases where the device is no longer operating as
>>>>> expected and has become unrecoverable from driver context. Purpose of
>>>>> this implementation is to provide drivers a generic way to recover with
>>>>> the help of userspace intervention without taking any drastic measures
>>>>> in the driver.
>>>>>
>>>>> A 'wedged' device is basically a dead device that needs attention. The
>>>>> uevent is the notification that is sent to userspace along with a hint
>>>>> about what could possibly be attempted to recover the device and bring
>>>>> it back to usable state. Different drivers may have different ideas of
>>>>> a 'wedged' device depending on their hardware implementation, and hence
>>>>> the vendor agnostic nature of the event. It is up to the drivers to
>>>>> decide when they see the need for recovery and how they want to recover
>>>>> from the available methods.
>>>>>
>>>>> Prerequisites
>>>>> -------------
>>>>>
>>>>> The driver, before opting for recovery, needs to make sure that the
>>>>> 'wedged' device doesn't harm the system as a whole by taking care of the
>>>>> prerequisites. Necessary actions must include disabling DMA to system
>>>>> memory as well as any communication channels with other devices. Further,
>>>>> the driver must ensure that all dma_fences are signalled and any device
>>>>> state that the core kernel might depend on are cleaned up. Once the event
>>>>> is sent, the device must be kept in 'wedged' state until the recovery is
>>>>> performed. New accesses to the device (IOCTLs) should be blocked,
>>>>> preferably with an error code that resembles the type of failure the
>>>>> device has encountered. This will signify the reason for wegeding which
>>>>> can be reported to the application if needed.
>>>> should we even drop the mmaps we created?
>>> Whatever is required for a clean recovery, yes.
>>>
>>> Although how would this play out? Do we risk loosing display?
>>> Or any other possible side-effects?
>> Before sending a wedge event all DMA transfers of the device have to be
>> blocked.
>>
>> So yes, all display, mmap() and file descriptor connections you had with the
>> device would need to be re-created.
> Does it mean we'd have to rely on userspace to unmap()?

Yes and no :)

The handling should be similar to how hotplug is handled. E.g. the 
device becomes inaccessible by normal applications all mappings become 
invalid.

But we don't send a SIGBUS or similar on access, instead all mappings 
redirected to a dummy page which basically shallows all writes and gives 
undefined data on reads.

On IOCTLs the applications should get an error code and eventually 
restart or at least unmap all their mappings.

Regards,
Christian.

>
> Raag

Re: [PATCH v9 1/4] drm: Introduce device wedged event

[Raag Jadav]

On Mon, Nov 25, 2024 at 10:32:42AM +0100, Christian König wrote:
> Am 22.11.24 um 17:02 schrieb Raag Jadav:
> > On Fri, Nov 22, 2024 at 11:09:32AM +0100, Christian König wrote:
> > > Am 22.11.24 um 08:07 schrieb Raag Jadav:
> > > > On Mon, Nov 18, 2024 at 08:26:37PM +0530, Aravind Iddamsetty wrote:
> > > > > On 15/11/24 10:37, Raag Jadav wrote:
> > > > > > Introduce device wedged event, which notifies userspace of 'wedged'
> > > > > > (hanged/unusable) state of the DRM device through a uevent. This is
> > > > > > useful especially in cases where the device is no longer operating as
> > > > > > expected and has become unrecoverable from driver context. Purpose of
> > > > > > this implementation is to provide drivers a generic way to recover with
> > > > > > the help of userspace intervention without taking any drastic measures
> > > > > > in the driver.
> > > > > > 
> > > > > > A 'wedged' device is basically a dead device that needs attention. The
> > > > > > uevent is the notification that is sent to userspace along with a hint
> > > > > > about what could possibly be attempted to recover the device and bring
> > > > > > it back to usable state. Different drivers may have different ideas of
> > > > > > a 'wedged' device depending on their hardware implementation, and hence
> > > > > > the vendor agnostic nature of the event. It is up to the drivers to
> > > > > > decide when they see the need for recovery and how they want to recover
> > > > > > from the available methods.
> > > > > > 
> > > > > > Prerequisites
> > > > > > -------------
> > > > > > 
> > > > > > The driver, before opting for recovery, needs to make sure that the
> > > > > > 'wedged' device doesn't harm the system as a whole by taking care of the
> > > > > > prerequisites. Necessary actions must include disabling DMA to system
> > > > > > memory as well as any communication channels with other devices. Further,
> > > > > > the driver must ensure that all dma_fences are signalled and any device
> > > > > > state that the core kernel might depend on are cleaned up. Once the event
> > > > > > is sent, the device must be kept in 'wedged' state until the recovery is
> > > > > > performed. New accesses to the device (IOCTLs) should be blocked,
> > > > > > preferably with an error code that resembles the type of failure the
> > > > > > device has encountered. This will signify the reason for wegeding which
> > > > > > can be reported to the application if needed.
> > > > > should we even drop the mmaps we created?
> > > > Whatever is required for a clean recovery, yes.
> > > > 
> > > > Although how would this play out? Do we risk loosing display?
> > > > Or any other possible side-effects?
> > > Before sending a wedge event all DMA transfers of the device have to be
> > > blocked.
> > > 
> > > So yes, all display, mmap() and file descriptor connections you had with the
> > > device would need to be re-created.
> > Does it mean we'd have to rely on userspace to unmap()?
> 
> Yes and no :)
> 
> The handling should be similar to how hotplug is handled. E.g. the device
> becomes inaccessible by normal applications all mappings become invalid.

Isn't that just unbind (which is already part of recovery)?

> But we don't send a SIGBUS or similar on access, instead all mappings
> redirected to a dummy page which basically shallows all writes and gives
> undefined data on reads.
> 
> On IOCTLs the applications should get an error code and eventually restart
> or at least unmap all their mappings.

Thanks for the detailed explanation.

Rethinking about this, the criteria set for prerequisites is to not do
anything that could possibly harm the system. So I think the important
question is,

with fences signalled and ioctls already blocked, is live mmap on a wedged
device capable of producing harmful behaviour or unintended side-effects
(atleast until the application has the opportunity to unmap() as part of
recovery)?

Raag

Re: [PATCH v9 1/4] drm: Introduce device wedged event

[Christian König]

[-- Attachment #1: Type: text/plain, Size: 4559 bytes --]

Am 26.11.24 um 07:38 schrieb Raag Jadav:
> On Mon, Nov 25, 2024 at 10:32:42AM +0100, Christian König wrote:
>> Am 22.11.24 um 17:02 schrieb Raag Jadav:
>>> On Fri, Nov 22, 2024 at 11:09:32AM +0100, Christian König wrote:
>>>> Am 22.11.24 um 08:07 schrieb Raag Jadav:
>>>>> On Mon, Nov 18, 2024 at 08:26:37PM +0530, Aravind Iddamsetty wrote:
>>>>>> On 15/11/24 10:37, Raag Jadav wrote:
>>>>>>> Introduce device wedged event, which notifies userspace of 'wedged'
>>>>>>> (hanged/unusable) state of the DRM device through a uevent. This is
>>>>>>> useful especially in cases where the device is no longer operating as
>>>>>>> expected and has become unrecoverable from driver context. Purpose of
>>>>>>> this implementation is to provide drivers a generic way to recover with
>>>>>>> the help of userspace intervention without taking any drastic measures
>>>>>>> in the driver.
>>>>>>>
>>>>>>> A 'wedged' device is basically a dead device that needs attention. The
>>>>>>> uevent is the notification that is sent to userspace along with a hint
>>>>>>> about what could possibly be attempted to recover the device and bring
>>>>>>> it back to usable state. Different drivers may have different ideas of
>>>>>>> a 'wedged' device depending on their hardware implementation, and hence
>>>>>>> the vendor agnostic nature of the event. It is up to the drivers to
>>>>>>> decide when they see the need for recovery and how they want to recover
>>>>>>> from the available methods.
>>>>>>>
>>>>>>> Prerequisites
>>>>>>> -------------
>>>>>>>
>>>>>>> The driver, before opting for recovery, needs to make sure that the
>>>>>>> 'wedged' device doesn't harm the system as a whole by taking care of the
>>>>>>> prerequisites. Necessary actions must include disabling DMA to system
>>>>>>> memory as well as any communication channels with other devices. Further,
>>>>>>> the driver must ensure that all dma_fences are signalled and any device
>>>>>>> state that the core kernel might depend on are cleaned up. Once the event
>>>>>>> is sent, the device must be kept in 'wedged' state until the recovery is
>>>>>>> performed. New accesses to the device (IOCTLs) should be blocked,
>>>>>>> preferably with an error code that resembles the type of failure the
>>>>>>> device has encountered. This will signify the reason for wegeding which
>>>>>>> can be reported to the application if needed.
>>>>>> should we even drop the mmaps we created?
>>>>> Whatever is required for a clean recovery, yes.
>>>>>
>>>>> Although how would this play out? Do we risk loosing display?
>>>>> Or any other possible side-effects?
>>>> Before sending a wedge event all DMA transfers of the device have to be
>>>> blocked.
>>>>
>>>> So yes, all display, mmap() and file descriptor connections you had with the
>>>> device would need to be re-created.
>>> Does it mean we'd have to rely on userspace to unmap()?
>> Yes and no :)
>>
>> The handling should be similar to how hotplug is handled. E.g. the device
>> becomes inaccessible by normal applications all mappings become invalid.
> Isn't that just unbind (which is already part of recovery)?

No, unbind just invalidates all mappings but it doesn't catches any page 
faults which would validate them again.

The driver or framework must make sure that page faults now get 
redirected to a dummy page. See ttm_bo_vm_dummy_page() for how TTM 
handles that for all drivers using it.

Not sure about i915, since it never deals with device memory it can 
potentially just keep the access to the allocated system memory intact.

>> But we don't send a SIGBUS or similar on access, instead all mappings
>> redirected to a dummy page which basically shallows all writes and gives
>> undefined data on reads.
>>
>> On IOCTLs the applications should get an error code and eventually restart
>> or at least unmap all their mappings.
> Thanks for the detailed explanation.
>
> Rethinking about this, the criteria set for prerequisites is to not do
> anything that could possibly harm the system. So I think the important
> question is,
>
> with fences signalled and ioctls already blocked, is live mmap on a wedged
> device capable of producing harmful behaviour or unintended side-effects
> (atleast until the application has the opportunity to unmap() as part of
> recovery)?

I think we are already rather good there.

The potential options are to redirect everything to a dummy page or to 
crash the application by sending a SIGBUS.

Redirecting everything to the dummy page sounds like the more defensive 
approach.

Regards,
Christian.

>
> Raag

[-- Attachment #2: Type: text/html, Size: 6078 bytes --]

[PATCH v9 2/4] drm/doc: Document device wedged event

[Raag Jadav]

Add documentation for device wedged event in a new 'Device wedging'
chapter. The describes basic definitions and consumer expectations
along with an example.

v8: Improve documentation (Christian, Rodrigo)
v9: Add prerequisites section (Christian)

Signed-off-by: Raag Jadav <raag.jadav@intel.com>
---
 Documentation/gpu/drm-uapi.rst | 102 ++++++++++++++++++++++++++++++++-
 1 file changed, 99 insertions(+), 3 deletions(-)

diff --git a/Documentation/gpu/drm-uapi.rst b/Documentation/gpu/drm-uapi.rst
index b75cc9a70d1f..33d9c253d4d6 100644
--- a/Documentation/gpu/drm-uapi.rst
+++ b/Documentation/gpu/drm-uapi.rst
@@ -371,9 +371,105 @@ Reporting causes of resets
 
 Apart from propagating the reset through the stack so apps can recover, it's
 really useful for driver developers to learn more about what caused the reset in
-the first place. DRM devices should make use of devcoredump to store relevant
-information about the reset, so this information can be added to user bug
-reports.
+the first place. For this, drivers can make use of devcoredump to store relevant
+information about the reset and send device wedged event without recovery method
+(as explained in next chapter) to notify userspace, so this information can be
+collected and added to user bug reports.
+
+Device wedging
+==============
+
+Drivers can optionally make use of device wedged event (implemented as
+drm_dev_wedged_event() in DRM subsystem), which notifies userspace of 'wedged'
+(hanged/unusable) state of the DRM device through a uevent. This is useful
+especially in cases where the device is no longer operating as expected and
+has become unrecoverable from driver context. Purpose of this implementation
+is to provide drivers a generic way to recover with the help of userspace
+intervention without taking any drastic measures in the driver.
+
+A 'wedged' device is basically a dead device that needs attention. The
+uevent is the notification that is sent to userspace along with a hint about
+what could possibly be attempted to recover the device and bring it back to
+usable state. Different drivers may have different ideas of a 'wedged' device
+depending on their hardware implementation, and hence the vendor agnostic
+nature of the event. It is up to the drivers to decide when they see the need
+for recovery and how they want to recover from the available methods.
+
+Prerequisites
+-------------
+
+The driver, before opting for recovery, needs to make sure that the 'wedged'
+device doesn't harm the system as a whole by taking care of the prerequisites.
+Necessary actions must include disabling DMA to system memory as well as any
+communication channels with other devices. Further, the driver must ensure
+that all dma_fences are signalled and any device state that the core kernel
+might depend on are cleaned up. Once the event is sent, the device must be
+kept in 'wedged' state until the recovery is performed. New accesses to the
+device (IOCTLs) should be blocked, preferably with an error code that
+resembles the type of failure the device has encountered. This will signify
+the reason for wegeding which can be reported to the application if needed.
+
+Recovery
+--------
+
+Current implementation defines three recovery methods, out of which, drivers
+can use any one, multiple or none. Method(s) of choice will be sent in the
+uevent environment as ``WEDGED=<method1>[,<method2>]`` in order of less to
+more side-effects. If driver is unsure about recovery or method is unknown
+(like soft/hard reboot, firmware flashing, hardware replacement or any other
+procedure which can't be attempted on the fly), ``WEDGED=unknown`` will be
+sent instead.
+
+Userspace consumers can parse this event and attempt recovery as per the
+following expectations.
+
+    =============== ================================
+    Recovery method Consumer expectations
+    =============== ================================
+    none            optional telemetry collection
+    rebind          unbind + bind driver
+    bus-reset       unbind + reset bus device + bind
+    unknown         admin/user policy
+    =============== ================================
+
+The only exception to this is ``WEDGED=none``, which signifies that the
+device was temporarily 'wedged' at some point but was able to recover using
+device specific methods like reset. No explicit action is expected from
+userspace consumers in this case, but they can still take additional steps
+like gathering telemetry information (devcoredump, syslog). This is useful
+because the first hang is usually the most critical one which can result in
+consequential hangs or complete wedging.
+
+Example
+-------
+
+Udev rule::
+
+    SUBSYSTEM=="drm", ENV{WEDGED}=="rebind", DEVPATH=="*/drm/card[0-9]",
+    RUN+="/path/to/rebind.sh $env{DEVPATH}"
+
+Recovery script::
+
+    #!/bin/sh
+
+    DEVPATH=$(readlink -f /sys/$1/device)
+    DEVICE=$(basename $DEVPATH)
+    DRIVER=$(readlink -f $DEVPATH/driver)
+
+    echo -n $DEVICE > $DRIVER/unbind
+    sleep 1
+    echo -n $DEVICE > $DRIVER/bind
+
+Customization
+-------------
+
+Although basic recovery is possible with a simple script, admin/users can
+define custom policies around recovery action. For example, if the driver
+supports multiple recovery methods, consumers can opt for the suitable one
+based on policy definition. Consumers can also choose to have the device
+available for debugging or additional data collection before performing the
+recovery. This is useful especially when the driver is unsure about recovery
+or method is unknown.
 
 .. _drm_driver_ioctl:
 
-- 
2.34.1

Re: [PATCH v9 2/4] drm/doc: Document device wedged event

[Christian König]

Am 15.11.24 um 06:07 schrieb Raag Jadav:
> Add documentation for device wedged event in a new 'Device wedging'
> chapter. The describes basic definitions and consumer expectations
> along with an example.
>
> v8: Improve documentation (Christian, Rodrigo)
> v9: Add prerequisites section (Christian)
>
> Signed-off-by: Raag Jadav <raag.jadav@intel.com>

Sounds totally sane to me, but I'm not a native speaker of English so 
other should probably look at it as well.

Anyway feel free to add Reviewed-by: Christian König 
<christian.koenig@amd.com>.

Regards,
Christian.

> ---
>   Documentation/gpu/drm-uapi.rst | 102 ++++++++++++++++++++++++++++++++-
>   1 file changed, 99 insertions(+), 3 deletions(-)
>
> diff --git a/Documentation/gpu/drm-uapi.rst b/Documentation/gpu/drm-uapi.rst
> index b75cc9a70d1f..33d9c253d4d6 100644
> --- a/Documentation/gpu/drm-uapi.rst
> +++ b/Documentation/gpu/drm-uapi.rst
> @@ -371,9 +371,105 @@ Reporting causes of resets
>   
>   Apart from propagating the reset through the stack so apps can recover, it's
>   really useful for driver developers to learn more about what caused the reset in
> -the first place. DRM devices should make use of devcoredump to store relevant
> -information about the reset, so this information can be added to user bug
> -reports.
> +the first place. For this, drivers can make use of devcoredump to store relevant
> +information about the reset and send device wedged event without recovery method
> +(as explained in next chapter) to notify userspace, so this information can be
> +collected and added to user bug reports.
> +
> +Device wedging
> +==============
> +
> +Drivers can optionally make use of device wedged event (implemented as
> +drm_dev_wedged_event() in DRM subsystem), which notifies userspace of 'wedged'
> +(hanged/unusable) state of the DRM device through a uevent. This is useful
> +especially in cases where the device is no longer operating as expected and
> +has become unrecoverable from driver context. Purpose of this implementation
> +is to provide drivers a generic way to recover with the help of userspace
> +intervention without taking any drastic measures in the driver.
> +
> +A 'wedged' device is basically a dead device that needs attention. The
> +uevent is the notification that is sent to userspace along with a hint about
> +what could possibly be attempted to recover the device and bring it back to
> +usable state. Different drivers may have different ideas of a 'wedged' device
> +depending on their hardware implementation, and hence the vendor agnostic
> +nature of the event. It is up to the drivers to decide when they see the need
> +for recovery and how they want to recover from the available methods.
> +
> +Prerequisites
> +-------------
> +
> +The driver, before opting for recovery, needs to make sure that the 'wedged'
> +device doesn't harm the system as a whole by taking care of the prerequisites.
> +Necessary actions must include disabling DMA to system memory as well as any
> +communication channels with other devices. Further, the driver must ensure
> +that all dma_fences are signalled and any device state that the core kernel
> +might depend on are cleaned up. Once the event is sent, the device must be
> +kept in 'wedged' state until the recovery is performed. New accesses to the
> +device (IOCTLs) should be blocked, preferably with an error code that
> +resembles the type of failure the device has encountered. This will signify
> +the reason for wegeding which can be reported to the application if needed.
> +
> +Recovery
> +--------
> +
> +Current implementation defines three recovery methods, out of which, drivers
> +can use any one, multiple or none. Method(s) of choice will be sent in the
> +uevent environment as ``WEDGED=<method1>[,<method2>]`` in order of less to
> +more side-effects. If driver is unsure about recovery or method is unknown
> +(like soft/hard reboot, firmware flashing, hardware replacement or any other
> +procedure which can't be attempted on the fly), ``WEDGED=unknown`` will be
> +sent instead.
> +
> +Userspace consumers can parse this event and attempt recovery as per the
> +following expectations.
> +
> +    =============== ================================
> +    Recovery method Consumer expectations
> +    =============== ================================
> +    none            optional telemetry collection
> +    rebind          unbind + bind driver
> +    bus-reset       unbind + reset bus device + bind
> +    unknown         admin/user policy
> +    =============== ================================
> +
> +The only exception to this is ``WEDGED=none``, which signifies that the
> +device was temporarily 'wedged' at some point but was able to recover using
> +device specific methods like reset. No explicit action is expected from
> +userspace consumers in this case, but they can still take additional steps
> +like gathering telemetry information (devcoredump, syslog). This is useful
> +because the first hang is usually the most critical one which can result in
> +consequential hangs or complete wedging.
> +
> +Example
> +-------
> +
> +Udev rule::
> +
> +    SUBSYSTEM=="drm", ENV{WEDGED}=="rebind", DEVPATH=="*/drm/card[0-9]",
> +    RUN+="/path/to/rebind.sh $env{DEVPATH}"
> +
> +Recovery script::
> +
> +    #!/bin/sh
> +
> +    DEVPATH=$(readlink -f /sys/$1/device)
> +    DEVICE=$(basename $DEVPATH)
> +    DRIVER=$(readlink -f $DEVPATH/driver)
> +
> +    echo -n $DEVICE > $DRIVER/unbind
> +    sleep 1
> +    echo -n $DEVICE > $DRIVER/bind
> +
> +Customization
> +-------------
> +
> +Although basic recovery is possible with a simple script, admin/users can
> +define custom policies around recovery action. For example, if the driver
> +supports multiple recovery methods, consumers can opt for the suitable one
> +based on policy definition. Consumers can also choose to have the device
> +available for debugging or additional data collection before performing the
> +recovery. This is useful especially when the driver is unsure about recovery
> +or method is unknown.
>   
>   .. _drm_driver_ioctl:
>   

Re: [PATCH v9 2/4] drm/doc: Document device wedged event

[Andy Shevchenko]

On Fri, Nov 15, 2024 at 10:19:42AM +0100, Christian König wrote:
> Am 15.11.24 um 06:07 schrieb Raag Jadav:
> > Add documentation for device wedged event in a new 'Device wedging'
> > chapter. The describes basic definitions and consumer expectations
> > along with an example.
> > 
> > v8: Improve documentation (Christian, Rodrigo)
> > v9: Add prerequisites section (Christian)
> > 
> > Signed-off-by: Raag Jadav <raag.jadav@intel.com>
> 
> Sounds totally sane to me, but I'm not a native speaker of English so other
> should probably look at it as well.

> Anyway feel free to add Reviewed-by: Christian König
> <christian.koenig@amd.com>.

Side note: I don't believe tools support embedded tags, so we usually give
a tag as one tag per one line without. Otherwise it adds a manual job to
harvest them and ensure no typos made during that.

-- 
With Best Regards,
Andy Shevchenko

[PATCH v9 3/4] drm/xe: Use device wedged event

[Raag Jadav]

This was previously attempted as xe specific reset uevent but dropped
in commit 77a0d4d1cea2 ("drm/xe/uapi: Remove reset uevent for now")
as part of refactoring.

Now that we have device wedged event provided by DRM core, make use
of it and support both driver rebind and bus-reset based recovery.
With this in place userspace will be notified of wedged device, on
the basis of which, userspace may take respective action to recover
the device.

$ udevadm monitor --property --kernel
monitor will print the received events for:
KERNEL - the kernel uevent

KERNEL[265.802982] change   /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:01.0/0000:03:00.0/drm/card0 (drm)
ACTION=change
DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:01.0/0000:03:00.0/drm/card0
SUBSYSTEM=drm
WEDGED=rebind,bus-reset
DEVNAME=/dev/dri/card0
DEVTYPE=drm_minor
SEQNUM=5208
MAJOR=226
MINOR=0

v2: Change authorship to Himal (Aravind)
    Add uevent for all device wedged cases (Aravind)
v3: Generic re-implementation in DRM subsystem (Lucas)
v4: Change authorship to Raag (Aravind)

Signed-off-by: Raag Jadav <raag.jadav@intel.com>
---
 drivers/gpu/drm/xe/xe_device.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c
index 0e2dd691bdae..5878b331e35c 100644
--- a/drivers/gpu/drm/xe/xe_device.c
+++ b/drivers/gpu/drm/xe/xe_device.c
@@ -989,11 +989,12 @@ static void xe_device_wedged_fini(struct drm_device *drm, void *arg)
  * xe_device_declare_wedged - Declare device wedged
  * @xe: xe device instance
  *
- * This is a final state that can only be cleared with a mudule
+ * This is a final state that can only be cleared with a module
  * re-probe (unbind + bind).
  * In this state every IOCTL will be blocked so the GT cannot be used.
  * In general it will be called upon any critical error such as gt reset
- * failure or guc loading failure.
+ * failure or guc loading failure. Userspace will be notified of this state
+ * by a DRM uevent.
  * If xe.wedged module parameter is set to 2, this function will be called
  * on every single execution timeout (a.k.a. GPU hang) right after devcoredump
  * snapshot capture. In this mode, GT reset won't be attempted so the state of
@@ -1023,6 +1024,10 @@ void xe_device_declare_wedged(struct xe_device *xe)
 			"IOCTLs and executions are blocked. Only a rebind may clear the failure\n"
 			"Please file a _new_ bug report at https://gitlab.freedesktop.org/drm/xe/kernel/issues/new\n",
 			dev_name(xe->drm.dev));
+
+		/* Notify userspace of wedged device */
+		drm_dev_wedged_event(&xe->drm,
+				     DRM_WEDGE_RECOVERY_REBIND | DRM_WEDGE_RECOVERY_BUS_RESET);
 	}
 
 	for_each_gt(gt, xe, id)
-- 
2.34.1

Re: [PATCH v9 3/4] drm/xe: Use device wedged event

[Aravind Iddamsetty]

On 15/11/24 10:37, Raag Jadav wrote:
> This was previously attempted as xe specific reset uevent but dropped
> in commit 77a0d4d1cea2 ("drm/xe/uapi: Remove reset uevent for now")
> as part of refactoring.
>
> Now that we have device wedged event provided by DRM core, make use
> of it and support both driver rebind and bus-reset based recovery.
> With this in place userspace will be notified of wedged device, on
> the basis of which, userspace may take respective action to recover
> the device.
>
> $ udevadm monitor --property --kernel
> monitor will print the received events for:
> KERNEL - the kernel uevent
>
> KERNEL[265.802982] change   /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:01.0/0000:03:00.0/drm/card0 (drm)
> ACTION=change
> DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:01.0/0000:03:00.0/drm/card0
> SUBSYSTEM=drm
> WEDGED=rebind,bus-reset
> DEVNAME=/dev/dri/card0
> DEVTYPE=drm_minor
> SEQNUM=5208
> MAJOR=226
> MINOR=0
LGTM.
Reviewed-by: Aravind Iddamsetty <aravind.iddamsetty@linux.intel.com>

Thanks,
Aravind.
>
> v2: Change authorship to Himal (Aravind)
>     Add uevent for all device wedged cases (Aravind)
> v3: Generic re-implementation in DRM subsystem (Lucas)
> v4: Change authorship to Raag (Aravind)
>
> Signed-off-by: Raag Jadav <raag.jadav@intel.com>
> ---
>  drivers/gpu/drm/xe/xe_device.c | 9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c
> index 0e2dd691bdae..5878b331e35c 100644
> --- a/drivers/gpu/drm/xe/xe_device.c
> +++ b/drivers/gpu/drm/xe/xe_device.c
> @@ -989,11 +989,12 @@ static void xe_device_wedged_fini(struct drm_device *drm, void *arg)
>   * xe_device_declare_wedged - Declare device wedged
>   * @xe: xe device instance
>   *
> - * This is a final state that can only be cleared with a mudule
> + * This is a final state that can only be cleared with a module
>   * re-probe (unbind + bind).
>   * In this state every IOCTL will be blocked so the GT cannot be used.
>   * In general it will be called upon any critical error such as gt reset
> - * failure or guc loading failure.
> + * failure or guc loading failure. Userspace will be notified of this state
> + * by a DRM uevent.
>   * If xe.wedged module parameter is set to 2, this function will be called
>   * on every single execution timeout (a.k.a. GPU hang) right after devcoredump
>   * snapshot capture. In this mode, GT reset won't be attempted so the state of
> @@ -1023,6 +1024,10 @@ void xe_device_declare_wedged(struct xe_device *xe)
>  			"IOCTLs and executions are blocked. Only a rebind may clear the failure\n"
>  			"Please file a _new_ bug report at https://gitlab.freedesktop.org/drm/xe/kernel/issues/new\n",
>  			dev_name(xe->drm.dev));
> +
> +		/* Notify userspace of wedged device */
> +		drm_dev_wedged_event(&xe->drm,
> +				     DRM_WEDGE_RECOVERY_REBIND | DRM_WEDGE_RECOVERY_BUS_RESET);
>  	}
>  
>  	for_each_gt(gt, xe, id)

Re: [PATCH v9 3/4] drm/xe: Use device wedged event

[Ghimiray, Himal Prasad]

On 15-11-2024 10:37, Raag Jadav wrote:
> This was previously attempted as xe specific reset uevent but dropped
> in commit 77a0d4d1cea2 ("drm/xe/uapi: Remove reset uevent for now")
> as part of refactoring.
> 
> Now that we have device wedged event provided by DRM core, make use
> of it and support both driver rebind and bus-reset based recovery.
> With this in place userspace will be notified of wedged device, on
> the basis of which, userspace may take respective action to recover
> the device.
> 
> $ udevadm monitor --property --kernel
> monitor will print the received events for:
> KERNEL - the kernel uevent
> 
> KERNEL[265.802982] change   /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:01.0/0000:03:00.0/drm/card0 (drm)
> ACTION=change
> DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:01.0/0000:03:00.0/drm/card0
> SUBSYSTEM=drm
> WEDGED=rebind,bus-reset
> DEVNAME=/dev/dri/card0
> DEVTYPE=drm_minor
> SEQNUM=5208
> MAJOR=226
> MINOR=0


The patch in itself looks good. Do we have IGT tests in place to 
validate this uevent ?

BR
Himal

> 
> v2: Change authorship to Himal (Aravind)
>      Add uevent for all device wedged cases (Aravind)
> v3: Generic re-implementation in DRM subsystem (Lucas)
> v4: Change authorship to Raag (Aravind)
> 
> Signed-off-by: Raag Jadav <raag.jadav@intel.com>
> ---
>   drivers/gpu/drm/xe/xe_device.c | 9 +++++++--
>   1 file changed, 7 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c
> index 0e2dd691bdae..5878b331e35c 100644
> --- a/drivers/gpu/drm/xe/xe_device.c
> +++ b/drivers/gpu/drm/xe/xe_device.c
> @@ -989,11 +989,12 @@ static void xe_device_wedged_fini(struct drm_device *drm, void *arg)
>    * xe_device_declare_wedged - Declare device wedged
>    * @xe: xe device instance
>    *
> - * This is a final state that can only be cleared with a mudule
> + * This is a final state that can only be cleared with a module
>    * re-probe (unbind + bind).
>    * In this state every IOCTL will be blocked so the GT cannot be used.
>    * In general it will be called upon any critical error such as gt reset
> - * failure or guc loading failure.
> + * failure or guc loading failure. Userspace will be notified of this state
> + * by a DRM uevent.
>    * If xe.wedged module parameter is set to 2, this function will be called
>    * on every single execution timeout (a.k.a. GPU hang) right after devcoredump
>    * snapshot capture. In this mode, GT reset won't be attempted so the state of
> @@ -1023,6 +1024,10 @@ void xe_device_declare_wedged(struct xe_device *xe)
>   			"IOCTLs and executions are blocked. Only a rebind may clear the failure\n"
>   			"Please file a _new_ bug report at https://gitlab.freedesktop.org/drm/xe/kernel/issues/new\n",
>   			dev_name(xe->drm.dev));
> +
> +		/* Notify userspace of wedged device */
> +		drm_dev_wedged_event(&xe->drm,
> +				     DRM_WEDGE_RECOVERY_REBIND | DRM_WEDGE_RECOVERY_BUS_RESET);
>   	}
>   
>   	for_each_gt(gt, xe, id)

Re: [PATCH v9 3/4] drm/xe: Use device wedged event

[Raag Jadav]

On Tue, Nov 19, 2024 at 10:25:10AM +0530, Ghimiray, Himal Prasad wrote:
> On 15-11-2024 10:37, Raag Jadav wrote:
> > This was previously attempted as xe specific reset uevent but dropped
> > in commit 77a0d4d1cea2 ("drm/xe/uapi: Remove reset uevent for now")
> > as part of refactoring.
> > 
> > Now that we have device wedged event provided by DRM core, make use
> > of it and support both driver rebind and bus-reset based recovery.
> > With this in place userspace will be notified of wedged device, on
> > the basis of which, userspace may take respective action to recover
> > the device.
> > 
> > $ udevadm monitor --property --kernel
> > monitor will print the received events for:
> > KERNEL - the kernel uevent
> > 
> > KERNEL[265.802982] change   /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:01.0/0000:03:00.0/drm/card0 (drm)
> > ACTION=change
> > DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:01.0/0000:03:00.0/drm/card0
> > SUBSYSTEM=drm
> > WEDGED=rebind,bus-reset
> > DEVNAME=/dev/dri/card0
> > DEVTYPE=drm_minor
> > SEQNUM=5208
> > MAJOR=226
> > MINOR=0
> 
> 
> The patch in itself looks good. Do we have IGT tests in place to validate
> this uevent ?

I unit tested it with documented example which seems to work. We can have an
igt in place once we have acceptance from the community.

Raag

[PATCH v9 4/4] drm/i915: Use device wedged event

[Raag Jadav]

Now that we have device wedged event provided by DRM core, make use
of it and support both driver rebind and bus-reset based recovery.
With this in place, userspace will be notified of wedged device on
gt reset failure.

Signed-off-by: Raag Jadav <raag.jadav@intel.com>
---
 drivers/gpu/drm/i915/gt/intel_reset.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/gpu/drm/i915/gt/intel_reset.c b/drivers/gpu/drm/i915/gt/intel_reset.c
index f42f21632306..18cf50a1e84d 100644
--- a/drivers/gpu/drm/i915/gt/intel_reset.c
+++ b/drivers/gpu/drm/i915/gt/intel_reset.c
@@ -1418,6 +1418,9 @@ static void intel_gt_reset_global(struct intel_gt *gt,
 
 	if (!test_bit(I915_WEDGED, &gt->reset.flags))
 		kobject_uevent_env(kobj, KOBJ_CHANGE, reset_done_event);
+	else
+		drm_dev_wedged_event(&gt->i915->drm,
+				     DRM_WEDGE_RECOVERY_REBIND | DRM_WEDGE_RECOVERY_BUS_RESET);
 }
 
 /**
-- 
2.34.1

Re: [PATCH v9 4/4] drm/i915: Use device wedged event

[Aravind Iddamsetty]

On 15/11/24 10:37, Raag Jadav wrote:
> Now that we have device wedged event provided by DRM core, make use
> of it and support both driver rebind and bus-reset based recovery.
> With this in place, userspace will be notified of wedged device on
> gt reset failure.
>
> Signed-off-by: Raag Jadav <raag.jadav@intel.com>
> ---
>  drivers/gpu/drm/i915/gt/intel_reset.c | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/drivers/gpu/drm/i915/gt/intel_reset.c b/drivers/gpu/drm/i915/gt/intel_reset.c
> index f42f21632306..18cf50a1e84d 100644
> --- a/drivers/gpu/drm/i915/gt/intel_reset.c
> +++ b/drivers/gpu/drm/i915/gt/intel_reset.c
> @@ -1418,6 +1418,9 @@ static void intel_gt_reset_global(struct intel_gt *gt,
>  
>  	if (!test_bit(I915_WEDGED, &gt->reset.flags))
>  		kobject_uevent_env(kobj, KOBJ_CHANGE, reset_done_event);
> +	else
> +		drm_dev_wedged_event(&gt->i915->drm,
> +				     DRM_WEDGE_RECOVERY_REBIND | DRM_WEDGE_RECOVERY_BUS_RESET);
Reviewed-by: Aravind Iddamsetty <aravind.iddamsetty@linux.intel.com>

Thanks,
Aravind.
>  }
>  
>  /**

✗ Fi.CI.CHECKPATCH: warning for Introduce DRM device wedged event (rev7)

[Patchwork]

== Series Details ==

Series: Introduce DRM device wedged event (rev7)
URL   : https://patchwork.freedesktop.org/series/138069/
State : warning

== Summary ==

Error: dim checkpatch failed
06ae6f60e8d0 drm: Introduce device wedged event
-:177: WARNING:STATIC_CONST_CHAR_ARRAY: char * array declaration might be better as static const
#177: FILE: drivers/gpu/drm/drm_drv.c:539:
+	char *envp[] = { event_string, NULL };

total: 0 errors, 1 warnings, 0 checks, 102 lines checked
11b8a665faa0 drm/doc: Document device wedged event
1a472f3ac74e drm/xe: Use device wedged event
-:20: WARNING:COMMIT_LOG_LONG_LINE: Prefer a maximum 75 chars per line (possible unwrapped commit description?)
#20: 
KERNEL[265.802982] change   /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:01.0/0000:03:00.0/drm/card0 (drm)

total: 0 errors, 1 warnings, 0 checks, 24 lines checked
3a89da3a5080 drm/i915: Use device wedged event

✗ Fi.CI.SPARSE: warning for Introduce DRM device wedged event (rev7)

[Patchwork]

== Series Details ==

Series: Introduce DRM device wedged event (rev7)
URL   : https://patchwork.freedesktop.org/series/138069/
State : warning

== Summary ==

Error: dim sparse failed
Sparse version: v0.6.2
Fast mode used, each commit won't be checked separately.

✓ Fi.CI.BAT: success for Introduce DRM device wedged event (rev7)

[Patchwork]

[-- Attachment #1: Type: text/plain, Size: 3149 bytes --]

== Series Details ==

Series: Introduce DRM device wedged event (rev7)
URL   : https://patchwork.freedesktop.org/series/138069/
State : success

== Summary ==

CI Bug Log - changes from CI_DRM_15703 -> Patchwork_138069v7
====================================================

Summary
-------

  **SUCCESS**

  No regressions found.

  External URL: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/index.html

Participating hosts (47 -> 43)
------------------------------

  Missing    (4): fi-tgl-1115g4 bat-jsl-1 fi-snb-2520m bat-adls-6 

Known issues
------------

  Here are the changes found in Patchwork_138069v7 that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@i915_selftest@live:
    - bat-arlh-3:         [PASS][1] -> [ABORT][2] ([i915#10341])
   [1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/bat-arlh-3/igt@i915_selftest@live.html
   [2]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/bat-arlh-3/igt@i915_selftest@live.html
    - bat-arlh-2:         [PASS][3] -> [DMESG-FAIL][4] ([i915#10341])
   [3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/bat-arlh-2/igt@i915_selftest@live.html
   [4]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/bat-arlh-2/igt@i915_selftest@live.html

  * igt@i915_selftest@live@hangcheck:
    - bat-arlh-2:         [PASS][5] -> [DMESG-FAIL][6] ([i915#9500])
   [5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/bat-arlh-2/igt@i915_selftest@live@hangcheck.html
   [6]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/bat-arlh-2/igt@i915_selftest@live@hangcheck.html

  * igt@i915_selftest@live@workarounds:
    - bat-arlh-3:         [PASS][7] -> [ABORT][8] ([i915#12061])
   [7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/bat-arlh-3/igt@i915_selftest@live@workarounds.html
   [8]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/bat-arlh-3/igt@i915_selftest@live@workarounds.html

  * igt@kms_chamelium_edid@hdmi-edid-read:
    - bat-dg2-13:         [PASS][9] -> [DMESG-WARN][10] ([i915#12253])
   [9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/bat-dg2-13/igt@kms_chamelium_edid@hdmi-edid-read.html
   [10]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/bat-dg2-13/igt@kms_chamelium_edid@hdmi-edid-read.html

  
  [i915#10341]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/10341
  [i915#12061]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12061
  [i915#12253]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12253
  [i915#9500]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9500


Build changes
-------------

  * Linux: CI_DRM_15703 -> Patchwork_138069v7

  CI-20190529: 20190529
  CI_DRM_15703: 36fec0eb87867bca47f8829c9e5dbf5b3e2b3aaf @ git://anongit.freedesktop.org/gfx-ci/linux
  IGT_8111: 67422a7b55b0278cf0d1bfdc0899ee637ed7d588 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
  Patchwork_138069v7: 36fec0eb87867bca47f8829c9e5dbf5b3e2b3aaf @ git://anongit.freedesktop.org/gfx-ci/linux

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/index.html

[-- Attachment #2: Type: text/html, Size: 3908 bytes --]

✗ Fi.CI.IGT: failure for Introduce DRM device wedged event (rev7)

[Patchwork]

[-- Attachment #1: Type: text/plain, Size: 93315 bytes --]

== Series Details ==

Series: Introduce DRM device wedged event (rev7)
URL   : https://patchwork.freedesktop.org/series/138069/
State : failure

== Summary ==

CI Bug Log - changes from CI_DRM_15703_full -> Patchwork_138069v7_full
====================================================

Summary
-------

  **FAILURE**

  Serious unknown changes coming with Patchwork_138069v7_full absolutely need to be
  verified manually.
  
  If you think the reported changes have nothing to do with the changes
  introduced in Patchwork_138069v7_full, please notify your bug team (I915-ci-infra@lists.freedesktop.org) to allow them
  to document this new failure mode, which will reduce false positives in CI.

  External URL: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/index.html

Participating hosts (10 -> 9)
------------------------------

  Missing    (1): shard-glk-0 

Possible new issues
-------------------

  Here are the unknown changes that may have been introduced in Patchwork_138069v7_full:

### IGT changes ###

#### Possible regressions ####

  * igt@i915_query@query-topology-known-pci-ids:
    - shard-dg2:          NOTRUN -> [INCOMPLETE][1]
   [1]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-2/igt@i915_query@query-topology-known-pci-ids.html

  * igt@kms_plane@plane-panning-bottom-right-suspend:
    - shard-snb:          NOTRUN -> [INCOMPLETE][2] +1 other test incomplete
   [2]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-snb2/igt@kms_plane@plane-panning-bottom-right-suspend.html

  
Known issues
------------

  Here are the changes found in Patchwork_138069v7_full that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@api_intel_bb@crc32:
    - shard-tglu-1:       NOTRUN -> [SKIP][3] ([i915#6230])
   [3]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@api_intel_bb@crc32.html

  * igt@debugfs_test@basic-hwmon:
    - shard-tglu:         NOTRUN -> [SKIP][4] ([i915#9318])
   [4]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-4/igt@debugfs_test@basic-hwmon.html

  * igt@device_reset@cold-reset-bound:
    - shard-rkl:          NOTRUN -> [SKIP][5] ([i915#11078])
   [5]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-7/igt@device_reset@cold-reset-bound.html

  * igt@drm_fdinfo@all-busy-idle-check-all:
    - shard-dg2:          NOTRUN -> [SKIP][6] ([i915#8414])
   [6]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-7/igt@drm_fdinfo@all-busy-idle-check-all.html

  * igt@gem_busy@semaphore:
    - shard-dg2:          NOTRUN -> [SKIP][7] ([i915#3936])
   [7]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-3/igt@gem_busy@semaphore.html

  * igt@gem_ccs@ctrl-surf-copy:
    - shard-tglu:         NOTRUN -> [SKIP][8] ([i915#3555] / [i915#9323])
   [8]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-2/igt@gem_ccs@ctrl-surf-copy.html

  * igt@gem_ccs@ctrl-surf-copy-new-ctx:
    - shard-tglu-1:       NOTRUN -> [SKIP][9] ([i915#9323])
   [9]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@gem_ccs@ctrl-surf-copy-new-ctx.html

  * igt@gem_close_race@multigpu-basic-process:
    - shard-mtlp:         NOTRUN -> [SKIP][10] ([i915#7697])
   [10]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@gem_close_race@multigpu-basic-process.html

  * igt@gem_create@create-ext-cpu-access-sanity-check:
    - shard-tglu-1:       NOTRUN -> [SKIP][11] ([i915#6335])
   [11]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@gem_create@create-ext-cpu-access-sanity-check.html

  * igt@gem_create@create-ext-set-pat:
    - shard-tglu:         NOTRUN -> [SKIP][12] ([i915#8562])
   [12]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-4/igt@gem_create@create-ext-set-pat.html

  * igt@gem_ctx_engines@invalid-engines:
    - shard-rkl:          NOTRUN -> [FAIL][13] ([i915#12031])
   [13]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-4/igt@gem_ctx_engines@invalid-engines.html
    - shard-tglu:         [PASS][14] -> [FAIL][15] ([i915#12031])
   [14]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-tglu-6/igt@gem_ctx_engines@invalid-engines.html
   [15]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-5/igt@gem_ctx_engines@invalid-engines.html

  * igt@gem_ctx_freq@sysfs@gt0:
    - shard-dg2:          [PASS][16] -> [FAIL][17] ([i915#9561]) +1 other test fail
   [16]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg2-11/igt@gem_ctx_freq@sysfs@gt0.html
   [17]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-2/igt@gem_ctx_freq@sysfs@gt0.html

  * igt@gem_ctx_persistence@heartbeat-close:
    - shard-mtlp:         NOTRUN -> [SKIP][18] ([i915#8555])
   [18]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@gem_ctx_persistence@heartbeat-close.html

  * igt@gem_ctx_persistence@saturated-hostile-nopreempt@vcs1:
    - shard-mtlp:         NOTRUN -> [SKIP][19] ([i915#5882]) +6 other tests skip
   [19]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@gem_ctx_persistence@saturated-hostile-nopreempt@vcs1.html

  * igt@gem_ctx_sseu@invalid-sseu:
    - shard-rkl:          NOTRUN -> [SKIP][20] ([i915#280])
   [20]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-3/igt@gem_ctx_sseu@invalid-sseu.html

  * igt@gem_eio@hibernate:
    - shard-dg2:          NOTRUN -> [ABORT][21] ([i915#10030] / [i915#7975] / [i915#8213])
   [21]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-1/igt@gem_eio@hibernate.html

  * igt@gem_exec_balancer@bonded-semaphore:
    - shard-dg2:          NOTRUN -> [SKIP][22] ([i915#4812])
   [22]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-3/igt@gem_exec_balancer@bonded-semaphore.html

  * igt@gem_exec_balancer@nop:
    - shard-mtlp:         NOTRUN -> [DMESG-WARN][23] ([i915#12412])
   [23]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-5/igt@gem_exec_balancer@nop.html

  * igt@gem_exec_capture@capture@vecs0-lmem0:
    - shard-dg1:          NOTRUN -> [FAIL][24] ([i915#11965] / [i915#12558]) +2 other tests fail
   [24]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-17/igt@gem_exec_capture@capture@vecs0-lmem0.html

  * igt@gem_exec_fair@basic-none-rrul@rcs0:
    - shard-tglu:         NOTRUN -> [FAIL][25] ([i915#2842]) +1 other test fail
   [25]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-2/igt@gem_exec_fair@basic-none-rrul@rcs0.html

  * igt@gem_exec_fair@basic-none-share:
    - shard-mtlp:         NOTRUN -> [SKIP][26] ([i915#4473] / [i915#4771])
   [26]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@gem_exec_fair@basic-none-share.html

  * igt@gem_exec_reloc@basic-gtt-cpu:
    - shard-rkl:          NOTRUN -> [SKIP][27] ([i915#3281]) +7 other tests skip
   [27]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-3/igt@gem_exec_reloc@basic-gtt-cpu.html

  * igt@gem_exec_reloc@basic-wc:
    - shard-dg2:          NOTRUN -> [SKIP][28] ([i915#3281]) +4 other tests skip
   [28]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-7/igt@gem_exec_reloc@basic-wc.html

  * igt@gem_exec_reloc@basic-wc-gtt-noreloc:
    - shard-dg1:          NOTRUN -> [SKIP][29] ([i915#3281])
   [29]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-17/igt@gem_exec_reloc@basic-wc-gtt-noreloc.html

  * igt@gem_exec_reloc@basic-write-cpu-noreloc:
    - shard-mtlp:         NOTRUN -> [SKIP][30] ([i915#3281]) +3 other tests skip
   [30]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@gem_exec_reloc@basic-write-cpu-noreloc.html

  * igt@gem_exec_schedule@pi-common:
    - shard-mtlp:         NOTRUN -> [FAIL][31] ([i915#12296]) +6 other tests fail
   [31]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@gem_exec_schedule@pi-common.html

  * igt@gem_exec_schedule@pi-ringfull@rcs0:
    - shard-rkl:          NOTRUN -> [FAIL][32] ([i915#12296]) +4 other tests fail
   [32]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-3/igt@gem_exec_schedule@pi-ringfull@rcs0.html

  * igt@gem_exec_schedule@preempt-queue:
    - shard-dg2:          NOTRUN -> [SKIP][33] ([i915#4537] / [i915#4812])
   [33]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-7/igt@gem_exec_schedule@preempt-queue.html

  * igt@gem_exec_suspend@basic-s4-devices:
    - shard-dg2:          NOTRUN -> [ABORT][34] ([i915#7975] / [i915#8213]) +1 other test abort
   [34]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-3/igt@gem_exec_suspend@basic-s4-devices.html
    - shard-tglu:         [PASS][35] -> [ABORT][36] ([i915#7975] / [i915#8213]) +1 other test abort
   [35]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-tglu-7/igt@gem_exec_suspend@basic-s4-devices.html
   [36]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-10/igt@gem_exec_suspend@basic-s4-devices.html

  * igt@gem_fenced_exec_thrash@no-spare-fences:
    - shard-dg2:          NOTRUN -> [SKIP][37] ([i915#4860])
   [37]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-7/igt@gem_fenced_exec_thrash@no-spare-fences.html

  * igt@gem_lmem_swapping@massive-random:
    - shard-rkl:          NOTRUN -> [SKIP][38] ([i915#4613]) +1 other test skip
   [38]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-4/igt@gem_lmem_swapping@massive-random.html

  * igt@gem_lmem_swapping@parallel-multi:
    - shard-tglu:         NOTRUN -> [SKIP][39] ([i915#4613]) +2 other tests skip
   [39]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-4/igt@gem_lmem_swapping@parallel-multi.html

  * igt@gem_lmem_swapping@parallel-random:
    - shard-tglu-1:       NOTRUN -> [SKIP][40] ([i915#4613])
   [40]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@gem_lmem_swapping@parallel-random.html

  * igt@gem_lmem_swapping@parallel-random-engines:
    - shard-mtlp:         NOTRUN -> [SKIP][41] ([i915#4613])
   [41]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@gem_lmem_swapping@parallel-random-engines.html

  * igt@gem_lmem_swapping@parallel-random-verify:
    - shard-glk:          NOTRUN -> [SKIP][42] ([i915#4613])
   [42]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-glk3/igt@gem_lmem_swapping@parallel-random-verify.html

  * igt@gem_mmap@basic-small-bo:
    - shard-dg2:          NOTRUN -> [SKIP][43] ([i915#4083]) +2 other tests skip
   [43]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-7/igt@gem_mmap@basic-small-bo.html

  * igt@gem_mmap_gtt@bad-object:
    - shard-dg2:          NOTRUN -> [SKIP][44] ([i915#4077]) +3 other tests skip
   [44]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-7/igt@gem_mmap_gtt@bad-object.html

  * igt@gem_mmap_gtt@basic:
    - shard-dg1:          NOTRUN -> [SKIP][45] ([i915#4077]) +1 other test skip
   [45]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-14/igt@gem_mmap_gtt@basic.html

  * igt@gem_mmap_gtt@basic-write-cpu-read-gtt:
    - shard-mtlp:         NOTRUN -> [SKIP][46] ([i915#4077]) +3 other tests skip
   [46]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@gem_mmap_gtt@basic-write-cpu-read-gtt.html

  * igt@gem_mmap_wc@write-gtt-read-wc:
    - shard-mtlp:         NOTRUN -> [SKIP][47] ([i915#4083])
   [47]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@gem_mmap_wc@write-gtt-read-wc.html

  * igt@gem_mmap_wc@write-prefaulted:
    - shard-dg1:          NOTRUN -> [SKIP][48] ([i915#4083])
   [48]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-14/igt@gem_mmap_wc@write-prefaulted.html

  * igt@gem_partial_pwrite_pread@reads-snoop:
    - shard-mtlp:         NOTRUN -> [SKIP][49] ([i915#3282])
   [49]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@gem_partial_pwrite_pread@reads-snoop.html

  * igt@gem_partial_pwrite_pread@reads-uncached:
    - shard-dg2:          NOTRUN -> [SKIP][50] ([i915#3282]) +1 other test skip
   [50]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-3/igt@gem_partial_pwrite_pread@reads-uncached.html

  * igt@gem_pread@snoop:
    - shard-rkl:          NOTRUN -> [SKIP][51] ([i915#3282]) +4 other tests skip
   [51]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-7/igt@gem_pread@snoop.html

  * igt@gem_pxp@create-valid-protected-context:
    - shard-tglu-1:       NOTRUN -> [SKIP][52] ([i915#4270])
   [52]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@gem_pxp@create-valid-protected-context.html

  * igt@gem_pxp@dmabuf-shared-protected-dst-is-context-refcounted:
    - shard-mtlp:         NOTRUN -> [SKIP][53] ([i915#4270])
   [53]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@gem_pxp@dmabuf-shared-protected-dst-is-context-refcounted.html

  * igt@gem_pxp@protected-raw-src-copy-not-readible:
    - shard-tglu:         NOTRUN -> [SKIP][54] ([i915#4270])
   [54]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-4/igt@gem_pxp@protected-raw-src-copy-not-readible.html

  * igt@gem_pxp@reject-modify-context-protection-on:
    - shard-dg2:          NOTRUN -> [SKIP][55] ([i915#4270]) +1 other test skip
   [55]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-3/igt@gem_pxp@reject-modify-context-protection-on.html

  * igt@gem_pxp@verify-pxp-stale-ctx-execution:
    - shard-rkl:          NOTRUN -> [SKIP][56] ([i915#4270]) +2 other tests skip
   [56]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-4/igt@gem_pxp@verify-pxp-stale-ctx-execution.html

  * igt@gem_render_copy@y-tiled-ccs-to-x-tiled:
    - shard-dg2:          NOTRUN -> [SKIP][57] ([i915#5190] / [i915#8428])
   [57]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-3/igt@gem_render_copy@y-tiled-ccs-to-x-tiled.html

  * igt@gem_render_copy@yf-tiled-ccs-to-yf-tiled:
    - shard-mtlp:         NOTRUN -> [SKIP][58] ([i915#8428]) +1 other test skip
   [58]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@gem_render_copy@yf-tiled-ccs-to-yf-tiled.html

  * igt@gem_set_tiling_vs_blt@tiled-to-tiled:
    - shard-rkl:          NOTRUN -> [SKIP][59] ([i915#8411])
   [59]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-4/igt@gem_set_tiling_vs_blt@tiled-to-tiled.html

  * igt@gem_softpin@evict-snoop:
    - shard-dg2:          NOTRUN -> [SKIP][60] ([i915#4885])
   [60]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-3/igt@gem_softpin@evict-snoop.html

  * igt@gem_tiled_pread_basic:
    - shard-dg2:          NOTRUN -> [SKIP][61] ([i915#4079]) +1 other test skip
   [61]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-3/igt@gem_tiled_pread_basic.html

  * igt@gem_tiled_pread_pwrite:
    - shard-dg1:          NOTRUN -> [SKIP][62] ([i915#4079])
   [62]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-19/igt@gem_tiled_pread_pwrite.html

  * igt@gem_userptr_blits@coherency-sync:
    - shard-tglu-1:       NOTRUN -> [SKIP][63] ([i915#3297]) +1 other test skip
   [63]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@gem_userptr_blits@coherency-sync.html

  * igt@gem_userptr_blits@create-destroy-unsync:
    - shard-dg2:          NOTRUN -> [SKIP][64] ([i915#3297]) +2 other tests skip
   [64]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-3/igt@gem_userptr_blits@create-destroy-unsync.html

  * igt@gem_userptr_blits@dmabuf-sync:
    - shard-glk:          NOTRUN -> [SKIP][65] ([i915#3323])
   [65]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-glk9/igt@gem_userptr_blits@dmabuf-sync.html
    - shard-tglu-1:       NOTRUN -> [SKIP][66] ([i915#3297] / [i915#3323])
   [66]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@gem_userptr_blits@dmabuf-sync.html

  * igt@gem_userptr_blits@dmabuf-unsync:
    - shard-tglu:         NOTRUN -> [SKIP][67] ([i915#3297])
   [67]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-4/igt@gem_userptr_blits@dmabuf-unsync.html

  * igt@gem_userptr_blits@forbidden-operations:
    - shard-rkl:          NOTRUN -> [SKIP][68] ([i915#3282] / [i915#3297])
   [68]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-3/igt@gem_userptr_blits@forbidden-operations.html

  * igt@gem_userptr_blits@relocations:
    - shard-dg2:          NOTRUN -> [SKIP][69] ([i915#3281] / [i915#3297])
   [69]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-1/igt@gem_userptr_blits@relocations.html

  * igt@gem_userptr_blits@unsync-overlap:
    - shard-rkl:          NOTRUN -> [SKIP][70] ([i915#3297]) +2 other tests skip
   [70]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-7/igt@gem_userptr_blits@unsync-overlap.html

  * igt@gen3_render_tiledy_blits:
    - shard-mtlp:         NOTRUN -> [SKIP][71] +3 other tests skip
   [71]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@gen3_render_tiledy_blits.html

  * igt@gen9_exec_parse@allowed-all:
    - shard-tglu:         NOTRUN -> [SKIP][72] ([i915#2527] / [i915#2856]) +2 other tests skip
   [72]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-2/igt@gen9_exec_parse@allowed-all.html

  * igt@gen9_exec_parse@bb-start-far:
    - shard-mtlp:         NOTRUN -> [SKIP][73] ([i915#2856])
   [73]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@gen9_exec_parse@bb-start-far.html

  * igt@gen9_exec_parse@secure-batches:
    - shard-dg2:          NOTRUN -> [SKIP][74] ([i915#2856]) +1 other test skip
   [74]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-1/igt@gen9_exec_parse@secure-batches.html
    - shard-tglu-1:       NOTRUN -> [SKIP][75] ([i915#2527] / [i915#2856])
   [75]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@gen9_exec_parse@secure-batches.html

  * igt@gen9_exec_parse@valid-registers:
    - shard-rkl:          NOTRUN -> [SKIP][76] ([i915#2527]) +1 other test skip
   [76]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-7/igt@gen9_exec_parse@valid-registers.html

  * igt@i915_module_load@reload-with-fault-injection:
    - shard-glk:          [PASS][77] -> [ABORT][78] ([i915#9820])
   [77]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-glk3/igt@i915_module_load@reload-with-fault-injection.html
   [78]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-glk4/igt@i915_module_load@reload-with-fault-injection.html
    - shard-dg2:          [PASS][79] -> [ABORT][80] ([i915#9820])
   [79]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg2-7/igt@i915_module_load@reload-with-fault-injection.html
   [80]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-6/igt@i915_module_load@reload-with-fault-injection.html
    - shard-rkl:          [PASS][81] -> [ABORT][82] ([i915#9697])
   [81]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-rkl-4/igt@i915_module_load@reload-with-fault-injection.html
   [82]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-4/igt@i915_module_load@reload-with-fault-injection.html

  * igt@i915_module_load@resize-bar:
    - shard-dg1:          NOTRUN -> [SKIP][83] ([i915#7178])
   [83]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-14/igt@i915_module_load@resize-bar.html

  * igt@i915_pm_freq_mult@media-freq@gt0:
    - shard-rkl:          NOTRUN -> [SKIP][84] ([i915#6590]) +1 other test skip
   [84]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-4/igt@i915_pm_freq_mult@media-freq@gt0.html

  * igt@i915_pm_rc6_residency@rc6-idle@gt0-bcs0:
    - shard-dg1:          [PASS][85] -> [FAIL][86] ([i915#12548] / [i915#3591])
   [85]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg1-18/igt@i915_pm_rc6_residency@rc6-idle@gt0-bcs0.html
   [86]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-17/igt@i915_pm_rc6_residency@rc6-idle@gt0-bcs0.html

  * igt@i915_pm_rc6_residency@rc6-idle@gt0-rcs0:
    - shard-dg1:          [PASS][87] -> [FAIL][88] ([i915#12739] / [i915#3591])
   [87]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg1-18/igt@i915_pm_rc6_residency@rc6-idle@gt0-rcs0.html
   [88]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-17/igt@i915_pm_rc6_residency@rc6-idle@gt0-rcs0.html

  * igt@i915_query@test-query-geometry-subslices:
    - shard-tglu:         NOTRUN -> [SKIP][89] ([i915#5723])
   [89]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-4/igt@i915_query@test-query-geometry-subslices.html

  * igt@kms_addfb_basic@addfb25-y-tiled-small-legacy:
    - shard-dg2:          NOTRUN -> [SKIP][90] ([i915#4212] / [i915#5190])
   [90]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-3/igt@kms_addfb_basic@addfb25-y-tiled-small-legacy.html

  * igt@kms_async_flips@alternate-sync-async-flip:
    - shard-dg1:          [PASS][91] -> [FAIL][92] ([i915#12518] / [i915#12766])
   [91]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg1-18/igt@kms_async_flips@alternate-sync-async-flip.html
   [92]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-17/igt@kms_async_flips@alternate-sync-async-flip.html
    - shard-dg2:          [PASS][93] -> [FAIL][94] ([i915#10991] / [i915#12766])
   [93]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg2-1/igt@kms_async_flips@alternate-sync-async-flip.html
   [94]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-6/igt@kms_async_flips@alternate-sync-async-flip.html

  * igt@kms_async_flips@alternate-sync-async-flip@pipe-a-hdmi-a-3:
    - shard-dg2:          [PASS][95] -> [FAIL][96] ([i915#12518])
   [95]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg2-1/igt@kms_async_flips@alternate-sync-async-flip@pipe-a-hdmi-a-3.html
   [96]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-6/igt@kms_async_flips@alternate-sync-async-flip@pipe-a-hdmi-a-3.html

  * igt@kms_async_flips@alternate-sync-async-flip@pipe-a-hdmi-a-4:
    - shard-dg1:          [PASS][97] -> [FAIL][98] ([i915#12518])
   [97]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg1-18/igt@kms_async_flips@alternate-sync-async-flip@pipe-a-hdmi-a-4.html
   [98]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-17/igt@kms_async_flips@alternate-sync-async-flip@pipe-a-hdmi-a-4.html

  * igt@kms_async_flips@async-flip-with-page-flip-events@pipe-d-hdmi-a-3-4-mc-ccs:
    - shard-dg2:          NOTRUN -> [SKIP][99] ([i915#8709]) +11 other tests skip
   [99]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-7/igt@kms_async_flips@async-flip-with-page-flip-events@pipe-d-hdmi-a-3-4-mc-ccs.html

  * igt@kms_atomic@plane-primary-overlay-mutable-zpos:
    - shard-rkl:          NOTRUN -> [SKIP][100] ([i915#9531])
   [100]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-4/igt@kms_atomic@plane-primary-overlay-mutable-zpos.html

  * igt@kms_atomic_transition@modeset-transition:
    - shard-glk:          [PASS][101] -> [FAIL][102] ([i915#12238])
   [101]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-glk1/igt@kms_atomic_transition@modeset-transition.html
   [102]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-glk8/igt@kms_atomic_transition@modeset-transition.html

  * igt@kms_atomic_transition@modeset-transition@2x-outputs:
    - shard-glk:          [PASS][103] -> [FAIL][104] ([i915#11859])
   [103]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-glk1/igt@kms_atomic_transition@modeset-transition@2x-outputs.html
   [104]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-glk8/igt@kms_atomic_transition@modeset-transition@2x-outputs.html

  * igt@kms_atomic_transition@plane-all-modeset-transition-internal-panels:
    - shard-dg1:          NOTRUN -> [SKIP][105] ([i915#1769] / [i915#3555])
   [105]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-14/igt@kms_atomic_transition@plane-all-modeset-transition-internal-panels.html

  * igt@kms_big_fb@4-tiled-16bpp-rotate-0:
    - shard-rkl:          NOTRUN -> [SKIP][106] ([i915#5286]) +4 other tests skip
   [106]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-7/igt@kms_big_fb@4-tiled-16bpp-rotate-0.html

  * igt@kms_big_fb@4-tiled-16bpp-rotate-180:
    - shard-tglu-1:       NOTRUN -> [SKIP][107] ([i915#5286]) +1 other test skip
   [107]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@kms_big_fb@4-tiled-16bpp-rotate-180.html

  * igt@kms_big_fb@4-tiled-32bpp-rotate-90:
    - shard-dg1:          NOTRUN -> [SKIP][108] ([i915#4538] / [i915#5286])
   [108]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-14/igt@kms_big_fb@4-tiled-32bpp-rotate-90.html

  * igt@kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-180-async-flip:
    - shard-tglu:         NOTRUN -> [SKIP][109] ([i915#5286]) +3 other tests skip
   [109]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-4/igt@kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-180-async-flip.html

  * igt@kms_big_fb@linear-8bpp-rotate-270:
    - shard-rkl:          NOTRUN -> [SKIP][110] ([i915#3638]) +2 other tests skip
   [110]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-7/igt@kms_big_fb@linear-8bpp-rotate-270.html

  * igt@kms_big_fb@y-tiled-8bpp-rotate-180:
    - shard-dg2:          NOTRUN -> [SKIP][111] ([i915#4538] / [i915#5190]) +3 other tests skip
   [111]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-2/igt@kms_big_fb@y-tiled-8bpp-rotate-180.html

  * igt@kms_big_fb@yf-tiled-16bpp-rotate-180:
    - shard-rkl:          NOTRUN -> [SKIP][112] +12 other tests skip
   [112]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-4/igt@kms_big_fb@yf-tiled-16bpp-rotate-180.html

  * igt@kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-0-hflip-async-flip:
    - shard-dg1:          NOTRUN -> [SKIP][113] ([i915#4538]) +1 other test skip
   [113]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-16/igt@kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-0-hflip-async-flip.html

  * igt@kms_ccs@bad-aux-stride-y-tiled-gen12-rc-ccs@pipe-d-hdmi-a-1:
    - shard-dg2:          NOTRUN -> [SKIP][114] ([i915#10307] / [i915#10434] / [i915#6095]) +2 other tests skip
   [114]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-4/igt@kms_ccs@bad-aux-stride-y-tiled-gen12-rc-ccs@pipe-d-hdmi-a-1.html

  * igt@kms_ccs@bad-pixel-format-y-tiled-gen12-rc-ccs-cc@pipe-a-edp-1:
    - shard-mtlp:         NOTRUN -> [SKIP][115] ([i915#6095]) +9 other tests skip
   [115]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@kms_ccs@bad-pixel-format-y-tiled-gen12-rc-ccs-cc@pipe-a-edp-1.html

  * igt@kms_ccs@bad-rotation-90-4-tiled-mtl-rc-ccs-cc@pipe-b-hdmi-a-4:
    - shard-dg1:          NOTRUN -> [SKIP][116] ([i915#6095]) +129 other tests skip
   [116]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-15/igt@kms_ccs@bad-rotation-90-4-tiled-mtl-rc-ccs-cc@pipe-b-hdmi-a-4.html

  * igt@kms_ccs@ccs-on-another-bo-y-tiled-gen12-mc-ccs@pipe-b-hdmi-a-1:
    - shard-tglu:         NOTRUN -> [SKIP][117] ([i915#6095]) +19 other tests skip
   [117]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-2/igt@kms_ccs@ccs-on-another-bo-y-tiled-gen12-mc-ccs@pipe-b-hdmi-a-1.html

  * igt@kms_ccs@crc-primary-basic-4-tiled-mtl-mc-ccs@pipe-b-hdmi-a-1:
    - shard-rkl:          NOTRUN -> [SKIP][118] ([i915#6095]) +73 other tests skip
   [118]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-2/igt@kms_ccs@crc-primary-basic-4-tiled-mtl-mc-ccs@pipe-b-hdmi-a-1.html

  * igt@kms_ccs@crc-primary-rotation-180-4-tiled-bmg-ccs:
    - shard-tglu:         NOTRUN -> [SKIP][119] ([i915#12313])
   [119]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-2/igt@kms_ccs@crc-primary-rotation-180-4-tiled-bmg-ccs.html

  * igt@kms_ccs@crc-primary-rotation-180-4-tiled-lnl-ccs:
    - shard-rkl:          NOTRUN -> [SKIP][120] ([i915#12313])
   [120]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-3/igt@kms_ccs@crc-primary-rotation-180-4-tiled-lnl-ccs.html

  * igt@kms_ccs@crc-primary-suspend-4-tiled-bmg-ccs:
    - shard-tglu:         NOTRUN -> [SKIP][121] ([i915#12805])
   [121]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-4/igt@kms_ccs@crc-primary-suspend-4-tiled-bmg-ccs.html

  * igt@kms_ccs@crc-primary-suspend-4-tiled-mtl-rc-ccs-cc@pipe-a-hdmi-a-1:
    - shard-dg2:          NOTRUN -> [SKIP][122] ([i915#6095]) +7 other tests skip
   [122]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-8/igt@kms_ccs@crc-primary-suspend-4-tiled-mtl-rc-ccs-cc@pipe-a-hdmi-a-1.html

  * igt@kms_ccs@missing-ccs-buffer-4-tiled-mtl-mc-ccs@pipe-b-hdmi-a-1:
    - shard-dg2:          NOTRUN -> [SKIP][123] ([i915#10307] / [i915#6095]) +130 other tests skip
   [123]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-4/igt@kms_ccs@missing-ccs-buffer-4-tiled-mtl-mc-ccs@pipe-b-hdmi-a-1.html

  * igt@kms_ccs@missing-ccs-buffer-yf-tiled-ccs@pipe-b-hdmi-a-1:
    - shard-tglu-1:       NOTRUN -> [SKIP][124] ([i915#6095]) +29 other tests skip
   [124]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@kms_ccs@missing-ccs-buffer-yf-tiled-ccs@pipe-b-hdmi-a-1.html

  * igt@kms_cdclk@mode-transition@pipe-d-dp-4:
    - shard-dg2:          NOTRUN -> [SKIP][125] ([i915#11616] / [i915#7213]) +3 other tests skip
   [125]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-10/igt@kms_cdclk@mode-transition@pipe-d-dp-4.html

  * igt@kms_cdclk@plane-scaling@pipe-a-hdmi-a-2:
    - shard-dg2:          NOTRUN -> [SKIP][126] ([i915#4087]) +3 other tests skip
   [126]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-11/igt@kms_cdclk@plane-scaling@pipe-a-hdmi-a-2.html

  * igt@kms_chamelium_color@ctm-green-to-red:
    - shard-dg2:          NOTRUN -> [SKIP][127] +1 other test skip
   [127]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-3/igt@kms_chamelium_color@ctm-green-to-red.html

  * igt@kms_chamelium_edid@dp-edid-resolution-list:
    - shard-dg2:          NOTRUN -> [SKIP][128] ([i915#7828]) +3 other tests skip
   [128]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-7/igt@kms_chamelium_edid@dp-edid-resolution-list.html

  * igt@kms_chamelium_edid@dp-mode-timings:
    - shard-rkl:          NOTRUN -> [SKIP][129] ([i915#7828]) +6 other tests skip
   [129]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-4/igt@kms_chamelium_edid@dp-mode-timings.html

  * igt@kms_chamelium_edid@hdmi-edid-read:
    - shard-tglu-1:       NOTRUN -> [SKIP][130] ([i915#7828]) +3 other tests skip
   [130]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@kms_chamelium_edid@hdmi-edid-read.html

  * igt@kms_chamelium_edid@hdmi-edid-stress-resolution-4k:
    - shard-tglu:         NOTRUN -> [SKIP][131] ([i915#7828]) +2 other tests skip
   [131]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-2/igt@kms_chamelium_edid@hdmi-edid-stress-resolution-4k.html

  * igt@kms_chamelium_frames@dp-crc-fast:
    - shard-mtlp:         NOTRUN -> [SKIP][132] ([i915#7828]) +1 other test skip
   [132]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@kms_chamelium_frames@dp-crc-fast.html

  * igt@kms_chamelium_frames@vga-frame-dump:
    - shard-dg1:          NOTRUN -> [SKIP][133] ([i915#7828])
   [133]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-14/igt@kms_chamelium_frames@vga-frame-dump.html

  * igt@kms_color@deep-color:
    - shard-tglu:         NOTRUN -> [SKIP][134] ([i915#3555] / [i915#9979])
   [134]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-2/igt@kms_color@deep-color.html

  * igt@kms_content_protection@dp-mst-lic-type-0:
    - shard-rkl:          NOTRUN -> [SKIP][135] ([i915#3116])
   [135]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-7/igt@kms_content_protection@dp-mst-lic-type-0.html

  * igt@kms_content_protection@dp-mst-lic-type-1:
    - shard-tglu-1:       NOTRUN -> [SKIP][136] ([i915#3116] / [i915#3299])
   [136]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@kms_content_protection@dp-mst-lic-type-1.html
    - shard-dg1:          NOTRUN -> [SKIP][137] ([i915#3299]) +1 other test skip
   [137]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-19/igt@kms_content_protection@dp-mst-lic-type-1.html

  * igt@kms_content_protection@lic-type-0:
    - shard-rkl:          NOTRUN -> [SKIP][138] ([i915#9424])
   [138]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-4/igt@kms_content_protection@lic-type-0.html

  * igt@kms_content_protection@lic-type-1:
    - shard-tglu:         NOTRUN -> [SKIP][139] ([i915#6944] / [i915#9424])
   [139]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-4/igt@kms_content_protection@lic-type-1.html

  * igt@kms_content_protection@srm@pipe-a-dp-4:
    - shard-dg2:          NOTRUN -> [TIMEOUT][140] ([i915#7173])
   [140]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-10/igt@kms_content_protection@srm@pipe-a-dp-4.html

  * igt@kms_cursor_crc@cursor-offscreen-32x10:
    - shard-rkl:          NOTRUN -> [SKIP][141] ([i915#3555]) +3 other tests skip
   [141]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-7/igt@kms_cursor_crc@cursor-offscreen-32x10.html

  * igt@kms_cursor_crc@cursor-rapid-movement-32x32:
    - shard-tglu-1:       NOTRUN -> [SKIP][142] ([i915#3555]) +2 other tests skip
   [142]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@kms_cursor_crc@cursor-rapid-movement-32x32.html
    - shard-mtlp:         NOTRUN -> [SKIP][143] ([i915#3555] / [i915#8814])
   [143]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-5/igt@kms_cursor_crc@cursor-rapid-movement-32x32.html

  * igt@kms_cursor_crc@cursor-rapid-movement-512x512:
    - shard-snb:          NOTRUN -> [SKIP][144] +6 other tests skip
   [144]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-snb7/igt@kms_cursor_crc@cursor-rapid-movement-512x512.html
    - shard-tglu:         NOTRUN -> [SKIP][145] ([i915#11453] / [i915#3359])
   [145]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-6/igt@kms_cursor_crc@cursor-rapid-movement-512x512.html

  * igt@kms_cursor_crc@cursor-sliding-32x32:
    - shard-tglu:         NOTRUN -> [SKIP][146] ([i915#3555]) +1 other test skip
   [146]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-4/igt@kms_cursor_crc@cursor-sliding-32x32.html

  * igt@kms_cursor_crc@cursor-sliding-512x512:
    - shard-rkl:          NOTRUN -> [SKIP][147] ([i915#11453] / [i915#3359]) +1 other test skip
   [147]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-3/igt@kms_cursor_crc@cursor-sliding-512x512.html

  * igt@kms_cursor_legacy@cursorb-vs-flipb-varying-size:
    - shard-mtlp:         NOTRUN -> [SKIP][148] ([i915#9809])
   [148]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@kms_cursor_legacy@cursorb-vs-flipb-varying-size.html

  * igt@kms_cursor_legacy@flip-vs-cursor-atomic-transitions:
    - shard-glk:          [PASS][149] -> [FAIL][150] ([i915#2346])
   [149]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-glk9/igt@kms_cursor_legacy@flip-vs-cursor-atomic-transitions.html
   [150]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-glk7/igt@kms_cursor_legacy@flip-vs-cursor-atomic-transitions.html

  * igt@kms_cursor_legacy@modeset-atomic-cursor-hotspot:
    - shard-rkl:          NOTRUN -> [SKIP][151] ([i915#9067])
   [151]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-7/igt@kms_cursor_legacy@modeset-atomic-cursor-hotspot.html

  * igt@kms_dirtyfb@drrs-dirtyfb-ioctl:
    - shard-rkl:          NOTRUN -> [SKIP][152] ([i915#9723])
   [152]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-4/igt@kms_dirtyfb@drrs-dirtyfb-ioctl.html

  * igt@kms_dither@fb-8bpc-vs-panel-6bpc:
    - shard-dg2:          [PASS][153] -> [SKIP][154] ([i915#3555])
   [153]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg2-10/igt@kms_dither@fb-8bpc-vs-panel-6bpc.html
   [154]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-4/igt@kms_dither@fb-8bpc-vs-panel-6bpc.html
    - shard-tglu-1:       NOTRUN -> [SKIP][155] ([i915#1769] / [i915#3555] / [i915#3804])
   [155]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@kms_dither@fb-8bpc-vs-panel-6bpc.html

  * igt@kms_dither@fb-8bpc-vs-panel-6bpc@pipe-a-hdmi-a-1:
    - shard-tglu-1:       NOTRUN -> [SKIP][156] ([i915#3804])
   [156]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@kms_dither@fb-8bpc-vs-panel-6bpc@pipe-a-hdmi-a-1.html

  * igt@kms_dsc@dsc-basic:
    - shard-rkl:          NOTRUN -> [SKIP][157] ([i915#3555] / [i915#3840])
   [157]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-3/igt@kms_dsc@dsc-basic.html
    - shard-dg1:          NOTRUN -> [SKIP][158] ([i915#3555] / [i915#3840])
   [158]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-16/igt@kms_dsc@dsc-basic.html

  * igt@kms_dsc@dsc-fractional-bpp-with-bpc:
    - shard-dg2:          NOTRUN -> [SKIP][159] ([i915#3840])
   [159]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-7/igt@kms_dsc@dsc-fractional-bpp-with-bpc.html

  * igt@kms_dsc@dsc-with-formats:
    - shard-tglu:         NOTRUN -> [SKIP][160] ([i915#3555] / [i915#3840])
   [160]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-4/igt@kms_dsc@dsc-with-formats.html

  * igt@kms_dsc@dsc-with-output-formats-with-bpc:
    - shard-dg2:          NOTRUN -> [SKIP][161] ([i915#3840] / [i915#9053])
   [161]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-3/igt@kms_dsc@dsc-with-output-formats-with-bpc.html

  * igt@kms_feature_discovery@chamelium:
    - shard-dg2:          NOTRUN -> [SKIP][162] ([i915#4854])
   [162]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-7/igt@kms_feature_discovery@chamelium.html

  * igt@kms_feature_discovery@display-4x:
    - shard-rkl:          NOTRUN -> [SKIP][163] ([i915#1839])
   [163]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-4/igt@kms_feature_discovery@display-4x.html

  * igt@kms_flip@2x-dpms-vs-vblank-race-interruptible:
    - shard-mtlp:         NOTRUN -> [SKIP][164] ([i915#3637]) +2 other tests skip
   [164]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@kms_flip@2x-dpms-vs-vblank-race-interruptible.html

  * igt@kms_flip@2x-flip-vs-fences:
    - shard-rkl:          NOTRUN -> [SKIP][165] ([i915#9934]) +1 other test skip
   [165]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-3/igt@kms_flip@2x-flip-vs-fences.html

  * igt@kms_flip@2x-modeset-vs-vblank-race-interruptible:
    - shard-tglu:         NOTRUN -> [SKIP][166] ([i915#3637])
   [166]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-6/igt@kms_flip@2x-modeset-vs-vblank-race-interruptible.html

  * igt@kms_flip@2x-single-buffer-flip-vs-dpms-off-vs-modeset-interruptible:
    - shard-dg2:          NOTRUN -> [SKIP][167] ([i915#5354]) +15 other tests skip
   [167]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-3/igt@kms_flip@2x-single-buffer-flip-vs-dpms-off-vs-modeset-interruptible.html

  * igt@kms_flip@2x-wf_vblank-ts-check:
    - shard-tglu-1:       NOTRUN -> [SKIP][168] ([i915#3637])
   [168]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@kms_flip@2x-wf_vblank-ts-check.html

  * igt@kms_flip@flip-vs-absolute-wf_vblank-interruptible:
    - shard-tglu:         [PASS][169] -> [FAIL][170] ([i915#2122]) +3 other tests fail
   [169]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-tglu-4/igt@kms_flip@flip-vs-absolute-wf_vblank-interruptible.html
   [170]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-6/igt@kms_flip@flip-vs-absolute-wf_vblank-interruptible.html

  * igt@kms_flip@flip-vs-expired-vblank@a-hdmi-a3:
    - shard-dg2:          [PASS][171] -> [FAIL][172] ([i915#79]) +1 other test fail
   [171]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg2-1/igt@kms_flip@flip-vs-expired-vblank@a-hdmi-a3.html
   [172]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-2/igt@kms_flip@flip-vs-expired-vblank@a-hdmi-a3.html

  * igt@kms_flip@plain-flip-fb-recreate:
    - shard-glk:          [PASS][173] -> [FAIL][174] ([i915#2122]) +2 other tests fail
   [173]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-glk9/igt@kms_flip@plain-flip-fb-recreate.html
   [174]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-glk7/igt@kms_flip@plain-flip-fb-recreate.html

  * igt@kms_flip@plain-flip-fb-recreate@a-hdmi-a1:
    - shard-snb:          [PASS][175] -> [FAIL][176] ([i915#2122])
   [175]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-snb6/igt@kms_flip@plain-flip-fb-recreate@a-hdmi-a1.html
   [176]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-snb7/igt@kms_flip@plain-flip-fb-recreate@a-hdmi-a1.html

  * igt@kms_flip_scaled_crc@flip-32bpp-yftileccs-to-64bpp-yftile-downscaling:
    - shard-rkl:          NOTRUN -> [SKIP][177] ([i915#2672] / [i915#3555])
   [177]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-4/igt@kms_flip_scaled_crc@flip-32bpp-yftileccs-to-64bpp-yftile-downscaling.html

  * igt@kms_flip_scaled_crc@flip-32bpp-yftileccs-to-64bpp-yftile-downscaling@pipe-a-valid-mode:
    - shard-rkl:          NOTRUN -> [SKIP][178] ([i915#2672])
   [178]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-4/igt@kms_flip_scaled_crc@flip-32bpp-yftileccs-to-64bpp-yftile-downscaling@pipe-a-valid-mode.html

  * igt@kms_flip_scaled_crc@flip-32bpp-yftileccs-to-64bpp-yftile-upscaling:
    - shard-mtlp:         NOTRUN -> [SKIP][179] ([i915#2672] / [i915#3555] / [i915#8813]) +2 other tests skip
   [179]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@kms_flip_scaled_crc@flip-32bpp-yftileccs-to-64bpp-yftile-upscaling.html

  * igt@kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling:
    - shard-dg2:          NOTRUN -> [SKIP][180] ([i915#2672] / [i915#3555] / [i915#5190])
   [180]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-8/igt@kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling.html

  * igt@kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling@pipe-a-default-mode:
    - shard-mtlp:         NOTRUN -> [SKIP][181] ([i915#2672] / [i915#8813]) +2 other tests skip
   [181]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling@pipe-a-default-mode.html

  * igt@kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling:
    - shard-tglu:         NOTRUN -> [SKIP][182] ([i915#2587] / [i915#2672] / [i915#3555])
   [182]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-4/igt@kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling.html

  * igt@kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling@pipe-a-valid-mode:
    - shard-tglu:         NOTRUN -> [SKIP][183] ([i915#2587] / [i915#2672]) +1 other test skip
   [183]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-4/igt@kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling@pipe-a-valid-mode.html

  * igt@kms_flip_scaled_crc@flip-64bpp-4tile-to-16bpp-4tile-downscaling:
    - shard-tglu:         NOTRUN -> [SKIP][184] ([i915#2672] / [i915#3555])
   [184]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-2/igt@kms_flip_scaled_crc@flip-64bpp-4tile-to-16bpp-4tile-downscaling.html

  * igt@kms_flip_scaled_crc@flip-64bpp-4tile-to-16bpp-4tile-upscaling:
    - shard-tglu-1:       NOTRUN -> [SKIP][185] ([i915#2672] / [i915#3555]) +3 other tests skip
   [185]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@kms_flip_scaled_crc@flip-64bpp-4tile-to-16bpp-4tile-upscaling.html

  * igt@kms_flip_scaled_crc@flip-64bpp-4tile-to-16bpp-4tile-upscaling@pipe-a-valid-mode:
    - shard-tglu-1:       NOTRUN -> [SKIP][186] ([i915#2587] / [i915#2672]) +3 other tests skip
   [186]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@kms_flip_scaled_crc@flip-64bpp-4tile-to-16bpp-4tile-upscaling@pipe-a-valid-mode.html

  * igt@kms_flip_scaled_crc@flip-64bpp-yftile-to-16bpp-yftile-upscaling:
    - shard-dg2:          NOTRUN -> [SKIP][187] ([i915#2672] / [i915#3555])
   [187]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-1/igt@kms_flip_scaled_crc@flip-64bpp-yftile-to-16bpp-yftile-upscaling.html

  * igt@kms_flip_scaled_crc@flip-64bpp-yftile-to-16bpp-yftile-upscaling@pipe-a-valid-mode:
    - shard-dg2:          NOTRUN -> [SKIP][188] ([i915#2672]) +1 other test skip
   [188]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-1/igt@kms_flip_scaled_crc@flip-64bpp-yftile-to-16bpp-yftile-upscaling@pipe-a-valid-mode.html

  * igt@kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-draw-pwrite:
    - shard-dg2:          [PASS][189] -> [FAIL][190] ([i915#6880])
   [189]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg2-2/igt@kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-draw-pwrite.html
   [190]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-7/igt@kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-draw-pwrite.html

  * igt@kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-render:
    - shard-dg1:          NOTRUN -> [SKIP][191] +4 other tests skip
   [191]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-17/igt@kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-render.html

  * igt@kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-blt:
    - shard-mtlp:         NOTRUN -> [SKIP][192] ([i915#1825]) +6 other tests skip
   [192]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-blt.html

  * igt@kms_frontbuffer_tracking@fbc-tiling-4:
    - shard-rkl:          NOTRUN -> [SKIP][193] ([i915#5439])
   [193]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-4/igt@kms_frontbuffer_tracking@fbc-tiling-4.html

  * igt@kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-render:
    - shard-rkl:          NOTRUN -> [SKIP][194] ([i915#3023]) +13 other tests skip
   [194]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-7/igt@kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-render.html

  * igt@kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-shrfb-draw-mmap-wc:
    - shard-dg2:          NOTRUN -> [SKIP][195] ([i915#8708]) +7 other tests skip
   [195]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-1/igt@kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-shrfb-draw-mmap-wc.html

  * igt@kms_frontbuffer_tracking@fbcpsr-tiling-y:
    - shard-dg2:          NOTRUN -> [SKIP][196] ([i915#10055])
   [196]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-3/igt@kms_frontbuffer_tracking@fbcpsr-tiling-y.html

  * igt@kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-pwrite:
    - shard-dg2:          NOTRUN -> [SKIP][197] ([i915#3458]) +1 other test skip
   [197]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-3/igt@kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-pwrite.html

  * igt@kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-draw-mmap-wc:
    - shard-tglu-1:       NOTRUN -> [SKIP][198] +23 other tests skip
   [198]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-draw-mmap-wc.html

  * igt@kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-draw-mmap-gtt:
    - shard-dg1:          NOTRUN -> [SKIP][199] ([i915#8708]) +2 other tests skip
   [199]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-19/igt@kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-draw-mmap-gtt.html

  * igt@kms_frontbuffer_tracking@psr-2p-scndscrn-indfb-msflip-blt:
    - shard-rkl:          NOTRUN -> [SKIP][200] ([i915#1825]) +27 other tests skip
   [200]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-3/igt@kms_frontbuffer_tracking@psr-2p-scndscrn-indfb-msflip-blt.html

  * igt@kms_frontbuffer_tracking@psr-2p-scndscrn-pri-shrfb-draw-render:
    - shard-tglu:         NOTRUN -> [SKIP][201] +35 other tests skip
   [201]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-2/igt@kms_frontbuffer_tracking@psr-2p-scndscrn-pri-shrfb-draw-render.html

  * igt@kms_frontbuffer_tracking@psr-suspend:
    - shard-dg1:          NOTRUN -> [SKIP][202] ([i915#3458])
   [202]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-19/igt@kms_frontbuffer_tracking@psr-suspend.html

  * igt@kms_hdr@brightness-with-hdr:
    - shard-tglu:         NOTRUN -> [SKIP][203] ([i915#12713])
   [203]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-4/igt@kms_hdr@brightness-with-hdr.html

  * igt@kms_hdr@invalid-metadata-sizes:
    - shard-dg2:          [PASS][204] -> [SKIP][205] ([i915#3555] / [i915#8228])
   [204]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg2-10/igt@kms_hdr@invalid-metadata-sizes.html
   [205]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-4/igt@kms_hdr@invalid-metadata-sizes.html
    - shard-tglu-1:       NOTRUN -> [SKIP][206] ([i915#3555] / [i915#8228])
   [206]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@kms_hdr@invalid-metadata-sizes.html

  * igt@kms_hdr@static-swap:
    - shard-dg2:          NOTRUN -> [SKIP][207] ([i915#3555] / [i915#8228])
   [207]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-7/igt@kms_hdr@static-swap.html

  * igt@kms_hdr@static-toggle:
    - shard-tglu:         NOTRUN -> [SKIP][208] ([i915#3555] / [i915#8228])
   [208]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-2/igt@kms_hdr@static-toggle.html

  * igt@kms_joiner@basic-force-big-joiner:
    - shard-tglu:         NOTRUN -> [SKIP][209] ([i915#12388])
   [209]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-2/igt@kms_joiner@basic-force-big-joiner.html

  * igt@kms_joiner@basic-ultra-joiner:
    - shard-dg2:          NOTRUN -> [SKIP][210] ([i915#12339])
   [210]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-3/igt@kms_joiner@basic-ultra-joiner.html

  * igt@kms_joiner@invalid-modeset-big-joiner:
    - shard-rkl:          NOTRUN -> [SKIP][211] ([i915#10656])
   [211]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-3/igt@kms_joiner@invalid-modeset-big-joiner.html

  * igt@kms_panel_fitting@legacy:
    - shard-tglu:         NOTRUN -> [SKIP][212] ([i915#6301])
   [212]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-2/igt@kms_panel_fitting@legacy.html

  * igt@kms_plane_scaling@intel-max-src-size@pipe-a-hdmi-a-1:
    - shard-tglu:         NOTRUN -> [FAIL][213] ([i915#8292]) +1 other test fail
   [213]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-4/igt@kms_plane_scaling@intel-max-src-size@pipe-a-hdmi-a-1.html

  * igt@kms_plane_scaling@invalid-num-scalers@pipe-a-hdmi-a-4-invalid-num-scalers:
    - shard-dg1:          [PASS][214] -> [DMESG-WARN][215] ([i915#4423]) +1 other test dmesg-warn
   [214]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg1-15/igt@kms_plane_scaling@invalid-num-scalers@pipe-a-hdmi-a-4-invalid-num-scalers.html
   [215]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-15/igt@kms_plane_scaling@invalid-num-scalers@pipe-a-hdmi-a-4-invalid-num-scalers.html

  * igt@kms_plane_scaling@planes-downscale-factor-0-25-upscale-20x20:
    - shard-dg2:          NOTRUN -> [SKIP][216] ([i915#12247] / [i915#9423])
   [216]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-1/igt@kms_plane_scaling@planes-downscale-factor-0-25-upscale-20x20.html

  * igt@kms_plane_scaling@planes-downscale-factor-0-25-upscale-20x20@pipe-b:
    - shard-dg2:          NOTRUN -> [SKIP][217] ([i915#12247]) +3 other tests skip
   [217]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-1/igt@kms_plane_scaling@planes-downscale-factor-0-25-upscale-20x20@pipe-b.html

  * igt@kms_plane_scaling@planes-downscale-factor-0-25-upscale-factor-0-25:
    - shard-tglu-1:       NOTRUN -> [SKIP][218] ([i915#12247] / [i915#6953])
   [218]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@kms_plane_scaling@planes-downscale-factor-0-25-upscale-factor-0-25.html

  * igt@kms_plane_scaling@planes-downscale-factor-0-25-upscale-factor-0-25@pipe-a:
    - shard-tglu-1:       NOTRUN -> [SKIP][219] ([i915#12247]) +8 other tests skip
   [219]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@kms_plane_scaling@planes-downscale-factor-0-25-upscale-factor-0-25@pipe-a.html

  * igt@kms_plane_scaling@planes-downscale-factor-0-5:
    - shard-mtlp:         NOTRUN -> [SKIP][220] ([i915#12247] / [i915#6953])
   [220]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-5/igt@kms_plane_scaling@planes-downscale-factor-0-5.html

  * igt@kms_plane_scaling@planes-downscale-factor-0-5@pipe-a:
    - shard-mtlp:         NOTRUN -> [SKIP][221] ([i915#12247]) +3 other tests skip
   [221]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-5/igt@kms_plane_scaling@planes-downscale-factor-0-5@pipe-a.html

  * igt@kms_pm_backlight@bad-brightness:
    - shard-rkl:          NOTRUN -> [SKIP][222] ([i915#5354])
   [222]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-4/igt@kms_pm_backlight@bad-brightness.html

  * igt@kms_pm_backlight@basic-brightness:
    - shard-tglu-1:       NOTRUN -> [SKIP][223] ([i915#9812])
   [223]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@kms_pm_backlight@basic-brightness.html

  * igt@kms_pm_backlight@brightness-with-dpms:
    - shard-rkl:          NOTRUN -> [SKIP][224] ([i915#12343])
   [224]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-7/igt@kms_pm_backlight@brightness-with-dpms.html

  * igt@kms_pm_backlight@fade-with-suspend:
    - shard-tglu:         NOTRUN -> [SKIP][225] ([i915#9812])
   [225]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-2/igt@kms_pm_backlight@fade-with-suspend.html

  * igt@kms_pm_dc@dc9-dpms:
    - shard-tglu:         [PASS][226] -> [SKIP][227] ([i915#4281])
   [226]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-tglu-10/igt@kms_pm_dc@dc9-dpms.html
   [227]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-9/igt@kms_pm_dc@dc9-dpms.html

  * igt@kms_pm_rpm@dpms-lpsp:
    - shard-rkl:          [PASS][228] -> [SKIP][229] ([i915#9519]) +1 other test skip
   [228]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-rkl-7/igt@kms_pm_rpm@dpms-lpsp.html
   [229]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-6/igt@kms_pm_rpm@dpms-lpsp.html

  * igt@kms_pm_rpm@dpms-mode-unset-lpsp:
    - shard-dg2:          [PASS][230] -> [SKIP][231] ([i915#9519])
   [230]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg2-8/igt@kms_pm_rpm@dpms-mode-unset-lpsp.html
   [231]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-10/igt@kms_pm_rpm@dpms-mode-unset-lpsp.html

  * igt@kms_pm_rpm@dpms-mode-unset-non-lpsp:
    - shard-tglu:         NOTRUN -> [SKIP][232] ([i915#9519])
   [232]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-6/igt@kms_pm_rpm@dpms-mode-unset-non-lpsp.html

  * igt@kms_prime@basic-modeset-hybrid:
    - shard-dg2:          NOTRUN -> [SKIP][233] ([i915#6524] / [i915#6805])
   [233]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-7/igt@kms_prime@basic-modeset-hybrid.html

  * igt@kms_psr2_sf@fbc-pr-overlay-primary-update-sf-dmg-area:
    - shard-tglu-1:       NOTRUN -> [SKIP][234] ([i915#11520]) +3 other tests skip
   [234]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@kms_psr2_sf@fbc-pr-overlay-primary-update-sf-dmg-area.html

  * igt@kms_psr2_sf@fbc-psr2-cursor-plane-move-continuous-exceed-sf:
    - shard-tglu:         NOTRUN -> [SKIP][235] ([i915#11520]) +2 other tests skip
   [235]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-4/igt@kms_psr2_sf@fbc-psr2-cursor-plane-move-continuous-exceed-sf.html

  * igt@kms_psr2_sf@fbc-psr2-cursor-plane-move-continuous-sf:
    - shard-rkl:          NOTRUN -> [SKIP][236] ([i915#11520]) +3 other tests skip
   [236]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-3/igt@kms_psr2_sf@fbc-psr2-cursor-plane-move-continuous-sf.html

  * igt@kms_psr2_sf@fbc-psr2-overlay-plane-move-continuous-exceed-sf:
    - shard-dg2:          NOTRUN -> [SKIP][237] ([i915#11520]) +4 other tests skip
   [237]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-2/igt@kms_psr2_sf@fbc-psr2-overlay-plane-move-continuous-exceed-sf.html
    - shard-dg1:          NOTRUN -> [SKIP][238] ([i915#11520]) +1 other test skip
   [238]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-14/igt@kms_psr2_sf@fbc-psr2-overlay-plane-move-continuous-exceed-sf.html

  * igt@kms_psr2_sf@fbc-psr2-overlay-plane-move-continuous-sf@pipe-a-edp-1:
    - shard-mtlp:         NOTRUN -> [SKIP][239] ([i915#9808])
   [239]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@kms_psr2_sf@fbc-psr2-overlay-plane-move-continuous-sf@pipe-a-edp-1.html

  * igt@kms_psr2_sf@fbc-psr2-overlay-plane-move-continuous-sf@pipe-b-edp-1:
    - shard-mtlp:         NOTRUN -> [SKIP][240] ([i915#12316]) +1 other test skip
   [240]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@kms_psr2_sf@fbc-psr2-overlay-plane-move-continuous-sf@pipe-b-edp-1.html

  * igt@kms_psr2_sf@pr-cursor-plane-move-continuous-sf:
    - shard-glk:          NOTRUN -> [SKIP][241] ([i915#11520]) +2 other tests skip
   [241]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-glk7/igt@kms_psr2_sf@pr-cursor-plane-move-continuous-sf.html

  * igt@kms_psr2_su@page_flip-nv12:
    - shard-rkl:          NOTRUN -> [SKIP][242] ([i915#9683]) +1 other test skip
   [242]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-3/igt@kms_psr2_su@page_flip-nv12.html

  * igt@kms_psr@fbc-psr2-cursor-blt@edp-1:
    - shard-mtlp:         NOTRUN -> [SKIP][243] ([i915#9688]) +6 other tests skip
   [243]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@kms_psr@fbc-psr2-cursor-blt@edp-1.html

  * igt@kms_psr@fbc-psr2-cursor-mmap-gtt:
    - shard-glk:          NOTRUN -> [SKIP][244] +116 other tests skip
   [244]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-glk9/igt@kms_psr@fbc-psr2-cursor-mmap-gtt.html

  * igt@kms_psr@fbc-psr2-sprite-render:
    - shard-rkl:          NOTRUN -> [SKIP][245] ([i915#1072] / [i915#9732]) +13 other tests skip
   [245]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-4/igt@kms_psr@fbc-psr2-sprite-render.html

  * igt@kms_psr@pr-dpms:
    - shard-tglu:         NOTRUN -> [SKIP][246] ([i915#9732]) +10 other tests skip
   [246]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-4/igt@kms_psr@pr-dpms.html

  * igt@kms_psr@psr-sprite-render:
    - shard-dg1:          NOTRUN -> [SKIP][247] ([i915#1072] / [i915#9732])
   [247]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-19/igt@kms_psr@psr-sprite-render.html

  * igt@kms_psr@psr2-cursor-blt:
    - shard-dg2:          NOTRUN -> [SKIP][248] ([i915#1072] / [i915#9732]) +8 other tests skip
   [248]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-1/igt@kms_psr@psr2-cursor-blt.html
    - shard-tglu-1:       NOTRUN -> [SKIP][249] ([i915#9732]) +8 other tests skip
   [249]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@kms_psr@psr2-cursor-blt.html

  * igt@kms_psr_stress_test@invalidate-primary-flip-overlay:
    - shard-dg2:          NOTRUN -> [SKIP][250] ([i915#9685])
   [250]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-3/igt@kms_psr_stress_test@invalidate-primary-flip-overlay.html

  * igt@kms_rotation_crc@primary-4-tiled-reflect-x-0:
    - shard-tglu:         NOTRUN -> [SKIP][251] ([i915#5289])
   [251]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-2/igt@kms_rotation_crc@primary-4-tiled-reflect-x-0.html

  * igt@kms_rotation_crc@primary-y-tiled-reflect-x-0:
    - shard-mtlp:         NOTRUN -> [SKIP][252] ([i915#5289])
   [252]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@kms_rotation_crc@primary-y-tiled-reflect-x-0.html

  * igt@kms_rotation_crc@primary-yf-tiled-reflect-x-0:
    - shard-dg2:          NOTRUN -> [SKIP][253] ([i915#5190])
   [253]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-1/igt@kms_rotation_crc@primary-yf-tiled-reflect-x-0.html
    - shard-tglu-1:       NOTRUN -> [SKIP][254] ([i915#5289])
   [254]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@kms_rotation_crc@primary-yf-tiled-reflect-x-0.html

  * igt@kms_rotation_crc@primary-yf-tiled-reflect-x-90:
    - shard-rkl:          NOTRUN -> [SKIP][255] ([i915#5289])
   [255]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-7/igt@kms_rotation_crc@primary-yf-tiled-reflect-x-90.html

  * igt@kms_selftest@drm_framebuffer:
    - shard-tglu-1:       NOTRUN -> [ABORT][256] ([i915#12231]) +1 other test abort
   [256]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@kms_selftest@drm_framebuffer.html

  * igt@kms_setmode@basic:
    - shard-tglu:         [PASS][257] -> [FAIL][258] ([i915#5465]) +2 other tests fail
   [257]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-tglu-6/igt@kms_setmode@basic.html
   [258]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-10/igt@kms_setmode@basic.html

  * igt@kms_setmode@basic-clone-single-crtc:
    - shard-dg2:          NOTRUN -> [SKIP][259] ([i915#3555]) +1 other test skip
   [259]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-1/igt@kms_setmode@basic-clone-single-crtc.html

  * igt@kms_vrr@seamless-rr-switch-virtual:
    - shard-rkl:          NOTRUN -> [SKIP][260] ([i915#9906])
   [260]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-7/igt@kms_vrr@seamless-rr-switch-virtual.html

  * igt@kms_vrr@seamless-rr-switch-vrr:
    - shard-dg2:          NOTRUN -> [SKIP][261] ([i915#9906]) +1 other test skip
   [261]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-1/igt@kms_vrr@seamless-rr-switch-vrr.html
    - shard-tglu-1:       NOTRUN -> [SKIP][262] ([i915#9906])
   [262]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-1/igt@kms_vrr@seamless-rr-switch-vrr.html

  * igt@kms_writeback@writeback-check-output:
    - shard-glk:          NOTRUN -> [SKIP][263] ([i915#2437])
   [263]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-glk6/igt@kms_writeback@writeback-check-output.html
    - shard-mtlp:         NOTRUN -> [SKIP][264] ([i915#2437])
   [264]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-5/igt@kms_writeback@writeback-check-output.html

  * igt@kms_writeback@writeback-fb-id:
    - shard-tglu:         NOTRUN -> [SKIP][265] ([i915#2437])
   [265]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-2/igt@kms_writeback@writeback-fb-id.html

  * igt@perf_pmu@cpu-hotplug:
    - shard-dg2:          NOTRUN -> [SKIP][266] ([i915#8850])
   [266]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-7/igt@perf_pmu@cpu-hotplug.html

  * igt@perf_pmu@semaphore-busy@vecs0:
    - shard-dg2:          NOTRUN -> [FAIL][267] ([i915#4349]) +5 other tests fail
   [267]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-1/igt@perf_pmu@semaphore-busy@vecs0.html

  * igt@prime_vgem@basic-fence-mmap:
    - shard-mtlp:         NOTRUN -> [SKIP][268] ([i915#3708] / [i915#4077])
   [268]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-5/igt@prime_vgem@basic-fence-mmap.html

  * igt@prime_vgem@basic-fence-read:
    - shard-rkl:          NOTRUN -> [SKIP][269] ([i915#3291] / [i915#3708])
   [269]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-3/igt@prime_vgem@basic-fence-read.html

  * igt@prime_vgem@fence-flip-hang:
    - shard-rkl:          NOTRUN -> [SKIP][270] ([i915#3708])
   [270]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-3/igt@prime_vgem@fence-flip-hang.html

  
#### Possible fixes ####

  * igt@gem_exec_endless@dispatch@bcs0:
    - shard-dg2:          [TIMEOUT][271] ([i915#3778] / [i915#7016]) -> [PASS][272] +1 other test pass
   [271]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg2-2/igt@gem_exec_endless@dispatch@bcs0.html
   [272]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-7/igt@gem_exec_endless@dispatch@bcs0.html

  * igt@gem_exec_fair@basic-pace-share@rcs0:
    - shard-glk:          [FAIL][273] ([i915#2842]) -> [PASS][274] +3 other tests pass
   [273]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-glk2/igt@gem_exec_fair@basic-pace-share@rcs0.html
   [274]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-glk3/igt@gem_exec_fair@basic-pace-share@rcs0.html

  * igt@i915_module_load@reload-with-fault-injection:
    - shard-mtlp:         [ABORT][275] ([i915#10131] / [i915#10887]) -> [PASS][276]
   [275]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-mtlp-8/igt@i915_module_load@reload-with-fault-injection.html
   [276]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@i915_module_load@reload-with-fault-injection.html

  * igt@i915_pm_rc6_residency@rc6-idle@gt0-vcs0:
    - shard-dg1:          [FAIL][277] ([i915#12548] / [i915#3591]) -> [PASS][278]
   [277]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg1-18/igt@i915_pm_rc6_residency@rc6-idle@gt0-vcs0.html
   [278]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-17/igt@i915_pm_rc6_residency@rc6-idle@gt0-vcs0.html

  * igt@i915_power@sanity:
    - shard-mtlp:         [SKIP][279] ([i915#7984]) -> [PASS][280]
   [279]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-mtlp-8/igt@i915_power@sanity.html
   [280]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-1/igt@i915_power@sanity.html

  * igt@i915_suspend@basic-s3-without-i915:
    - shard-rkl:          [INCOMPLETE][281] ([i915#4817]) -> [PASS][282]
   [281]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-rkl-3/igt@i915_suspend@basic-s3-without-i915.html
   [282]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-rkl-7/igt@i915_suspend@basic-s3-without-i915.html

  * igt@i915_suspend@forcewake:
    - shard-dg1:          [INCOMPLETE][283] ([i915#4817]) -> [PASS][284]
   [283]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg1-13/igt@i915_suspend@forcewake.html
   [284]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-16/igt@i915_suspend@forcewake.html

  * igt@kms_cursor_crc@cursor-onscreen-256x256@pipe-c-hdmi-a-1:
    - shard-glk:          [INCOMPLETE][285] -> [PASS][286] +1 other test pass
   [285]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-glk4/igt@kms_cursor_crc@cursor-onscreen-256x256@pipe-c-hdmi-a-1.html
   [286]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-glk6/igt@kms_cursor_crc@cursor-onscreen-256x256@pipe-c-hdmi-a-1.html

  * igt@kms_flip@flip-vs-dpms-off-vs-modeset-interruptible@c-edp1:
    - shard-mtlp:         [INCOMPLETE][287] ([i915#6113]) -> [PASS][288] +1 other test pass
   [287]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-mtlp-4/igt@kms_flip@flip-vs-dpms-off-vs-modeset-interruptible@c-edp1.html
   [288]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-5/igt@kms_flip@flip-vs-dpms-off-vs-modeset-interruptible@c-edp1.html

  * igt@kms_flip@flip-vs-suspend-interruptible:
    - shard-tglu:         [INCOMPLETE][289] ([i915#6113]) -> [PASS][290]
   [289]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-tglu-8/igt@kms_flip@flip-vs-suspend-interruptible.html
   [290]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-4/igt@kms_flip@flip-vs-suspend-interruptible.html

  * igt@kms_flip@flip-vs-suspend-interruptible@b-hdmi-a1:
    - shard-snb:          [INCOMPLETE][291] ([i915#4839]) -> [PASS][292] +1 other test pass
   [291]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-snb4/igt@kms_flip@flip-vs-suspend-interruptible@b-hdmi-a1.html
   [292]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-snb2/igt@kms_flip@flip-vs-suspend-interruptible@b-hdmi-a1.html

  * igt@kms_flip@flip-vs-suspend-interruptible@d-hdmi-a1:
    - shard-tglu:         [INCOMPLETE][293] -> [PASS][294]
   [293]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-tglu-8/igt@kms_flip@flip-vs-suspend-interruptible@d-hdmi-a1.html
   [294]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-4/igt@kms_flip@flip-vs-suspend-interruptible@d-hdmi-a1.html

  * igt@kms_flip@plain-flip-fb-recreate@a-edp1:
    - shard-mtlp:         [FAIL][295] ([i915#2122]) -> [PASS][296] +1 other test pass
   [295]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-mtlp-3/igt@kms_flip@plain-flip-fb-recreate@a-edp1.html
   [296]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-mtlp-6/igt@kms_flip@plain-flip-fb-recreate@a-edp1.html

  * igt@kms_flip@plain-flip-fb-recreate@c-hdmi-a1:
    - shard-tglu:         [FAIL][297] ([i915#2122]) -> [PASS][298] +3 other tests pass
   [297]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-tglu-4/igt@kms_flip@plain-flip-fb-recreate@c-hdmi-a1.html
   [298]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-6/igt@kms_flip@plain-flip-fb-recreate@c-hdmi-a1.html

  * igt@kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-onoff:
    - shard-snb:          [SKIP][299] -> [PASS][300]
   [299]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-snb1/igt@kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-onoff.html
   [300]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-snb4/igt@kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-onoff.html

  * igt@kms_frontbuffer_tracking@fbc-rgb565-draw-pwrite:
    - shard-dg2:          [FAIL][301] ([i915#6880]) -> [PASS][302]
   [301]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg2-10/igt@kms_frontbuffer_tracking@fbc-rgb565-draw-pwrite.html
   [302]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-4/igt@kms_frontbuffer_tracking@fbc-rgb565-draw-pwrite.html

  * igt@kms_joiner@invalid-modeset-force-big-joiner:
    - shard-dg2:          [SKIP][303] ([i915#12388]) -> [PASS][304]
   [303]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg2-8/igt@kms_joiner@invalid-modeset-force-big-joiner.html
   [304]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-10/igt@kms_joiner@invalid-modeset-force-big-joiner.html

  * igt@kms_pm_dc@dc6-dpms:
    - shard-tglu:         [FAIL][305] ([i915#9295]) -> [PASS][306]
   [305]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-tglu-9/igt@kms_pm_dc@dc6-dpms.html
   [306]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-3/igt@kms_pm_dc@dc6-dpms.html

  * igt@kms_pm_rpm@dpms-lpsp:
    - shard-dg2:          [SKIP][307] ([i915#9519]) -> [PASS][308]
   [307]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg2-3/igt@kms_pm_rpm@dpms-lpsp.html
   [308]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-8/igt@kms_pm_rpm@dpms-lpsp.html

  
#### Warnings ####

  * igt@gem_exec_big@single:
    - shard-tglu:         [ABORT][309] -> [ABORT][310] ([i915#11713])
   [309]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-tglu-6/igt@gem_exec_big@single.html
   [310]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-5/igt@gem_exec_big@single.html

  * igt@gem_exec_fair@basic-pace@rcs0:
    - shard-tglu:         [FAIL][311] ([i915#2842]) -> [FAIL][312] ([i915#2876])
   [311]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-tglu-6/igt@gem_exec_fair@basic-pace@rcs0.html
   [312]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-tglu-3/igt@gem_exec_fair@basic-pace@rcs0.html

  * igt@kms_ccs@crc-primary-basic-4-tiled-dg2-rc-ccs-cc@pipe-a-hdmi-a-4:
    - shard-dg1:          [SKIP][313] ([i915#4423] / [i915#6095]) -> [SKIP][314] ([i915#6095]) +1 other test skip
   [313]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg1-14/igt@kms_ccs@crc-primary-basic-4-tiled-dg2-rc-ccs-cc@pipe-a-hdmi-a-4.html
   [314]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-19/igt@kms_ccs@crc-primary-basic-4-tiled-dg2-rc-ccs-cc@pipe-a-hdmi-a-4.html

  * igt@kms_content_protection@mei-interface:
    - shard-dg1:          [SKIP][315] ([i915#9433]) -> [SKIP][316] ([i915#9424])
   [315]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg1-12/igt@kms_content_protection@mei-interface.html
   [316]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-19/igt@kms_content_protection@mei-interface.html

  * igt@kms_content_protection@srm:
    - shard-dg2:          [SKIP][317] ([i915#7118]) -> [TIMEOUT][318] ([i915#7173])
   [317]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg2-8/igt@kms_content_protection@srm.html
   [318]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-10/igt@kms_content_protection@srm.html

  * igt@kms_cursor_crc@cursor-random-32x32:
    - shard-dg1:          [SKIP][319] ([i915#3555]) -> [SKIP][320] ([i915#3555] / [i915#4423])
   [319]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg1-14/igt@kms_cursor_crc@cursor-random-32x32.html
   [320]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-17/igt@kms_cursor_crc@cursor-random-32x32.html

  * igt@kms_cursor_legacy@cursora-vs-flipb-atomic-transitions-varying-size:
    - shard-dg1:          [SKIP][321] -> [SKIP][322] ([i915#4423])
   [321]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg1-18/igt@kms_cursor_legacy@cursora-vs-flipb-atomic-transitions-varying-size.html
   [322]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-18/igt@kms_cursor_legacy@cursora-vs-flipb-atomic-transitions-varying-size.html

  * igt@kms_flip_scaled_crc@flip-64bpp-4tile-to-32bpp-4tile-upscaling:
    - shard-dg1:          [SKIP][323] ([i915#2672] / [i915#3555]) -> [SKIP][324] ([i915#2672] / [i915#3555] / [i915#4423])
   [323]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg1-14/igt@kms_flip_scaled_crc@flip-64bpp-4tile-to-32bpp-4tile-upscaling.html
   [324]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-17/igt@kms_flip_scaled_crc@flip-64bpp-4tile-to-32bpp-4tile-upscaling.html

  * igt@kms_flip_scaled_crc@flip-64bpp-4tile-to-32bpp-4tile-upscaling@pipe-a-valid-mode:
    - shard-dg1:          [SKIP][325] ([i915#2587] / [i915#2672]) -> [SKIP][326] ([i915#2587] / [i915#2672] / [i915#4423])
   [325]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg1-14/igt@kms_flip_scaled_crc@flip-64bpp-4tile-to-32bpp-4tile-upscaling@pipe-a-valid-mode.html
   [326]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-17/igt@kms_flip_scaled_crc@flip-64bpp-4tile-to-32bpp-4tile-upscaling@pipe-a-valid-mode.html

  * igt@kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-blt:
    - shard-dg2:          [SKIP][327] ([i915#10433] / [i915#3458]) -> [SKIP][328] ([i915#3458]) +2 other tests skip
   [327]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg2-4/igt@kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-blt.html
   [328]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-5/igt@kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-blt.html

  * igt@kms_frontbuffer_tracking@fbcpsr-rgb565-draw-render:
    - shard-dg2:          [SKIP][329] ([i915#3458]) -> [SKIP][330] ([i915#10433] / [i915#3458]) +2 other tests skip
   [329]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg2-6/igt@kms_frontbuffer_tracking@fbcpsr-rgb565-draw-render.html
   [330]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg2-4/igt@kms_frontbuffer_tracking@fbcpsr-rgb565-draw-render.html

  * igt@kms_hdr@brightness-with-hdr:
    - shard-dg1:          [SKIP][331] ([i915#12713]) -> [SKIP][332] ([i915#1187] / [i915#12713])
   [331]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg1-17/igt@kms_hdr@brightness-with-hdr.html
   [332]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-13/igt@kms_hdr@brightness-with-hdr.html

  * igt@kms_psr@fbc-psr-sprite-plane-move:
    - shard-glk:          [INCOMPLETE][333] -> [SKIP][334]
   [333]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-glk2/igt@kms_psr@fbc-psr-sprite-plane-move.html
   [334]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-glk6/igt@kms_psr@fbc-psr-sprite-plane-move.html

  * igt@kms_psr@psr2-primary-render:
    - shard-dg1:          [SKIP][335] ([i915#1072] / [i915#4423] / [i915#9732]) -> [SKIP][336] ([i915#1072] / [i915#9732]) +2 other tests skip
   [335]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_15703/shard-dg1-14/igt@kms_psr@psr2-primary-render.html
   [336]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/shard-dg1-19/igt@kms_psr@psr2-primary-render.html

  
  {name}: This element is suppressed. This means it is ignored when computing
          the status of the difference (SUCCESS, WARNING, or FAILURE).

  [i915#10030]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/10030
  [i915#10055]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/10055
  [i915#10131]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/10131
  [i915#10307]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/10307
  [i915#10433]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/10433
  [i915#10434]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/10434
  [i915#10656]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/10656
  [i915#1072]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/1072
  [i915#10887]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/10887
  [i915#10991]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/10991
  [i915#11078]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/11078
  [i915#11453]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/11453
  [i915#11520]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/11520
  [i915#11527]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/11527
  [i915#11616]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/11616
  [i915#11681]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/11681
  [i915#11713]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/11713
  [i915#11859]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/11859
  [i915#1187]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/1187
  [i915#11965]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/11965
  [i915#12031]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12031
  [i915#12231]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12231
  [i915#12238]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12238
  [i915#12247]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12247
  [i915#12296]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12296
  [i915#12313]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12313
  [i915#12316]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12316
  [i915#12339]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12339
  [i915#12343]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12343
  [i915#12388]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12388
  [i915#12412]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12412
  [i915#12518]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12518
  [i915#12548]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12548
  [i915#12558]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12558
  [i915#12577]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12577
  [i915#12713]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12713
  [i915#12739]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12739
  [i915#12755]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12755
  [i915#12766]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12766
  [i915#12805]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12805
  [i915#1769]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/1769
  [i915#1825]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/1825
  [i915#1839]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/1839
  [i915#2122]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/2122
  [i915#2346]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/2346
  [i915#2437]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/2437
  [i915#2527]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/2527
  [i915#2587]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/2587
  [i915#2672]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/2672
  [i915#280]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/280
  [i915#2842]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/2842
  [i915#2856]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/2856
  [i915#2876]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/2876
  [i915#3023]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3023
  [i915#3116]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3116
  [i915#3281]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3281
  [i915#3282]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3282
  [i915#3291]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3291
  [i915#3297]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3297
  [i915#3299]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3299
  [i915#3323]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3323
  [i915#3359]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3359
  [i915#3458]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3458
  [i915#3539]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3539
  [i915#3555]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3555
  [i915#3591]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3591
  [i915#3637]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3637
  [i915#3638]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3638
  [i915#3708]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3708
  [i915#3778]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3778
  [i915#3804]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3804
  [i915#3840]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3840
  [i915#3936]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/3936
  [i915#4077]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4077
  [i915#4079]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4079
  [i915#4083]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4083
  [i915#4087]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4087
  [i915#4212]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4212
  [i915#4235]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4235
  [i915#4270]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4270
  [i915#4281]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4281
  [i915#4349]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4349
  [i915#4423]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4423
  [i915#4473]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4473
  [i915#4537]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4537
  [i915#4538]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4538
  [i915#4613]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4613
  [i915#4771]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4771
  [i915#4812]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4812
  [i915#4817]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4817
  [i915#4839]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4839
  [i915#4852]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4852
  [i915#4854]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4854
  [i915#4860]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4860
  [i915#4880]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4880
  [i915#4881]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4881
  [i915#4885]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/4885
  [i915#5190]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/5190
  [i915#5286]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/5286
  [i915#5289]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/5289
  [i915#5354]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/5354
  [i915#5439]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/5439
  [i915#5465]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/5465
  [i915#5723]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/5723
  [i915#5882]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/5882
  [i915#6095]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/6095
  [i915#6113]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/6113
  [i915#6188]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/6188
  [i915#6230]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/6230
  [i915#6301]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/6301
  [i915#6335]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/6335
  [i915#6524]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/6524
  [i915#6590]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/6590
  [i915#6621]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/6621
  [i915#6805]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/6805
  [i915#6880]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/6880
  [i915#6944]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/6944
  [i915#6953]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/6953
  [i915#7016]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/7016
  [i915#7118]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/7118
  [i915#7173]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/7173
  [i915#7178]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/7178
  [i915#7213]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/7213
  [i915#7697]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/7697
  [i915#7828]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/7828
  [i915#79]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/79
  [i915#7975]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/7975
  [i915#7984]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/7984
  [i915#8213]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/8213
  [i915#8228]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/8228
  [i915#8292]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/8292
  [i915#8381]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/8381
  [i915#8411]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/8411
  [i915#8414]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/8414
  [i915#8428]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/8428
  [i915#8555]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/8555
  [i915#8562]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/8562
  [i915#8708]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/8708
  [i915#8709]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/8709
  [i915#8813]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/8813
  [i915#8814]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/8814
  [i915#8850]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/8850
  [i915#9053]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9053
  [i915#9067]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9067
  [i915#9295]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9295
  [i915#9318]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9318
  [i915#9323]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9323
  [i915#9423]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9423
  [i915#9424]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9424
  [i915#9433]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9433
  [i915#9519]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9519
  [i915#9531]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9531
  [i915#9561]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9561
  [i915#9683]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9683
  [i915#9685]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9685
  [i915#9688]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9688
  [i915#9697]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9697
  [i915#9723]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9723
  [i915#9732]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9732
  [i915#9808]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9808
  [i915#9809]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9809
  [i915#9812]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9812
  [i915#9820]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9820
  [i915#9833]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9833
  [i915#9846]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9846
  [i915#9906]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9906
  [i915#9917]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9917
  [i915#9934]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9934
  [i915#9979]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/9979


Build changes
-------------

  * Linux: CI_DRM_15703 -> Patchwork_138069v7
  * Piglit: piglit_4509 -> None

  CI-20190529: 20190529
  CI_DRM_15703: 36fec0eb87867bca47f8829c9e5dbf5b3e2b3aaf @ git://anongit.freedesktop.org/gfx-ci/linux
  IGT_8111: 67422a7b55b0278cf0d1bfdc0899ee637ed7d588 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
  Patchwork_138069v7: 36fec0eb87867bca47f8829c9e5dbf5b3e2b3aaf @ git://anongit.freedesktop.org/gfx-ci/linux
  piglit_4509: fdc5a4ca11124ab8413c7988896eec4c97336694 @ git://anongit.freedesktop.org/piglit

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_138069v7/index.html

[-- Attachment #2: Type: text/html, Size: 113973 bytes --]