[PATCH v3] tests/intel/xe_fault_injection: Ignore expected errors

[PATCH v3] tests/intel/xe_fault_injection: Ignore expected errors

[Jonathan Cavitt]

The following errors can be observed when running the xe_fault_injection
subtests:

[drm] *ERROR* GT0: GuC init failed with -ENOMEM
[drm] *ERROR* GT0: Failed to initialize uC (-ENOMEM)
probe with driver xe failed with error -12

Add these messages to the dmesg ignore regex to the applicable tests
(specifically, all tests for the last error, and all tests that target
GuC subsystems for the first two errors).

v2:
- Fix and merge regex (Kamil)

v3:
- Rebase change to be compatible with latest revision (Kamil)

Closes: https://gitlab.freedesktop.org/drm/xe/kernel/issues/3343
Signed-off-by: Jonathan Cavitt <jonathan.cavitt@intel.com>
CC: Francois Dugast <francois.dugast@intel.com>
CC: Lucas De Marchi <lucas.demarchi@intel.com>
CC: Matthew Brost <matthew.brost@intel.com>
CC: Rodrigo Vivi <rodrigo.vivi@intel.com>
CC: Michal Wajdeczko <michal.wajdeczko@intel.com>
CC: Kamil Konieczny <kamil.konieczny@linux.intel.com>
---
 tests/intel/xe_fault_injection.c | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/tests/intel/xe_fault_injection.c b/tests/intel/xe_fault_injection.c
index 1b29041745..7d6c902761 100644
--- a/tests/intel/xe_fault_injection.c
+++ b/tests/intel/xe_fault_injection.c
@@ -50,6 +50,30 @@ static int fail_function_open(void)
 	return debugfs_fail_function_dir_fd;
 }
 
+static bool function_is_part_of_guc(const char function_name[])
+{
+	return strstr(function_name, "_guc_") != NULL ||
+	       strstr(function_name, "_uc_") != NULL ||
+	       strstr(function_name, "_wopcm_") != NULL;
+}
+
+static void ignore_faults_in_dmesg(const char function_name[])
+{
+	/* Driver probe is expected to fail in all cases, so ignore in igt_runner */
+	char regex[1024] = "probe with driver xe failed with error -12";
+
+	/*
+	 * If GuC module fault is injected, GuC is expected to fail,
+	 * so also ignore GuC init failures in igt_runner.
+	 */
+	if (function_is_part_of_guc(function_name)) {
+		strcat(regex, "|GT[0-9a-fA-F]*: GuC init failed with -ENOMEM");
+		strcat(regex, "|GT[0-9a-fA-F]*: Failed to initialize uC .-ENOMEM");
+	}
+
+	igt_emit_ignore_dmesg_regex(regex);
+}
+
 /*
  * The injectable file requires CONFIG_FUNCTION_ERROR_INJECTION in kernel.
  */
@@ -152,6 +176,7 @@ inject_fault_probe(int fd, char pci_slot[], const char function_name[])
 	igt_info("Injecting error \"%s\" (%d) in function \"%s\"\n",
 		 strerror(-INJECT_ERRNO), INJECT_ERRNO, function_name);
 
+	ignore_faults_in_dmesg(function_name);
 	injection_list_do(INJECTION_LIST_ADD, function_name);
 	set_retval(function_name, INJECT_ERRNO);
 	xe_sysfs_driver_do(fd, pci_slot, XE_SYSFS_DRIVER_TRY_BIND);
@@ -184,6 +209,7 @@ vm_create_fail(int fd, const char function_name[], unsigned int flags)
 {
 	igt_assert_eq(simple_vm_create(fd, flags), 0);
 
+	ignore_faults_in_dmesg(function_name);
 	injection_list_do(INJECTION_LIST_ADD, function_name);
 	set_retval(function_name, INJECT_ERRNO);
 	igt_assert(simple_vm_create(fd, flags) != 0);
@@ -243,6 +269,7 @@ vm_bind_fail(int fd, const char function_name[])
 
 	igt_assert_eq(simple_vm_bind(fd, vm), 0);
 
+	ignore_faults_in_dmesg(function_name);
 	injection_list_do(INJECTION_LIST_ADD, function_name);
 	set_retval(function_name, INJECT_ERRNO);
 	igt_assert(simple_vm_bind(fd, vm) != 0);
-- 
2.43.0

✗ CI.xeBAT: failure for tests/intel/xe_fault_injection: Ignore expected errors (rev3)

[Patchwork]

[-- Attachment #1: Type: text/plain, Size: 4704 bytes --]

== Series Details ==

Series: tests/intel/xe_fault_injection: Ignore expected errors (rev3)
URL   : https://patchwork.freedesktop.org/series/141026/
State : failure

== Summary ==

CI Bug Log - changes from XEIGT_8118_BAT -> XEIGTPW_12144_BAT
====================================================

Summary
-------

  **FAILURE**

  Serious unknown changes coming with XEIGTPW_12144_BAT absolutely need to be
  verified manually.
  
  If you think the reported changes have nothing to do with the changes
  introduced in XEIGTPW_12144_BAT, please notify your bug team (I915-ci-infra@lists.freedesktop.org) to allow them
  to document this new failure mode, which will reduce false positives in CI.

  

Participating hosts (9 -> 9)
------------------------------

  No changes in participating hosts

Possible new issues
-------------------

  Here are the unknown changes that may have been introduced in XEIGTPW_12144_BAT:

### IGT changes ###

#### Possible regressions ####

  * igt@kms_psr@psr-primary-page-flip@edp-1:
    - bat-adlp-7:         [PASS][1] -> [DMESG-WARN][2] +1 other test dmesg-warn
   [1]: https://intel-gfx-ci.01.org/tree/intel-xe/IGT_8118/bat-adlp-7/igt@kms_psr@psr-primary-page-flip@edp-1.html
   [2]: https://intel-gfx-ci.01.org/tree/intel-xe/IGTPW_12144/bat-adlp-7/igt@kms_psr@psr-primary-page-flip@edp-1.html

  
Known issues
------------

  Here are the changes found in XEIGTPW_12144_BAT that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@core_hotunplug@unbind-rebind:
    - bat-adlp-7:         [PASS][3] -> [DMESG-WARN][4] ([Intel XE#3517])
   [3]: https://intel-gfx-ci.01.org/tree/intel-xe/IGT_8118/bat-adlp-7/igt@core_hotunplug@unbind-rebind.html
   [4]: https://intel-gfx-ci.01.org/tree/intel-xe/IGTPW_12144/bat-adlp-7/igt@core_hotunplug@unbind-rebind.html

  * igt@kms_frontbuffer_tracking@basic:
    - bat-adlp-7:         [PASS][5] -> [DMESG-WARN][6] ([Intel XE#1033])
   [5]: https://intel-gfx-ci.01.org/tree/intel-xe/IGT_8118/bat-adlp-7/igt@kms_frontbuffer_tracking@basic.html
   [6]: https://intel-gfx-ci.01.org/tree/intel-xe/IGTPW_12144/bat-adlp-7/igt@kms_frontbuffer_tracking@basic.html

  * igt@kms_psr@psr-cursor-plane-move:
    - bat-adlp-7:         [PASS][7] -> [SKIP][8] ([Intel XE#3443] / [Intel XE#455]) +3 other tests skip
   [7]: https://intel-gfx-ci.01.org/tree/intel-xe/IGT_8118/bat-adlp-7/igt@kms_psr@psr-cursor-plane-move.html
   [8]: https://intel-gfx-ci.01.org/tree/intel-xe/IGTPW_12144/bat-adlp-7/igt@kms_psr@psr-cursor-plane-move.html

  * igt@xe_live_ktest@xe_migrate:
    - bat-adlp-vf:        [PASS][9] -> [SKIP][10] ([Intel XE#1192]) +1 other test skip
   [9]: https://intel-gfx-ci.01.org/tree/intel-xe/IGT_8118/bat-adlp-vf/igt@xe_live_ktest@xe_migrate.html
   [10]: https://intel-gfx-ci.01.org/tree/intel-xe/IGTPW_12144/bat-adlp-vf/igt@xe_live_ktest@xe_migrate.html

  * igt@xe_pat@pat-index-xelp@render:
    - bat-adlp-vf:        [PASS][11] -> [DMESG-WARN][12] ([Intel XE#358]) +1 other test dmesg-warn
   [11]: https://intel-gfx-ci.01.org/tree/intel-xe/IGT_8118/bat-adlp-vf/igt@xe_pat@pat-index-xelp@render.html
   [12]: https://intel-gfx-ci.01.org/tree/intel-xe/IGTPW_12144/bat-adlp-vf/igt@xe_pat@pat-index-xelp@render.html

  
#### Warnings ####

  * igt@xe_live_ktest@xe_bo:
    - bat-adlp-vf:        [SKIP][13] ([Intel XE#2229] / [Intel XE#455]) -> [SKIP][14] ([Intel XE#1192])
   [13]: https://intel-gfx-ci.01.org/tree/intel-xe/IGT_8118/bat-adlp-vf/igt@xe_live_ktest@xe_bo.html
   [14]: https://intel-gfx-ci.01.org/tree/intel-xe/IGTPW_12144/bat-adlp-vf/igt@xe_live_ktest@xe_bo.html

  
  [Intel XE#1033]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/1033
  [Intel XE#1192]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/1192
  [Intel XE#2229]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/2229
  [Intel XE#3443]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/3443
  [Intel XE#3517]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/3517
  [Intel XE#358]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/358
  [Intel XE#455]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/455


Build changes
-------------

  * IGT: IGT_8118 -> IGTPW_12144
  * Linux: xe-2252-f8f85a38f6c75e091805f01ceff4041ac2fdf3fd -> xe-2254-ccf64319402f90f68549e2047a7f985a46436e41

  IGTPW_12144: 12144
  IGT_8118: 17707095f1e5d3c30f463b43022f01c0160579b6 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
  xe-2252-f8f85a38f6c75e091805f01ceff4041ac2fdf3fd: f8f85a38f6c75e091805f01ceff4041ac2fdf3fd
  xe-2254-ccf64319402f90f68549e2047a7f985a46436e41: ccf64319402f90f68549e2047a7f985a46436e41

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/intel-xe/IGTPW_12144/index.html

[-- Attachment #2: Type: text/html, Size: 5574 bytes --]

✗ Fi.CI.BAT: failure for tests/intel/xe_fault_injection: Ignore expected errors (rev3)

[Patchwork]

[-- Attachment #1: Type: text/plain, Size: 5957 bytes --]

== Series Details ==

Series: tests/intel/xe_fault_injection: Ignore expected errors (rev3)
URL   : https://patchwork.freedesktop.org/series/141026/
State : failure

== Summary ==

CI Bug Log - changes from IGT_8118 -> IGTPW_12144
====================================================

Summary
-------

  **FAILURE**

  Serious unknown changes coming with IGTPW_12144 absolutely need to be
  verified manually.
  
  If you think the reported changes have nothing to do with the changes
  introduced in IGTPW_12144, please notify your bug team (I915-ci-infra@lists.freedesktop.org) to allow them
  to document this new failure mode, which will reduce false positives in CI.

  External URL: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/index.html

Participating hosts (46 -> 44)
------------------------------

  Missing    (2): fi-snb-2520m fi-pnv-d510 

Possible new issues
-------------------

  Here are the unknown changes that may have been introduced in IGTPW_12144:

### IGT changes ###

#### Possible regressions ####

  * igt@kms_flip@basic-flip-vs-modeset@b-dp1:
    - fi-cfl-8109u:       [PASS][1] -> [DMESG-WARN][2]
   [1]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_8118/fi-cfl-8109u/igt@kms_flip@basic-flip-vs-modeset@b-dp1.html
   [2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/fi-cfl-8109u/igt@kms_flip@basic-flip-vs-modeset@b-dp1.html

  
Known issues
------------

  Here are the changes found in IGTPW_12144 that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@i915_selftest@live:
    - bat-twl-1:          NOTRUN -> [ABORT][3] ([i915#12919]) +1 other test abort
   [3]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/bat-twl-1/igt@i915_selftest@live.html
    - bat-mtlp-6:         [PASS][4] -> [ABORT][5] ([i915#12829])
   [4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_8118/bat-mtlp-6/igt@i915_selftest@live.html
   [5]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/bat-mtlp-6/igt@i915_selftest@live.html

  * igt@i915_selftest@live@workarounds:
    - bat-mtlp-6:         [PASS][6] -> [ABORT][7] ([i915#12915])
   [6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_8118/bat-mtlp-6/igt@i915_selftest@live@workarounds.html
   [7]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/bat-mtlp-6/igt@i915_selftest@live@workarounds.html

  * igt@kms_flip@basic-flip-vs-modeset:
    - fi-cfl-8109u:       [PASS][8] -> [DMESG-WARN][9] ([i915#12914])
   [8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_8118/fi-cfl-8109u/igt@kms_flip@basic-flip-vs-modeset.html
   [9]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/fi-cfl-8109u/igt@kms_flip@basic-flip-vs-modeset.html

  * igt@runner@aborted:
    - bat-dg2-13:         NOTRUN -> [FAIL][10] ([i915#12658])
   [10]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/bat-dg2-13/igt@runner@aborted.html

  
#### Possible fixes ####

  * igt@i915_module_load@load:
    - bat-twl-2:          [DMESG-WARN][11] ([i915#1982]) -> [PASS][12]
   [11]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_8118/bat-twl-2/igt@i915_module_load@load.html
   [12]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/bat-twl-2/igt@i915_module_load@load.html

  * igt@i915_pm_rpm@module-reload:
    - bat-dg2-11:         [FAIL][13] -> [PASS][14]
   [13]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_8118/bat-dg2-11/igt@i915_pm_rpm@module-reload.html
   [14]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/bat-dg2-11/igt@i915_pm_rpm@module-reload.html
    - bat-adls-6:         [FAIL][15] ([i915#12903]) -> [PASS][16]
   [15]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_8118/bat-adls-6/igt@i915_pm_rpm@module-reload.html
   [16]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/bat-adls-6/igt@i915_pm_rpm@module-reload.html

  * igt@i915_selftest@live@gt_mocs:
    - bat-twl-2:          [ABORT][17] ([i915#12919]) -> [PASS][18]
   [17]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_8118/bat-twl-2/igt@i915_selftest@live@gt_mocs.html
   [18]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/bat-twl-2/igt@i915_selftest@live@gt_mocs.html

  * igt@kms_pipe_crc_basic@nonblocking-crc-frame-sequence:
    - bat-apl-1:          [DMESG-WARN][19] -> [PASS][20] +2 other tests pass
   [19]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_8118/bat-apl-1/igt@kms_pipe_crc_basic@nonblocking-crc-frame-sequence.html
   [20]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/bat-apl-1/igt@kms_pipe_crc_basic@nonblocking-crc-frame-sequence.html

  
#### Warnings ####

  * igt@i915_selftest@live@workarounds:
    - bat-mtlp-8:         [ABORT][21] -> [ABORT][22] ([i915#12915])
   [21]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_8118/bat-mtlp-8/igt@i915_selftest@live@workarounds.html
   [22]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/bat-mtlp-8/igt@i915_selftest@live@workarounds.html

  
  [i915#12658]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12658
  [i915#12829]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12829
  [i915#12903]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12903
  [i915#12914]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12914
  [i915#12915]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12915
  [i915#12919]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12919
  [i915#1982]: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/1982


Build changes
-------------

  * CI: CI-20190529 -> None
  * IGT: IGT_8118 -> IGTPW_12144
  * Linux: CI_DRM_15720 -> CI_DRM_15722

  CI-20190529: 20190529
  CI_DRM_15720: f8f85a38f6c75e091805f01ceff4041ac2fdf3fd @ git://anongit.freedesktop.org/gfx-ci/linux
  CI_DRM_15722: ccf64319402f90f68549e2047a7f985a46436e41 @ git://anongit.freedesktop.org/gfx-ci/linux
  IGTPW_12144: 12144
  IGT_8118: 17707095f1e5d3c30f463b43022f01c0160579b6 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/index.html

[-- Attachment #2: Type: text/html, Size: 6979 bytes --]

Re: [PATCH v3] tests/intel/xe_fault_injection: Ignore expected errors

[Kamil Konieczny]

Hi Jonathan,
On 2024-11-19 at 21:50:27 +0000, Jonathan Cavitt wrote:
> The following errors can be observed when running the xe_fault_injection
> subtests:
> 
> [drm] *ERROR* GT0: GuC init failed with -ENOMEM
> [drm] *ERROR* GT0: Failed to initialize uC (-ENOMEM)
> probe with driver xe failed with error -12
> 
> Add these messages to the dmesg ignore regex to the applicable tests
> (specifically, all tests for the last error, and all tests that target
> GuC subsystems for the first two errors).
> 
> v2:
> - Fix and merge regex (Kamil)
> 
> v3:
> - Rebase change to be compatible with latest revision (Kamil)
> 
> Closes: https://gitlab.freedesktop.org/drm/xe/kernel/issues/3343
> Signed-off-by: Jonathan Cavitt <jonathan.cavitt@intel.com>
> CC: Francois Dugast <francois.dugast@intel.com>
> CC: Lucas De Marchi <lucas.demarchi@intel.com>
> CC: Matthew Brost <matthew.brost@intel.com>
> CC: Rodrigo Vivi <rodrigo.vivi@intel.com>
> CC: Michal Wajdeczko <michal.wajdeczko@intel.com>
> CC: Kamil Konieczny <kamil.konieczny@linux.intel.com>

My r-b holds so
Reviewed-by: Kamil Konieczny <kamil.konieczny@linux.intel.com>

Please respond to CI fails and also look into fails
related to these tests, I checked few and they seem unrelated.

Regards,
Kamil

> ---
>  tests/intel/xe_fault_injection.c | 27 +++++++++++++++++++++++++++
>  1 file changed, 27 insertions(+)
> 
> diff --git a/tests/intel/xe_fault_injection.c b/tests/intel/xe_fault_injection.c
> index 1b29041745..7d6c902761 100644
> --- a/tests/intel/xe_fault_injection.c
> +++ b/tests/intel/xe_fault_injection.c
> @@ -50,6 +50,30 @@ static int fail_function_open(void)
>  	return debugfs_fail_function_dir_fd;
>  }
>  
> +static bool function_is_part_of_guc(const char function_name[])
> +{
> +	return strstr(function_name, "_guc_") != NULL ||
> +	       strstr(function_name, "_uc_") != NULL ||
> +	       strstr(function_name, "_wopcm_") != NULL;
> +}
> +
> +static void ignore_faults_in_dmesg(const char function_name[])
> +{
> +	/* Driver probe is expected to fail in all cases, so ignore in igt_runner */
> +	char regex[1024] = "probe with driver xe failed with error -12";
> +
> +	/*
> +	 * If GuC module fault is injected, GuC is expected to fail,
> +	 * so also ignore GuC init failures in igt_runner.
> +	 */
> +	if (function_is_part_of_guc(function_name)) {
> +		strcat(regex, "|GT[0-9a-fA-F]*: GuC init failed with -ENOMEM");
> +		strcat(regex, "|GT[0-9a-fA-F]*: Failed to initialize uC .-ENOMEM");
> +	}
> +
> +	igt_emit_ignore_dmesg_regex(regex);
> +}
> +
>  /*
>   * The injectable file requires CONFIG_FUNCTION_ERROR_INJECTION in kernel.
>   */
> @@ -152,6 +176,7 @@ inject_fault_probe(int fd, char pci_slot[], const char function_name[])
>  	igt_info("Injecting error \"%s\" (%d) in function \"%s\"\n",
>  		 strerror(-INJECT_ERRNO), INJECT_ERRNO, function_name);
>  
> +	ignore_faults_in_dmesg(function_name);
>  	injection_list_do(INJECTION_LIST_ADD, function_name);
>  	set_retval(function_name, INJECT_ERRNO);
>  	xe_sysfs_driver_do(fd, pci_slot, XE_SYSFS_DRIVER_TRY_BIND);
> @@ -184,6 +209,7 @@ vm_create_fail(int fd, const char function_name[], unsigned int flags)
>  {
>  	igt_assert_eq(simple_vm_create(fd, flags), 0);
>  
> +	ignore_faults_in_dmesg(function_name);
>  	injection_list_do(INJECTION_LIST_ADD, function_name);
>  	set_retval(function_name, INJECT_ERRNO);
>  	igt_assert(simple_vm_create(fd, flags) != 0);
> @@ -243,6 +269,7 @@ vm_bind_fail(int fd, const char function_name[])
>  
>  	igt_assert_eq(simple_vm_bind(fd, vm), 0);
>  
> +	ignore_faults_in_dmesg(function_name);
>  	injection_list_do(INJECTION_LIST_ADD, function_name);
>  	set_retval(function_name, INJECT_ERRNO);
>  	igt_assert(simple_vm_bind(fd, vm) != 0);
> -- 
> 2.43.0
> 

RE: [PATCH v3] tests/intel/xe_fault_injection: Ignore expected errors

[Cavitt, Jonathan]

-----Original Message-----
From: Kamil Konieczny <kamil.konieczny@linux.intel.com> 
Sent: Wednesday, November 20, 2024 10:50 AM
To: igt-dev@lists.freedesktop.org
Cc: Cavitt, Jonathan <jonathan.cavitt@intel.com>; Gupta, saurabhg <saurabhg.gupta@intel.com>; Zuo, Alex <alex.zuo@intel.com>; Dugast, Francois <francois.dugast@intel.com>; De Marchi, Lucas <lucas.demarchi@intel.com>; Brost, Matthew <matthew.brost@intel.com>; Vivi, Rodrigo <rodrigo.vivi@intel.com>; Wajdeczko, Michal <Michal.Wajdeczko@intel.com>
Subject: Re: [PATCH v3] tests/intel/xe_fault_injection: Ignore expected errors
> 
> Hi Jonathan,
> On 2024-11-19 at 21:50:27 +0000, Jonathan Cavitt wrote:
> > The following errors can be observed when running the xe_fault_injection
> > subtests:
> > 
> > [drm] *ERROR* GT0: GuC init failed with -ENOMEM
> > [drm] *ERROR* GT0: Failed to initialize uC (-ENOMEM)
> > probe with driver xe failed with error -12
> > 
> > Add these messages to the dmesg ignore regex to the applicable tests
> > (specifically, all tests for the last error, and all tests that target
> > GuC subsystems for the first two errors).
> > 
> > v2:
> > - Fix and merge regex (Kamil)
> > 
> > v3:
> > - Rebase change to be compatible with latest revision (Kamil)
> > 
> > Closes: https://gitlab.freedesktop.org/drm/xe/kernel/issues/3343
> > Signed-off-by: Jonathan Cavitt <jonathan.cavitt@intel.com>
> > CC: Francois Dugast <francois.dugast@intel.com>
> > CC: Lucas De Marchi <lucas.demarchi@intel.com>
> > CC: Matthew Brost <matthew.brost@intel.com>
> > CC: Rodrigo Vivi <rodrigo.vivi@intel.com>
> > CC: Michal Wajdeczko <michal.wajdeczko@intel.com>
> > CC: Kamil Konieczny <kamil.konieczny@linux.intel.com>
> 
> My r-b holds so
> Reviewed-by: Kamil Konieczny <kamil.konieczny@linux.intel.com>
> 
> Please respond to CI fails and also look into fails
> related to these tests, I checked few and they seem unrelated.

The failures seen in CI.XEBAT and FI.CI.BAT are all unrelated: this change only
adds error message ignore lists to tests in the xe_fault_injection test suite,
which should not have any effect in other tests where novel errors are occurring.

The failures in XE.CI.FULL are not fully reported yet, so I cannot respond to
them directly as of now.  However, the logs do not show any indication of the
prior observed error, and instead all logs I've seen report the following error
message:

<3> [377.478719] xe 0000:03:00.0: [drm] *ERROR* audio power refcount 1 after unbind

However, this error message was also present in prior revisions of the test, so
this error message is not novel and is very likely unrelated to the change here.

I'll at least respond directly to the CI errors I have access to.

-Jonathan Cavitt

> 
> Regards,
> Kamil
> 
> > ---
> >  tests/intel/xe_fault_injection.c | 27 +++++++++++++++++++++++++++
> >  1 file changed, 27 insertions(+)
> > 
> > diff --git a/tests/intel/xe_fault_injection.c b/tests/intel/xe_fault_injection.c
> > index 1b29041745..7d6c902761 100644
> > --- a/tests/intel/xe_fault_injection.c
> > +++ b/tests/intel/xe_fault_injection.c
> > @@ -50,6 +50,30 @@ static int fail_function_open(void)
> >  	return debugfs_fail_function_dir_fd;
> >  }
> >  
> > +static bool function_is_part_of_guc(const char function_name[])
> > +{
> > +	return strstr(function_name, "_guc_") != NULL ||
> > +	       strstr(function_name, "_uc_") != NULL ||
> > +	       strstr(function_name, "_wopcm_") != NULL;
> > +}
> > +
> > +static void ignore_faults_in_dmesg(const char function_name[])
> > +{
> > +	/* Driver probe is expected to fail in all cases, so ignore in igt_runner */
> > +	char regex[1024] = "probe with driver xe failed with error -12";
> > +
> > +	/*
> > +	 * If GuC module fault is injected, GuC is expected to fail,
> > +	 * so also ignore GuC init failures in igt_runner.
> > +	 */
> > +	if (function_is_part_of_guc(function_name)) {
> > +		strcat(regex, "|GT[0-9a-fA-F]*: GuC init failed with -ENOMEM");
> > +		strcat(regex, "|GT[0-9a-fA-F]*: Failed to initialize uC .-ENOMEM");
> > +	}
> > +
> > +	igt_emit_ignore_dmesg_regex(regex);
> > +}
> > +
> >  /*
> >   * The injectable file requires CONFIG_FUNCTION_ERROR_INJECTION in kernel.
> >   */
> > @@ -152,6 +176,7 @@ inject_fault_probe(int fd, char pci_slot[], const char function_name[])
> >  	igt_info("Injecting error \"%s\" (%d) in function \"%s\"\n",
> >  		 strerror(-INJECT_ERRNO), INJECT_ERRNO, function_name);
> >  
> > +	ignore_faults_in_dmesg(function_name);
> >  	injection_list_do(INJECTION_LIST_ADD, function_name);
> >  	set_retval(function_name, INJECT_ERRNO);
> >  	xe_sysfs_driver_do(fd, pci_slot, XE_SYSFS_DRIVER_TRY_BIND);
> > @@ -184,6 +209,7 @@ vm_create_fail(int fd, const char function_name[], unsigned int flags)
> >  {
> >  	igt_assert_eq(simple_vm_create(fd, flags), 0);
> >  
> > +	ignore_faults_in_dmesg(function_name);
> >  	injection_list_do(INJECTION_LIST_ADD, function_name);
> >  	set_retval(function_name, INJECT_ERRNO);
> >  	igt_assert(simple_vm_create(fd, flags) != 0);
> > @@ -243,6 +269,7 @@ vm_bind_fail(int fd, const char function_name[])
> >  
> >  	igt_assert_eq(simple_vm_bind(fd, vm), 0);
> >  
> > +	ignore_faults_in_dmesg(function_name);
> >  	injection_list_do(INJECTION_LIST_ADD, function_name);
> >  	set_retval(function_name, INJECT_ERRNO);
> >  	igt_assert(simple_vm_bind(fd, vm) != 0);
> > -- 
> > 2.43.0
> > 
> 

RE: ✗ CI.xeBAT: failure for tests/intel/xe_fault_injection: Ignore expected errors (rev3)

[Cavitt, Jonathan]

[-- Attachment #1: Type: text/plain, Size: 4742 bytes --]

These errors are unrelated to the test change, which only adds error
message blocking to xe_fault_injection tests and should not overrun.
-Jonathan Cavitt

From: Patchwork <patchwork@emeril.freedesktop.org>
Sent: Tuesday, November 19, 2024 5:02 PM
To: Cavitt, Jonathan <jonathan.cavitt@intel.com>
Cc: igt-dev@lists.freedesktop.org
Subject: ✗ CI.xeBAT: failure for tests/intel/xe_fault_injection: Ignore expected errors (rev3)

Patch Details
Series:
tests/intel/xe_fault_injection: Ignore expected errors (rev3)
URL:
https://patchwork.freedesktop.org/series/141026/
State:
failure
Details:
https://intel-gfx-ci.01.org/tree/intel-xe/IGTPW_12144/index.html
CI Bug Log - changes from XEIGT_8118_BAT -> XEIGTPW_12144_BAT
Summary

FAILURE

Serious unknown changes coming with XEIGTPW_12144_BAT absolutely need to be
verified manually.

If you think the reported changes have nothing to do with the changes
introduced in XEIGTPW_12144_BAT, please notify your bug team (I915-ci-infra@lists.freedesktop.org<mailto:I915-ci-infra@lists.freedesktop.org>) to allow them
to document this new failure mode, which will reduce false positives in CI.

Participating hosts (9 -> 9)

No changes in participating hosts

Possible new issues

Here are the unknown changes that may have been introduced in XEIGTPW_12144_BAT:

IGT changes
Possible regressions

  *   igt@kms_psr@psr-primary-page-flip@edp-1:
     *   bat-adlp-7: PASS<https://intel-gfx-ci.01.org/tree/intel-xe/IGT_8118/bat-adlp-7/igt@kms_psr@psr-primary-page-flip@edp-1.html> -> DMESG-WARN<https://intel-gfx-ci.01.org/tree/intel-xe/IGTPW_12144/bat-adlp-7/igt@kms_psr@psr-primary-page-flip@edp-1.html> +1 other test dmesg-warn

Known issues

Here are the changes found in XEIGTPW_12144_BAT that come from known issues:

IGT changes
Issues hit

  *   igt@core_hotunplug@unbind-rebind:
     *   bat-adlp-7: PASS<https://intel-gfx-ci.01.org/tree/intel-xe/IGT_8118/bat-adlp-7/igt@core_hotunplug@unbind-rebind.html> -> DMESG-WARN<https://intel-gfx-ci.01.org/tree/intel-xe/IGTPW_12144/bat-adlp-7/igt@core_hotunplug@unbind-rebind.html> (Intel XE#3517<https://gitlab.freedesktop.org/drm/xe/kernel/issues/3517>)
  *   igt@kms_frontbuffer_tracking@basic:
     *   bat-adlp-7: PASS<https://intel-gfx-ci.01.org/tree/intel-xe/IGT_8118/bat-adlp-7/igt@kms_frontbuffer_tracking@basic.html> -> DMESG-WARN<https://intel-gfx-ci.01.org/tree/intel-xe/IGTPW_12144/bat-adlp-7/igt@kms_frontbuffer_tracking@basic.html> (Intel XE#1033<https://gitlab.freedesktop.org/drm/xe/kernel/issues/1033>)
  *   igt@kms_psr@psr-cursor-plane-move:
     *   bat-adlp-7: PASS<https://intel-gfx-ci.01.org/tree/intel-xe/IGT_8118/bat-adlp-7/igt@kms_psr@psr-cursor-plane-move.html> -> SKIP<https://intel-gfx-ci.01.org/tree/intel-xe/IGTPW_12144/bat-adlp-7/igt@kms_psr@psr-cursor-plane-move.html> (Intel XE#3443<https://gitlab.freedesktop.org/drm/xe/kernel/issues/3443> / Intel XE#455<https://gitlab.freedesktop.org/drm/xe/kernel/issues/455>) +3 other tests skip
  *   igt@xe_live_ktest@xe_migrate:
     *   bat-adlp-vf: PASS<https://intel-gfx-ci.01.org/tree/intel-xe/IGT_8118/bat-adlp-vf/igt@xe_live_ktest@xe_migrate.html> -> SKIP<https://intel-gfx-ci.01.org/tree/intel-xe/IGTPW_12144/bat-adlp-vf/igt@xe_live_ktest@xe_migrate.html> (Intel XE#1192<https://gitlab.freedesktop.org/drm/xe/kernel/issues/1192>) +1 other test skip
  *   igt@xe_pat@pat-index-xelp@render:
     *   bat-adlp-vf: PASS<https://intel-gfx-ci.01.org/tree/intel-xe/IGT_8118/bat-adlp-vf/igt@xe_pat@pat-index-xelp@render.html> -> DMESG-WARN<https://intel-gfx-ci.01.org/tree/intel-xe/IGTPW_12144/bat-adlp-vf/igt@xe_pat@pat-index-xelp@render.html> (Intel XE#358<https://gitlab.freedesktop.org/drm/xe/kernel/issues/358>) +1 other test dmesg-warn

Warnings

  *   igt@xe_live_ktest@xe_bo:
     *   bat-adlp-vf: SKIP<https://intel-gfx-ci.01.org/tree/intel-xe/IGT_8118/bat-adlp-vf/igt@xe_live_ktest@xe_bo.html> (Intel XE#2229<https://gitlab.freedesktop.org/drm/xe/kernel/issues/2229> / Intel XE#455<https://gitlab.freedesktop.org/drm/xe/kernel/issues/455>) -> SKIP<https://intel-gfx-ci.01.org/tree/intel-xe/IGTPW_12144/bat-adlp-vf/igt@xe_live_ktest@xe_bo.html> (Intel XE#1192<https://gitlab.freedesktop.org/drm/xe/kernel/issues/1192>)

Build changes

  *   IGT: IGT_8118 -> IGTPW_12144
  *   Linux: xe-2252-f8f85a38f6c75e091805f01ceff4041ac2fdf3fd -> xe-2254-ccf64319402f90f68549e2047a7f985a46436e41

IGTPW_12144: 12144
IGT_8118: 17707095f1e5d3c30f463b43022f01c0160579b6 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
xe-2252-f8f85a38f6c75e091805f01ceff4041ac2fdf3fd: f8f85a38f6c75e091805f01ceff4041ac2fdf3fd
xe-2254-ccf64319402f90f68549e2047a7f985a46436e41: ccf64319402f90f68549e2047a7f985a46436e41

[-- Attachment #2: Type: text/html, Size: 20865 bytes --]

RE: ✗ Fi.CI.BAT: failure for tests/intel/xe_fault_injection: Ignore expected errors (rev3)

[Cavitt, Jonathan]

[-- Attachment #1: Type: text/plain, Size: 5891 bytes --]

These errors are unrelated to the test change, which only adds error
message blocking to xe_fault_injection tests and should not overrun.
-Jonathan Cavitt

From: Patchwork <patchwork@emeril.freedesktop.org>
Sent: Tuesday, November 19, 2024 5:08 PM
To: Cavitt, Jonathan <jonathan.cavitt@intel.com>
Cc: igt-dev@lists.freedesktop.org
Subject: ✗ Fi.CI.BAT: failure for tests/intel/xe_fault_injection: Ignore expected errors (rev3)

Patch Details
Series:
tests/intel/xe_fault_injection: Ignore expected errors (rev3)
URL:
https://patchwork.freedesktop.org/series/141026/
State:
failure
Details:
https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/index.html
CI Bug Log - changes from IGT_8118 -> IGTPW_12144
Summary

FAILURE

Serious unknown changes coming with IGTPW_12144 absolutely need to be
verified manually.

If you think the reported changes have nothing to do with the changes
introduced in IGTPW_12144, please notify your bug team (I915-ci-infra@lists.freedesktop.org<mailto:I915-ci-infra@lists.freedesktop.org>) to allow them
to document this new failure mode, which will reduce false positives in CI.

External URL: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/index.html

Participating hosts (46 -> 44)

Missing (2): fi-snb-2520m fi-pnv-d510

Possible new issues

Here are the unknown changes that may have been introduced in IGTPW_12144:

IGT changes
Possible regressions

  *   igt@kms_flip@basic-flip-vs-modeset@b-dp1:
     *   fi-cfl-8109u: PASS<https://intel-gfx-ci.01.org/tree/drm-tip/IGT_8118/fi-cfl-8109u/igt@kms_flip@basic-flip-vs-modeset@b-dp1.html> -> DMESG-WARN<https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/fi-cfl-8109u/igt@kms_flip@basic-flip-vs-modeset@b-dp1.html>

Known issues

Here are the changes found in IGTPW_12144 that come from known issues:

IGT changes
Issues hit

  *   igt@i915_selftest@live:
     *   bat-twl-1: NOTRUN -> ABORT<https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/bat-twl-1/igt@i915_selftest@live.html> (i915#12919<https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12919>) +1 other test abort
     *   bat-mtlp-6: PASS<https://intel-gfx-ci.01.org/tree/drm-tip/IGT_8118/bat-mtlp-6/igt@i915_selftest@live.html> -> ABORT<https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/bat-mtlp-6/igt@i915_selftest@live.html> (i915#12829<https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12829>)
  *   igt@i915_selftest@live@workarounds:
     *   bat-mtlp-6: PASS<https://intel-gfx-ci.01.org/tree/drm-tip/IGT_8118/bat-mtlp-6/igt@i915_selftest@live@workarounds.html> -> ABORT<https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/bat-mtlp-6/igt@i915_selftest@live@workarounds.html> (i915#12915<https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12915>)
  *   igt@kms_flip@basic-flip-vs-modeset:
     *   fi-cfl-8109u: PASS<https://intel-gfx-ci.01.org/tree/drm-tip/IGT_8118/fi-cfl-8109u/igt@kms_flip@basic-flip-vs-modeset.html> -> DMESG-WARN<https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/fi-cfl-8109u/igt@kms_flip@basic-flip-vs-modeset.html> (i915#12914<https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12914>)
  *   igt@runner@aborted:
     *   bat-dg2-13: NOTRUN -> FAIL<https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/bat-dg2-13/igt@runner@aborted.html> (i915#12658<https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12658>)

Possible fixes

  *   igt@i915_module_load@load:
     *   bat-twl-2: DMESG-WARN<https://intel-gfx-ci.01.org/tree/drm-tip/IGT_8118/bat-twl-2/igt@i915_module_load@load.html> (i915#1982<https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/1982>) -> PASS<https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/bat-twl-2/igt@i915_module_load@load.html>
  *   igt@i915_pm_rpm@module-reload:
     *   bat-dg2-11: FAIL<https://intel-gfx-ci.01.org/tree/drm-tip/IGT_8118/bat-dg2-11/igt@i915_pm_rpm@module-reload.html> -> PASS<https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/bat-dg2-11/igt@i915_pm_rpm@module-reload.html>
     *   bat-adls-6: FAIL<https://intel-gfx-ci.01.org/tree/drm-tip/IGT_8118/bat-adls-6/igt@i915_pm_rpm@module-reload.html> (i915#12903<https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12903>) -> PASS<https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/bat-adls-6/igt@i915_pm_rpm@module-reload.html>
  *   igt@i915_selftest@live@gt_mocs:
     *   bat-twl-2: ABORT<https://intel-gfx-ci.01.org/tree/drm-tip/IGT_8118/bat-twl-2/igt@i915_selftest@live@gt_mocs.html> (i915#12919<https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12919>) -> PASS<https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/bat-twl-2/igt@i915_selftest@live@gt_mocs.html>
  *   igt@kms_pipe_crc_basic@nonblocking-crc-frame-sequence:
     *   bat-apl-1: DMESG-WARN<https://intel-gfx-ci.01.org/tree/drm-tip/IGT_8118/bat-apl-1/igt@kms_pipe_crc_basic@nonblocking-crc-frame-sequence.html> -> PASS<https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/bat-apl-1/igt@kms_pipe_crc_basic@nonblocking-crc-frame-sequence.html> +2 other tests pass

Warnings

  *   igt@i915_selftest@live@workarounds:
     *   bat-mtlp-8: ABORT<https://intel-gfx-ci.01.org/tree/drm-tip/IGT_8118/bat-mtlp-8/igt@i915_selftest@live@workarounds.html> -> ABORT<https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_12144/bat-mtlp-8/igt@i915_selftest@live@workarounds.html> (i915#12915<https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/12915>)

Build changes

  *   CI: CI-20190529 -> None
  *   IGT: IGT_8118 -> IGTPW_12144
  *   Linux: CI_DRM_15720 -> CI_DRM_15722

CI-20190529: 20190529
CI_DRM_15720: f8f85a38f6c75e091805f01ceff4041ac2fdf3fd @ git://anongit.freedesktop.org/gfx-ci/linux
CI_DRM_15722: ccf64319402f90f68549e2047a7f985a46436e41 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_12144: 12144
IGT_8118: 17707095f1e5d3c30f463b43022f01c0160579b6 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git

[-- Attachment #2: Type: text/html, Size: 25206 bytes --]