From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6090DCC6B2F for ; Thu, 15 Oct 2026 05:06:08 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id F18ED11209F; Thu, 15 Oct 2026 05:06:02 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.9]) by gabe.freedesktop.org (Postfix) with ESMTPS id 26D4110E7C7 for ; Thu, 15 Oct 2026 05:05:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1792040757; x=1823576757; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=fT5UWy+djWgLi1wR9BALTr8AeXs58L7drgM42mFw0ss=; b=MFEM/Ea6/rlan0gHYnrQy+DeE76YLrjA96q9jH3PzJ5Mzxi0gCuRqaF4 QOfejsSphzuzzeMI3yqdwYOsSRqXqG6MxyRaE2Wu2lGIV4L4/M3tpPakT 3xU6IvlYU7xvBWWRtMxoYjkxNJMbq55Mr7QNYnd3vxSgxmimESX/XLrRn C6ugp616A2NFwaR7RFk6keHAbPEQuS3sT+hqi/mzxiu3jpnWOIWvU9dUi Hq6iesJO9L/4MI7jTTGgzYE3hgAMUxR5jeo3jecPTDRgmJ+ygQXhbNY0G 2YiXMLH0MtUqnL1CjpvKDAldnExIDAxLmAc8OvTA2IiFttUXPG6lSrcdE w==; X-CSE-ConnectionGUID: bnlfPjgbQ86b3lBl7no/LA== X-CSE-MsgGUID: jv17PWgqRAekm2WlJwomBQ== X-IronPort-AV: E=McAfee;i="6800,10657,11935"; a="687119" X-IronPort-AV: E=Sophos;i="6.27,159,1787036400"; d="scan'208";a="687119" Received: from orviesa009.jf.intel.com ([10.64.159.149]) by orvoesa101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Oct 2026 22:05:51 -0700 X-CSE-ConnectionGUID: FSxYV8NxSnqqRnKn88tPwg== X-CSE-MsgGUID: xIS3ugaqQhe6N7Ewnsqauw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.27,159,1787036400"; d="scan'208";a="1525113" Received: from danielcm-desk1.jf.intel.com ([10.24.81.120]) by orviesa009.jf.intel.com with ESMTP; 14 Oct 2026 22:05:52 -0700 From: Daniel Charles To: intel-xe@lists.freedesktop.org Cc: x.wang@intel.com, jia.yao@intel.com, jonathan.cavitt@intel.com, matthew.brost@intel.com Subject: [PATCH v3 2/3] drm/xe: Disable scheduling early on FD close to avoid CAT error cascade Date: Wed, 14 Oct 2026 22:01:35 -0700 Message-ID: <20261015050136.1069815-3-daniel.charles@intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20261015050136.1069815-1-daniel.charles@intel.com> References: <20261015050136.1069815-1-daniel.charles@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: intel-xe@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Xe graphics driver List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-xe-bounces@lists.freedesktop.org Sender: "Intel-xe" From: Matthew Brost When an FD is closed with many exec queues, teardown relies on the TDR path to clean up scheduling. However, the TDR handling is serialized (i.e., only one exec queue is processed at a time), which can make it too slow compared to GuC scheduling activity. In this window, GuC may continue to schedule contexts backed by invalid page tables, leading to a cascade of CAT errors and repeated engine resets. This significantly increases recovery time and can degrade system stability. To mitigate this, eagerly disable scheduling by sending a self-message outside of the TDR path. This prevents further scheduling of invalid contexts and avoids the CAT error/reset cascade. This change improves robustness and reduces recovery latency in multiple queue teardown scenarios. Cc: Wang Xin Cc: Jia Yao Signed-off-by: Matthew Brost Tested-by: Jia Yao Reviewed-by: Jonathan Cavitt --- drivers/gpu/drm/xe/xe_guc_exec_queue_types.h | 2 +- drivers/gpu/drm/xe/xe_guc_submit.c | 46 ++++++++++++++++++-- 2 files changed, 43 insertions(+), 5 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_guc_exec_queue_types.h b/drivers/gpu/drm/xe/xe_guc_exec_queue_types.h index 74b711abe2572..89397e1e65465 100644 --- a/drivers/gpu/drm/xe/xe_guc_exec_queue_types.h +++ b/drivers/gpu/drm/xe/xe_guc_exec_queue_types.h @@ -36,7 +36,7 @@ struct xe_guc_exec_queue { * a message needs to sent through the GPU scheduler but memory * allocations are not allowed. */ -#define MAX_STATIC_MSG_TYPE 4 +#define MAX_STATIC_MSG_TYPE 5 struct xe_sched_msg static_msgs[MAX_STATIC_MSG_TYPE]; /** @destroy_async: do final destroy async from this worker */ struct work_struct destroy_async; diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c index c4ef3e9bf7a18..609af66b343d5 100644 --- a/drivers/gpu/drm/xe/xe_guc_submit.c +++ b/drivers/gpu/drm/xe/xe_guc_submit.c @@ -2045,12 +2045,23 @@ static void __guc_exec_queue_process_msg_cgp_sync(struct xe_sched_msg *msg, guc_exec_queue_send_cgp_sync(q, 0); } +static void __guc_exec_queue_process_msg_kill(struct xe_sched_msg *msg, + bool bound) +{ + struct xe_exec_queue *q = msg->private_data; + struct xe_exec_queue *primary = xe_exec_queue_multi_queue_primary(q); + + if (exec_queue_enabled(primary) && bound) + disable_scheduling(primary, true); +} + #define CLEANUP 1 /* Non-zero values to catch uninitialized msg */ #define SET_SCHED_PROPS 2 #define SUSPEND 3 #define RESUME 4 #define SET_MULTI_QUEUE_PRIORITY 5 #define CGP_SYNC_MSG 6 +#define KILL 7 #define OPCODE_MASK 0xf #define MSG_LOCKED BIT(8) #define MSG_HEAD BIT(9) @@ -2085,6 +2096,9 @@ static void guc_exec_queue_process_msg(struct xe_sched_msg *msg, case CGP_SYNC_MSG: __guc_exec_queue_process_msg_cgp_sync(msg, bound); break; + case KILL: + __guc_exec_queue_process_msg_kill(msg, bound); + break; default: XE_WARN_ON("Unknown message type"); } @@ -2211,8 +2225,19 @@ static int guc_exec_queue_init(struct xe_exec_queue *q) return err; } +static bool guc_exec_queue_try_add_msg(struct xe_exec_queue *q, + struct xe_sched_msg *msg, + u32 opcode); + +#define STATIC_MSG_CLEANUP 0 +#define STATIC_MSG_SUSPEND 1 +#define STATIC_MSG_RESUME 2 +#define STATIC_MSG_CGP_SYNC 3 +#define STATIC_MSG_KILL 4 static void guc_exec_queue_kill(struct xe_exec_queue *q) { + struct xe_sched_msg *msg = q->guc->static_msgs + STATIC_MSG_KILL; + trace_xe_exec_queue_kill(q); set_exec_queue_killed(q); /* @@ -2222,6 +2247,23 @@ static void guc_exec_queue_kill(struct xe_exec_queue *q) * stays set until the queue is torn down. */ suspend_wait_wake(q); + + /* + * We eagerly send a message to ourselves to disable scheduling, as the + * TDR is serialized (i.e., only one exec queue is processed at a time). + * If an FD is closed with many exec queues, the TDR can be slower than + * the GuC scheduling contexts with invalid page tables, creating a + * cascade of CAT errors and engine resets, which is quite slow. Avoid + * this by immediately disabling scheduling outside of the TDR. + */ + if (kref_read(&q->refcount) && !exec_queue_wedged(q)) { + struct xe_gpu_scheduler *sched = &q->guc->sched; + + xe_sched_msg_lock(sched); + guc_exec_queue_try_add_msg(q, msg, KILL); + xe_sched_msg_unlock(sched); + } + xe_guc_exec_queue_trigger_cleanup(q); } @@ -2274,10 +2316,6 @@ static bool guc_exec_queue_try_add_msg(struct xe_exec_queue *q, return true; } -#define STATIC_MSG_CLEANUP 0 -#define STATIC_MSG_SUSPEND 1 -#define STATIC_MSG_RESUME 2 -#define STATIC_MSG_CGP_SYNC 3 static void guc_exec_queue_destroy(struct xe_exec_queue *q) { struct xe_sched_msg *msg = q->guc->static_msgs + STATIC_MSG_CLEANUP; -- 2.43.0