From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5DA2BC54ED0 for ; Fri, 23 May 2025 11:36:27 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 1632B10E2B8; Fri, 23 May 2025 11:36:27 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="auW58eJE"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.17]) by gabe.freedesktop.org (Postfix) with ESMTPS id 15ADC10E2B8 for ; Fri, 23 May 2025 11:36:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1748000186; x=1779536186; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=N4GHyOAfAgnXY1ldFLDtCrlg925+Kudsrn+llzgfbuc=; b=auW58eJEylq8bP1NCXSXNDShe6tBxAE8E7OBw6gOjvWf2rNZ9OZthF5b xf3tltszS9hl+gE99OvLFpsVVlu/Abk5Hgy7G0MCc/G6dN5va4XKMV0/f jej5DX9uiW0kljTSrUAA1XkyqC3ocE6Tx8UZ1HzMXbFREyNxi7QZe9pz1 8/qfNsvlai8+/F5jyALFqYKp3AWMMz3Lb4y1+HD2pInaZb9rW2O1/WDIk J6y1WMtmA11Gno8s1YIkAHzgR8vWSzMR77QxPZStueVJt+14aE6obesSu SgoKU4SAgKhZGEzarqWL5s9C0q8xX/NCNr8GHiikmbp8Jx3cmHvsCAP6v g==; X-CSE-ConnectionGUID: Ij4mgkEQSnmn+3UpZSTezQ== X-CSE-MsgGUID: iRubCmiaRcyTa73nG1hYuw== X-IronPort-AV: E=McAfee;i="6700,10204,11441"; a="50048703" X-IronPort-AV: E=Sophos;i="6.15,308,1739865600"; d="scan'208";a="50048703" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa109.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2025 04:36:25 -0700 X-CSE-ConnectionGUID: 5XCxXOXuSCCHjEop3Mm5HA== X-CSE-MsgGUID: wl1F8OmST9SWhjrBBBEuOA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.15,308,1739865600"; d="scan'208";a="141526196" Received: from dprybysh-mobl.ger.corp.intel.com (HELO mwauld-desk.intel.com) ([10.245.244.145]) by fmviesa008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2025 04:36:22 -0700 From: Matthew Auld To: intel-xe@lists.freedesktop.org Cc: =?UTF-8?q?Thomas=20Hellstr=C3=B6m?= , Matthew Brost , William Tseng , stable@vger.kernel.org Subject: [PATCH] drm/xe/sched: stop re-submitting signalled jobs Date: Fri, 23 May 2025 12:36:09 +0100 Message-ID: <20250523113608.121553-2-matthew.auld@intel.com> X-Mailer: git-send-email 2.49.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: intel-xe@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Xe graphics driver List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-xe-bounces@lists.freedesktop.org Sender: "Intel-xe" Customer is reporting a really subtle issue where we get random DMAR faults, hangs and other nasties for kernel migration jobs when stressing stuff like s2idle/s3/s4. The explosions seems to happen somewhere after resuming the system with splats looking something like: PM: suspend exit rfkill: input handler disabled xe 0000:00:02.0: [drm] GT0: Engine reset: engine_class=bcs, logical_mask: 0x2, guc_id=0 xe 0000:00:02.0: [drm] GT0: Timedout job: seqno=24496, lrc_seqno=24496, guc_id=0, flags=0x13 in no process [-1] xe 0000:00:02.0: [drm] GT0: Kernel-submitted job timed out The likely cause appears to be a race between suspend cancelling the worker that processes the free_job()'s, such that we still have pending jobs to be freed after the cancel. Following from this, on resume the pending_list will now contain at least one already complete job, but it looks like we call drm_sched_resubmit_jobs(), which will then call run_job() on everything still on the pending_list. But if the job was already complete, then all the resources tied to the job, like the bb itself, any memory that is being accessed, the iommu mappings etc. might be long gone since those are usually tied to the fence signalling. This scenario can be seen in ftrace when running a slightly modified xe_pm (kernel was only modified to inject artificial latency into free_job to make the race easier to hit): xe_sched_job_run: dev=0000:00:02.0, fence=0xffff888276cc8540, seqno=0, lrc_seqno=0, gt=0, guc_id=0, batch_addr=0x000000146910 ... xe_exec_queue_stop: dev=0000:00:02.0, 3:0x2, gt=0, width=1, guc_id=0, guc_state=0x0, flags=0x13 xe_exec_queue_stop: dev=0000:00:02.0, 3:0x2, gt=0, width=1, guc_id=1, guc_state=0x0, flags=0x4 xe_exec_queue_stop: dev=0000:00:02.0, 4:0x1, gt=1, width=1, guc_id=0, guc_state=0x0, flags=0x3 xe_exec_queue_stop: dev=0000:00:02.0, 1:0x1, gt=1, width=1, guc_id=1, guc_state=0x0, flags=0x3 xe_exec_queue_stop: dev=0000:00:02.0, 4:0x1, gt=1, width=1, guc_id=2, guc_state=0x0, flags=0x3 xe_exec_queue_resubmit: dev=0000:00:02.0, 3:0x2, gt=0, width=1, guc_id=0, guc_state=0x0, flags=0x13 xe_sched_job_run: dev=0000:00:02.0, fence=0xffff888276cc8540, seqno=0, lrc_seqno=0, gt=0, guc_id=0, batch_addr=0x000000146910 ... ..... xe_exec_queue_memory_cat_error: dev=0000:00:02.0, 3:0x2, gt=0, width=1, guc_id=0, guc_state=0x3, flags=0x13 So the job_run() is clearly triggered twice, even though the first must have already signalled to completion during suspend. We can also see a CAT error after the re-submit. To prevent this try to call xe_sched_stop() to forcefully remove anything on the pending_list that has already signalled, before we re-submit. Link: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/4856 Fixes: dd08ebf6c352 ("drm/xe: Introduce a new DRM driver for Intel GPUs") Signed-off-by: Matthew Auld Cc: Thomas Hellström Cc: Matthew Brost Cc: William Tseng Cc: # v6.8+ --- drivers/gpu/drm/xe/xe_gpu_scheduler.h | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/gpu/drm/xe/xe_gpu_scheduler.h b/drivers/gpu/drm/xe/xe_gpu_scheduler.h index c250ea773491..9315da58d02d 100644 --- a/drivers/gpu/drm/xe/xe_gpu_scheduler.h +++ b/drivers/gpu/drm/xe/xe_gpu_scheduler.h @@ -51,6 +51,7 @@ static inline void xe_sched_tdr_queue_imm(struct xe_gpu_scheduler *sched) static inline void xe_sched_resubmit_jobs(struct xe_gpu_scheduler *sched) { + xe_sched_stop(sched); drm_sched_resubmit_jobs(&sched->base); } -- 2.49.0