dri-devel Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: Oded Gabbay <ogabbay@kernel.org>
To: dri-devel@lists.freedesktop.org
Cc: Koby Elbaz <kelbaz@habana.ai>
Subject: [PATCH 14/27] habanalabs: disable PCI when escalating compute to hard-reset
Date: Sun, 12 Feb 2023 22:44:41 +0200	[thread overview]
Message-ID: <20230212204454.2938561-14-ogabbay@kernel.org> (raw)
In-Reply-To: <20230212204454.2938561-1-ogabbay@kernel.org>

From: Koby Elbaz <kelbaz@habana.ai>

In case a compute reset has failed or a request for a hard reset has
just arrived, then we escalate current reset procedure from compute
to hard-reset.
In such a case, the FW should be aware of the updated error cause,
and if LKD is the one who performs the reset (rather than the FW),
then we ask the FW to disable PCI access.

We would also like to have relevant debug info and therefore
we print the currently escalating reset type.

Signed-off-by: Koby Elbaz <kelbaz@habana.ai>
Reviewed-by: Oded Gabbay <ogabbay@kernel.org>
Signed-off-by: Oded Gabbay <ogabbay@kernel.org>
---
 drivers/accel/habanalabs/common/device.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/drivers/accel/habanalabs/common/device.c b/drivers/accel/habanalabs/common/device.c
index f91f3509336f..d140eaefc840 100644
--- a/drivers/accel/habanalabs/common/device.c
+++ b/drivers/accel/habanalabs/common/device.c
@@ -1452,7 +1452,7 @@ static void handle_reset_trigger(struct hl_device *hdev, u32 flags)
 		 */
 		if (hl_fw_send_pci_access_msg(hdev, CPUCP_PACKET_DISABLE_PCI_ACCESS, 0x0))
 			dev_warn(hdev->dev,
-				"Failed to disable PCI access by F/W\n");
+				"Failed to disable FW's PCI access\n");
 	}
 }
 
@@ -1530,14 +1530,14 @@ int hl_device_reset(struct hl_device *hdev, u32 flags)
 
 	/*
 	 * Prevent concurrency in this function - only one reset should be
-	 * done at any given time. Only need to perform this if we didn't
-	 * get from the dedicated hard reset thread
+	 * done at any given time. We need to perform this only if we didn't
+	 * get here from a dedicated hard reset thread.
 	 */
 	if (!from_hard_reset_thread) {
 		/* Block future CS/VM/JOB completion operations */
 		spin_lock(&hdev->reset_info.lock);
 		if (hdev->reset_info.in_reset) {
-			/* We only allow scheduling of a hard reset during compute reset */
+			/* We allow scheduling of a hard reset only during a compute reset */
 			if (hard_reset && hdev->reset_info.in_compute_reset)
 				hdev->reset_info.hard_reset_schedule_flags = flags;
 			spin_unlock(&hdev->reset_info.lock);
@@ -1574,6 +1574,7 @@ int hl_device_reset(struct hl_device *hdev, u32 flags)
 		if (delay_reset)
 			usleep_range(HL_RESET_DELAY_USEC, HL_RESET_DELAY_USEC << 1);
 
+escalate_reset_flow:
 		handle_reset_trigger(hdev, flags);
 
 		/* This also blocks future CS/VM/JOB completion operations */
@@ -1589,7 +1590,6 @@ int hl_device_reset(struct hl_device *hdev, u32 flags)
 			dev_dbg(hdev->dev, "Going to reset engines of inference device\n");
 	}
 
-again:
 	if ((hard_reset) && (!from_hard_reset_thread)) {
 		hdev->reset_info.hard_reset_pending = true;
 
@@ -1837,7 +1837,7 @@ int hl_device_reset(struct hl_device *hdev, u32 flags)
 			hdev->disabled = true;
 			hard_reset = true;
 			handle_reset_trigger(hdev, flags);
-			goto again;
+			goto escalate_reset_flow;
 		}
 	}
 
@@ -1860,14 +1860,14 @@ int hl_device_reset(struct hl_device *hdev, u32 flags)
 		flags |= HL_DRV_RESET_HARD;
 		flags &= ~HL_DRV_RESET_DEV_RELEASE;
 		hard_reset = true;
-		goto again;
+		goto escalate_reset_flow;
 	} else {
 		spin_unlock(&hdev->reset_info.lock);
 		dev_err(hdev->dev, "Failed to do compute reset\n");
 		hdev->reset_info.compute_reset_cnt++;
 		flags |= HL_DRV_RESET_HARD;
 		hard_reset = true;
-		goto again;
+		goto escalate_reset_flow;
 	}
 
 	hdev->reset_info.in_reset = 0;
-- 
2.25.1


  parent reply	other threads:[~2023-02-12 20:45 UTC|newest]

Thread overview: 50+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-02-12 20:44 [PATCH 01/27] habanalabs/gaudi2: increase user interrupt grace time Oded Gabbay
2023-02-12 20:44 ` [PATCH 02/27] habanalabs/gaudi: capture RAZWI info only if HW indication detected Oded Gabbay
2023-02-12 20:44 ` [PATCH 03/27] habanalabs: split cdev creation to separate function Oded Gabbay
2023-02-16 10:40   ` Tomer Tayar
2023-02-12 20:44 ` [PATCH 04/27] habanalabs: save class in hdev Oded Gabbay
2023-02-16 10:40   ` Tomer Tayar
2023-02-12 20:44 ` [PATCH 05/27] habanalabs: refactor debugfs init Oded Gabbay
2023-02-16 10:40   ` Tomer Tayar
2023-02-12 20:44 ` [PATCH 06/27] habanalabs: use memhash_node_export_put() in hl_release_dmabuf() Oded Gabbay
2023-02-16 11:48   ` Stanislaw Gruszka
2023-02-16 14:26     ` Tomer Tayar
2023-02-16 14:40       ` Stanislaw Gruszka
2023-02-12 20:44 ` [PATCH 07/27] habanalabs/gaudi2: fix address decode RAZWI handling Oded Gabbay
2023-02-12 20:44 ` [PATCH 08/27] habanalabs: add info when FD released while device still in use Oded Gabbay
2023-02-16 12:25   ` Stanislaw Gruszka
2023-02-16 14:21     ` Oded Gabbay
2023-02-16 15:04       ` Stanislaw Gruszka
2023-02-17 11:34         ` Tomer Tayar
2023-02-20 15:54           ` Stanislaw Gruszka
2023-02-20 16:16             ` Tomer Tayar
2023-02-12 20:44 ` [PATCH 09/27] habanalabs: enforce release order of compute device and dma-buf Oded Gabbay
2023-02-12 20:44 ` [PATCH 10/27] habanalabs: add critical-event bit in notifier Oded Gabbay
2023-02-12 20:44 ` [PATCH 11/27] habanalabs/gaudi2: expose engine core int reg address Oded Gabbay
2023-02-12 20:44 ` [PATCH 12/27] habanalabs/gaudi2: unsecure CFG_TPC_ID register Oded Gabbay
2023-02-12 20:44 ` [PATCH 13/27] habanalabs: minimize error prints when mem map fails Oded Gabbay
2023-02-12 20:44 ` Oded Gabbay [this message]
2023-02-12 20:44 ` [PATCH 15/27] habanalabs: enable graceful reset mechanism for compute-reset Oded Gabbay
2023-02-12 20:44 ` [PATCH 16/27] habanalabs/gaudi2: get reset type indication from irq_map Oded Gabbay
2023-02-12 20:44 ` [PATCH 17/27] habanalabs/gaudi2: modify events reset policy Oded Gabbay
2023-02-12 20:44 ` [PATCH 18/27] habanalabs: change user interrupt to threaded IRQ Oded Gabbay
2023-02-16 10:28   ` Stanislaw Gruszka
2023-02-16 13:47     ` Oded Gabbay
2023-02-16 14:29       ` Stanislaw Gruszka
2023-02-16 10:39   ` Stanislaw Gruszka
2023-02-16 13:49     ` Oded Gabbay
2023-02-12 20:44 ` [PATCH 19/27] habanalabs: capture interrupt timestamp in handler Oded Gabbay
2023-02-16 14:39   ` Stanislaw Gruszka
2023-02-19 12:42     ` Ofir Bitton
2023-02-12 20:44 ` [PATCH 20/27] habanalabs/gaudi2: add support for TPC assert Oded Gabbay
2023-02-12 20:44 ` [PATCH 21/27] habanalabs: fix print in hl_irq_handler_eq() Oded Gabbay
2023-02-12 20:44 ` [PATCH 22/27] habanalabs: remove hl_irq_handler_default() Oded Gabbay
2023-02-12 20:44 ` [PATCH 23/27] habanalabs: tiny refactor of hl_device_reset for readability Oded Gabbay
2023-02-12 20:44 ` [PATCH 24/27] habanalabs: rename security function parameters Oded Gabbay
2023-02-12 20:44 ` [PATCH 25/27] habanalabs: in hl_device_reset remove 'hard_instead_of_soft' Oded Gabbay
2023-02-12 20:44 ` [PATCH 26/27] habanalabs: in hl_device_reset small refactor for readabilty Oded Gabbay
2023-02-12 20:44 ` [PATCH 27/27] habanalabs: don't trace cpu accessible dma alloc/free Oded Gabbay
2023-02-16 10:53 ` [PATCH 01/27] habanalabs/gaudi2: increase user interrupt grace time Stanislaw Gruszka
2023-02-16 14:24   ` Oded Gabbay
2023-02-18 19:13     ` Ofir Bitton
2023-02-20 15:31 ` Stanislaw Gruszka

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20230212204454.2938561-14-ogabbay@kernel.org \
    --to=ogabbay@kernel.org \
    --cc=dri-devel@lists.freedesktop.org \
    --cc=kelbaz@habana.ai \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox