[PATCH 1/2] habanalabs: halt debug engines on user process close

[PATCH 1/2] habanalabs: halt debug engines on user process close

[Oded Gabbay]

From: Omer Shpigelman <oshpigelman@habana.ai>

This patch fix a potential bug where a user's process has closed
unexpectedly without disabling the debug engines. In that case, the debug
engines might continue running but because the user's MMU mappings are
going away, we will get page fault errors.

This behavior is also opposed to the general rule where nothing runs on
the device after the user process closes.

The patch stops the debug H/W engines upon process termination and thus
makes sure nothing runs on the device after the process goes away.

Signed-off-by: Omer Shpigelman <oshpigelman@habana.ai>
Signed-off-by: Oded Gabbay <oded.gabbay@gmail.com>
---
 drivers/misc/habanalabs/context.c             |  6 ++++++
 drivers/misc/habanalabs/goya/goya.c           |  3 ++-
 drivers/misc/habanalabs/goya/goyaP.h          |  1 +
 drivers/misc/habanalabs/goya/goya_coresight.c | 17 +++++++++++++++++
 drivers/misc/habanalabs/habanalabs.h          |  2 ++
 5 files changed, 28 insertions(+), 1 deletion(-)

diff --git a/drivers/misc/habanalabs/context.c b/drivers/misc/habanalabs/context.c
index 4804cdcf4c48..f4c92f110a72 100644
--- a/drivers/misc/habanalabs/context.c
+++ b/drivers/misc/habanalabs/context.c
@@ -26,6 +26,12 @@ static void hl_ctx_fini(struct hl_ctx *ctx)
 		dma_fence_put(ctx->cs_pending[i]);
 
 	if (ctx->asid != HL_KERNEL_ASID_ID) {
+		/*
+		 * The engines are stopped as there is no executing CS, but the
+		 * Coresight might be still working by accessing addresses
+		 * related to the stopped engines. Hence stop it explicitly.
+		 */
+		hdev->asic_funcs->halt_coresight(hdev);
 		hl_vm_ctx_fini(ctx);
 		hl_asid_free(hdev, ctx->asid);
 	}
diff --git a/drivers/misc/habanalabs/goya/goya.c b/drivers/misc/habanalabs/goya/goya.c
index a582e29c1ee4..02d116b01a1a 100644
--- a/drivers/misc/habanalabs/goya/goya.c
+++ b/drivers/misc/habanalabs/goya/goya.c
@@ -4819,7 +4819,8 @@ static const struct hl_asic_funcs goya_funcs = {
 	.set_dram_bar_base = goya_set_ddr_bar_base,
 	.init_iatu = goya_init_iatu,
 	.rreg = hl_rreg,
-	.wreg = hl_wreg
+	.wreg = hl_wreg,
+	.halt_coresight = goya_halt_coresight
 };
 
 /*
diff --git a/drivers/misc/habanalabs/goya/goyaP.h b/drivers/misc/habanalabs/goya/goyaP.h
index 14e216cb3668..c83cab0d641e 100644
--- a/drivers/misc/habanalabs/goya/goyaP.h
+++ b/drivers/misc/habanalabs/goya/goyaP.h
@@ -202,6 +202,7 @@ void goya_add_device_attr(struct hl_device *hdev,
 			struct attribute_group *dev_attr_grp);
 int goya_armcp_info_get(struct hl_device *hdev);
 int goya_debug_coresight(struct hl_device *hdev, void *data);
+void goya_halt_coresight(struct hl_device *hdev);
 
 void goya_mmu_prepare(struct hl_device *hdev, u32 asid);
 int goya_mmu_clear_pgt_range(struct hl_device *hdev);
diff --git a/drivers/misc/habanalabs/goya/goya_coresight.c b/drivers/misc/habanalabs/goya/goya_coresight.c
index 1ac951f52d1e..49b5efb3a912 100644
--- a/drivers/misc/habanalabs/goya/goya_coresight.c
+++ b/drivers/misc/habanalabs/goya/goya_coresight.c
@@ -626,3 +626,20 @@ int goya_debug_coresight(struct hl_device *hdev, void *data)
 
 	return rc;
 }
+
+void goya_halt_coresight(struct hl_device *hdev)
+{
+	struct hl_debug_params params = {0};
+	int i, rc;
+
+	for (i = GOYA_ETF_FIRST ; i <= GOYA_ETF_LAST ; i++) {
+		params.reg_idx = i;
+		rc = goya_config_etf(hdev, &params);
+		if (rc)
+			dev_err(hdev->dev, "halt ETF failed, %d/%d\n", rc, i);
+	}
+
+	rc = goya_config_etr(hdev, &params);
+	if (rc)
+		dev_err(hdev->dev, "halt ETR failed, %d\n", rc);
+}
diff --git a/drivers/misc/habanalabs/habanalabs.h b/drivers/misc/habanalabs/habanalabs.h
index 71243b319920..adef7d9d7488 100644
--- a/drivers/misc/habanalabs/habanalabs.h
+++ b/drivers/misc/habanalabs/habanalabs.h
@@ -501,6 +501,7 @@ enum hl_pll_frequency {
  * @init_iatu: Initialize the iATU unit inside the PCI controller.
  * @rreg: Read a register. Needed for simulator support.
  * @wreg: Write a register. Needed for simulator support.
+ * @halt_coresight: stop the ETF and ETR traces.
  */
 struct hl_asic_funcs {
 	int (*early_init)(struct hl_device *hdev);
@@ -578,6 +579,7 @@ struct hl_asic_funcs {
 	int (*init_iatu)(struct hl_device *hdev);
 	u32 (*rreg)(struct hl_device *hdev, u32 reg);
 	void (*wreg)(struct hl_device *hdev, u32 reg, u32 val);
+	void (*halt_coresight)(struct hl_device *hdev);
 };
 
 
-- 
2.17.1

[PATCH 2/2] uapi/habanalabs: add opcode for enable/disable device debug mode

[Oded Gabbay]

This patch defines a new opcode in the DEBUG IOCTL that is used by the
user to notify the driver when the user wants to start or stop using the
debug and profile infrastructure of the device. i.e. set the device to
debug mode or to non-debug mode.

There are a couple of restrictions that this new opcode introduces:

1. The user can't configure the debug/profiling infrastructure before he
   sets the device to debug mode, by using this new opcode.

2. The user can't set the device to debug mode unless he is the only user
   that is currently using (has an open FD) the device.

3. Other users can't use the device (open a new FD) in case an existing
   user has set the device into debug mode.

These restrictions are needed because the debug and profiling
infrastructure is a shared component in the ASIC and therefore, can't be
used while multiple users are working on the device.

Because the driver currently does NOT support multiple users, the
implementation of the restrictions is not required at this point. However,
the interface definition is needed in order to avoid changing the user API
later on.

Signed-off-by: Oded Gabbay <oded.gabbay@gmail.com>
---
 include/uapi/misc/habanalabs.h | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)

diff --git a/include/uapi/misc/habanalabs.h b/include/uapi/misc/habanalabs.h
index 8ac292cf4d00..204ab9b4ae67 100644
--- a/include/uapi/misc/habanalabs.h
+++ b/include/uapi/misc/habanalabs.h
@@ -413,6 +413,10 @@ struct hl_debug_params_spmu {
 #define HL_DEBUG_OP_SPMU	5
 /* Opcode for timestamp */
 #define HL_DEBUG_OP_TIMESTAMP	6
+/* Opcode for setting the device into or out of debug mode. The enable
+ * variable should be 1 for enabling debug mode and 0 for disabling it
+ */
+#define HL_DEBUG_OP_SET_MODE	7
 
 struct hl_debug_args {
 	/*
@@ -574,8 +578,22 @@ struct hl_debug_args {
  *
  * This IOCTL allows the user to get debug traces from the chip.
  *
- * The user needs to provide the register index and essential data such as
- * buffer address and size.
+ * Before the user can send configuration requests of the various
+ * debug/profile engines, it needs to set the device into debug mode.
+ * This is because the debug/profile infrastructure is shared component in the
+ * device and we can't allow multiple users to access it at the same time.
+ *
+ * Once a user set the device into debug mode, the driver won't allow other
+ * users to "work" with the device, i.e. open a FD. If there are multiple users
+ * opened on the device, the driver won't allow any user to debug the device.
+ *
+ * For each configuration request, the user needs to provide the register index
+ * and essential data such as buffer address and size.
+ *
+ * Once the user has finished using the debug/profile engines, he should
+ * set the device into non-debug mode, i.e. disable debug mode.
+ *
+ * The driver can decide to "kick out" the user if he abuses this interface.
  *
  */
 #define HL_IOCTL_DEBUG		\
-- 
2.17.1