[PATCH 0/4] Coresight TMC-ETR some bugfixes and cleanups

[PATCH 0/4] Coresight TMC-ETR some bugfixes and cleanups

[Junhao He]

This patchset builds upon Yicong's previous patches [1].

Introducing fix two race issues found by using TMC-ETR and CATU, Two
cleanups found when debugging the issues.

[1] https://lore.kernel.org/linux-arm-kernel/20241202092419.11777-1-yangyicong@huawei.com/

Junhao He (2):
  coresight: catu: add locking to catu enable/disable functions
  coresight: tmc: refactor the tmc-etr mode setting

Yicong Yang (2):
  coresight: tmc: Add missing doc of tmc_drvdata::reading
  coresight: tmc-etr: Decouple the perf buffer allocation from sysfs
    mode

 drivers/hwtracing/coresight/coresight-catu.c  |   6 +
 drivers/hwtracing/coresight/coresight-catu.h  |   1 +
 .../hwtracing/coresight/coresight-tmc-etr.c   | 106 +++++++++---------
 drivers/hwtracing/coresight/coresight-tmc.h   |   1 +
 4 files changed, 64 insertions(+), 50 deletions(-)

-- 
2.33.0

[PATCH 1/4] coresight: tmc: Add missing doc of tmc_drvdata::reading

[Junhao He]

From: Yicong Yang <yangyicong@hisilicon.com>

tmc_drvdata::reading is used to indicate whether a reading process
is performed through /dev/xyz.tmc. Document it.

Reviewed-by: James Clark <james.clark@linaro.org>
Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
Signed-off-by: Junhao He <hejunhao3@huawei.com>
---
 drivers/hwtracing/coresight/coresight-tmc.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/hwtracing/coresight/coresight-tmc.h b/drivers/hwtracing/coresight/coresight-tmc.h
index 2671926be62a..fdf7955e7350 100644
--- a/drivers/hwtracing/coresight/coresight-tmc.h
+++ b/drivers/hwtracing/coresight/coresight-tmc.h
@@ -174,6 +174,7 @@ struct etr_buf {
  * @pid:	Process ID of the process that owns the session that is using
  *		this component. For example this would be the pid of the Perf
  *		process.
+ * @reading:	buffer's in the reading through "/dev/xyz.tmc" entry
  * @buf:	Snapshot of the trace data for ETF/ETB.
  * @etr_buf:	details of buffer used in TMC-ETR
  * @len:	size of the available trace for ETF/ETB.
-- 
2.33.0

[PATCH 2/4] coresight: catu: add locking to catu enable/disable functions

[Junhao He]

In the function coresight_enable_path(), all adjacent helper components
along the path are enabled first. Without proper locking, a race
condition may occur between concurrent sysfs and perf calls to these
functions, potentially leading to incorrectly obtained settings or
misconfigured CATU control-enable flags.

Requires spinlock in catu_enable()/catu_disable() to:
- Serialize sysfs/perf concurrent accesses
- Prevent corruption of CATU control-enable flags

Signed-off-by: Junhao He <hejunhao3@huawei.com>
---
 drivers/hwtracing/coresight/coresight-catu.c | 6 ++++++
 drivers/hwtracing/coresight/coresight-catu.h | 1 +
 2 files changed, 7 insertions(+)

diff --git a/drivers/hwtracing/coresight/coresight-catu.c b/drivers/hwtracing/coresight/coresight-catu.c
index 275cc0d9f505..54ce13254d7c 100644
--- a/drivers/hwtracing/coresight/coresight-catu.c
+++ b/drivers/hwtracing/coresight/coresight-catu.c
@@ -461,6 +461,8 @@ static int catu_enable(struct coresight_device *csdev, enum cs_mode mode,
 	int rc;
 	struct catu_drvdata *catu_drvdata = csdev_to_catu_drvdata(csdev);
 
+	guard(spinlock_irqsave)(&catu_drvdata->spinlock);
+
 	CS_UNLOCK(catu_drvdata->base);
 	rc = catu_enable_hw(catu_drvdata, mode, data);
 	CS_LOCK(catu_drvdata->base);
@@ -489,6 +491,8 @@ static int catu_disable(struct coresight_device *csdev, void *__unused)
 	int rc;
 	struct catu_drvdata *catu_drvdata = csdev_to_catu_drvdata(csdev);
 
+	guard(spinlock_irqsave)(&catu_drvdata->spinlock);
+
 	CS_UNLOCK(catu_drvdata->base);
 	rc = catu_disable_hw(catu_drvdata);
 	CS_LOCK(catu_drvdata->base);
@@ -523,6 +527,8 @@ static int __catu_probe(struct device *dev, struct resource *res)
 		goto out;
 	}
 
+	spin_lock_init(&drvdata->spinlock);
+
 	/* Setup dma mask for the device */
 	dma_mask = readl_relaxed(base + CORESIGHT_DEVID) & 0x3f;
 	switch (dma_mask) {
diff --git a/drivers/hwtracing/coresight/coresight-catu.h b/drivers/hwtracing/coresight/coresight-catu.h
index 141feac1c14b..eb7c9189b066 100644
--- a/drivers/hwtracing/coresight/coresight-catu.h
+++ b/drivers/hwtracing/coresight/coresight-catu.h
@@ -64,6 +64,7 @@ struct catu_drvdata {
 	struct clk *pclk;
 	void __iomem *base;
 	struct coresight_device *csdev;
+	spinlock_t spinlock;
 	int irq;
 };
 
-- 
2.33.0

[PATCH 3/4] coresight: tmc: refactor the tmc-etr mode setting

[Junhao He]

When trying to run perf and sysfs mode simultaneously, the WARN_ON()
in tmc_etr_enable_hw() is triggered sometimes:

 WARNING: CPU: 42 PID: 3911571 at drivers/hwtracing/coresight/coresight-tmc-etr.c:1060 tmc_etr_enable_hw+0xc0/0xd8 [coresight_tmc]
 [..snip..]
 Call trace:
  tmc_etr_enable_hw+0xc0/0xd8 [coresight_tmc] (P)
  tmc_enable_etr_sink+0x11c/0x250 [coresight_tmc] (L)
  tmc_enable_etr_sink+0x11c/0x250 [coresight_tmc]
  coresight_enable_path+0x1c8/0x218 [coresight]
  coresight_enable_sysfs+0xa4/0x228 [coresight]
  enable_source_store+0x58/0xa8 [coresight]
  dev_attr_store+0x20/0x40
  sysfs_kf_write+0x4c/0x68
  kernfs_fop_write_iter+0x120/0x1b8
  vfs_write+0x2c8/0x388
  ksys_write+0x74/0x108
  __arm64_sys_write+0x24/0x38
  el0_svc_common.constprop.0+0x64/0x148
  do_el0_svc+0x24/0x38
  el0_svc+0x3c/0x130
  el0t_64_sync_handler+0xc8/0xd0
  el0t_64_sync+0x1ac/0x1b0
 ---[ end trace 0000000000000000 ]---

Since the sysfs buffer allocation and the hardware enablement is not
in the same critical region, it's possible to race with the perf

mode:
  [sysfs mode]                   [perf mode]
  tmc_etr_get_sysfs_buffer()
    spin_lock(&drvdata->spinlock)
    [sysfs buffer allocation]
    spin_unlock(&drvdata->spinlock)
                                 spin_lock(&drvdata->spinlock)
                                 tmc_etr_enable_hw()
                                   drvdata->etr_buf = etr_perf->etr_buf
                                 spin_unlock(&drvdata->spinlock)
 spin_lock(&drvdata->spinlock)
 tmc_etr_enable_hw()
   WARN_ON(drvdata->etr_buf) // WARN sicne etr_buf initialized at
                                the perf side
  spin_unlock(&drvdata->spinlock)

To resolve this, configure the tmc-etr mode before invoking
`enable_perf()` or sysfs interfaces. Prior to mode configuration,
explicitly check if the tmc-etr sink is already enabled in a
different mode to prevent race conditions between mode transitions.
Furthermore, enforce spinlock protection around the critical
sections to serialize concurrent accesses from sysfs and perf
subsystems.

Fixes: 296b01fd106e ("coresight: Refactor out buffer allocation function for ETR")
Reported-by: Yicong Yang <yangyicong@hisilicon.com>
Closes: https://lore.kernel.org/linux-arm-kernel/20241202092419.11777-2-yangyicong@huawei.com/
Signed-off-by: Junhao He <hejunhao3@huawei.com>
---
 .../hwtracing/coresight/coresight-tmc-etr.c   | 77 +++++++++++--------
 1 file changed, 47 insertions(+), 30 deletions(-)

diff --git a/drivers/hwtracing/coresight/coresight-tmc-etr.c b/drivers/hwtracing/coresight/coresight-tmc-etr.c
index a48bb85d0e7f..3d94d64cacaa 100644
--- a/drivers/hwtracing/coresight/coresight-tmc-etr.c
+++ b/drivers/hwtracing/coresight/coresight-tmc-etr.c
@@ -1190,11 +1190,6 @@ static struct etr_buf *tmc_etr_get_sysfs_buffer(struct coresight_device *csdev)
 		spin_lock_irqsave(&drvdata->spinlock, flags);
 	}
 
-	if (drvdata->reading || coresight_get_mode(csdev) == CS_MODE_PERF) {
-		ret = -EBUSY;
-		goto out;
-	}
-
 	/*
 	 * If we don't have a buffer or it doesn't match the requested size,
 	 * use the buffer allocated above. Otherwise reuse the existing buffer.
@@ -1205,7 +1200,6 @@ static struct etr_buf *tmc_etr_get_sysfs_buffer(struct coresight_device *csdev)
 		drvdata->sysfs_buf = new_buf;
 	}
 
-out:
 	spin_unlock_irqrestore(&drvdata->spinlock, flags);
 
 	/* Free memory outside the spinlock if need be */
@@ -1216,7 +1210,7 @@ static struct etr_buf *tmc_etr_get_sysfs_buffer(struct coresight_device *csdev)
 
 static int tmc_enable_etr_sink_sysfs(struct coresight_device *csdev)
 {
-	int ret = 0;
+	int ret;
 	unsigned long flags;
 	struct tmc_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
 	struct etr_buf *sysfs_buf = tmc_etr_get_sysfs_buffer(csdev);
@@ -1226,23 +1220,10 @@ static int tmc_enable_etr_sink_sysfs(struct coresight_device *csdev)
 
 	spin_lock_irqsave(&drvdata->spinlock, flags);
 
-	/*
-	 * In sysFS mode we can have multiple writers per sink.  Since this
-	 * sink is already enabled no memory is needed and the HW need not be
-	 * touched, even if the buffer size has changed.
-	 */
-	if (coresight_get_mode(csdev) == CS_MODE_SYSFS) {
-		csdev->refcnt++;
-		goto out;
-	}
-
 	ret = tmc_etr_enable_hw(drvdata, sysfs_buf);
-	if (!ret) {
-		coresight_set_mode(csdev, CS_MODE_SYSFS);
+	if (!ret)
 		csdev->refcnt++;
-	}
 
-out:
 	spin_unlock_irqrestore(&drvdata->spinlock, flags);
 
 	if (!ret)
@@ -1652,11 +1633,6 @@ static int tmc_enable_etr_sink_perf(struct coresight_device *csdev, void *data)
 	struct etr_perf_buffer *etr_perf = etm_perf_sink_config(handle);
 
 	spin_lock_irqsave(&drvdata->spinlock, flags);
-	 /* Don't use this sink if it is already claimed by sysFS */
-	if (coresight_get_mode(csdev) == CS_MODE_SYSFS) {
-		rc = -EBUSY;
-		goto unlock_out;
-	}
 
 	if (WARN_ON(!etr_perf || !etr_perf->etr_buf)) {
 		rc = -EINVAL;
@@ -1685,7 +1661,6 @@ static int tmc_enable_etr_sink_perf(struct coresight_device *csdev, void *data)
 	if (!rc) {
 		/* Associate with monitored process. */
 		drvdata->pid = pid;
-		coresight_set_mode(csdev, CS_MODE_PERF);
 		drvdata->perf_buf = etr_perf->etr_buf;
 		csdev->refcnt++;
 	}
@@ -1698,14 +1673,56 @@ static int tmc_enable_etr_sink_perf(struct coresight_device *csdev, void *data)
 static int tmc_enable_etr_sink(struct coresight_device *csdev,
 			       enum cs_mode mode, void *data)
 {
+	struct tmc_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
+	enum cs_mode old_mode;
+	int rc;
+
+	scoped_guard(spinlock_irqsave, &drvdata->spinlock) {
+		old_mode = coresight_get_mode(csdev);
+		if (old_mode != CS_MODE_DISABLED && old_mode != mode)
+			return -EBUSY;
+
+		if (drvdata->reading)
+			return -EBUSY;
+
+		/*
+		 * In sysFS mode we can have multiple writers per sink. Since this
+		 * sink is already enabled no memory is needed and the HW need not be
+		 * touched, even if the buffer size has changed.
+		 */
+		if (old_mode == CS_MODE_SYSFS) {
+			csdev->refcnt++;
+			return 0;
+		}
+
+		/*
+		 * minor note:
+		 * When sysfs-task1 get locked, it setup the mode first. Then
+		 * sysfs-task2 gets locked，it will directly return success even
+		 * when the tmc-etr is not enabled at this moment. Ultimately,
+		 * sysfs-task1 will still successfully enable tmc-etr.
+		 * This is a transient state and does not cause an anomaly.
+		 */
+		coresight_set_mode(csdev, mode);
+	}
+
 	switch (mode) {
 	case CS_MODE_SYSFS:
-		return tmc_enable_etr_sink_sysfs(csdev);
+		rc = tmc_enable_etr_sink_sysfs(csdev);
+		break;
 	case CS_MODE_PERF:
-		return tmc_enable_etr_sink_perf(csdev, data);
+		rc = tmc_enable_etr_sink_perf(csdev, data);
+		break;
 	default:
-		return -EINVAL;
+		rc = -EINVAL;
 	}
+
+	scoped_guard(spinlock_irqsave, &drvdata->spinlock) {
+		if (rc && old_mode != mode)
+			coresight_set_mode(csdev, old_mode);
+	}
+
+	return rc;
 }
 
 static int tmc_disable_etr_sink(struct coresight_device *csdev)
-- 
2.33.0

[PATCH 4/4] coresight: tmc-etr: Decouple the perf buffer allocation from sysfs mode

[Junhao He]

From: Yicong Yang <yangyicong@hisilicon.com>

Currently the perf buffer allocation follows the below logic:
- if the required AUX buffer size if larger, allocate the buffer with
  the required size
- otherwise allocate the size reference to the sysfs buffer size

This is not useful as we only collect to one AUX data, so just try to
allocate the buffer match the AUX buffer size.

Suggested-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Link: https://lore.kernel.org/linux-arm-kernel/df8967cd-2157-46a2-97d9-a1aea883cf63@arm.com/
Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
Signed-off-by: Junhao He <hejunhao3@huawei.com>
---
 .../hwtracing/coresight/coresight-tmc-etr.c   | 29 ++++++-------------
 1 file changed, 9 insertions(+), 20 deletions(-)

diff --git a/drivers/hwtracing/coresight/coresight-tmc-etr.c b/drivers/hwtracing/coresight/coresight-tmc-etr.c
index 3d94d64cacaa..26b56cb91bad 100644
--- a/drivers/hwtracing/coresight/coresight-tmc-etr.c
+++ b/drivers/hwtracing/coresight/coresight-tmc-etr.c
@@ -1254,9 +1254,7 @@ EXPORT_SYMBOL_GPL(tmc_etr_get_buffer);
 
 /*
  * alloc_etr_buf: Allocate ETR buffer for use by perf.
- * The size of the hardware buffer is dependent on the size configured
- * via sysfs and the perf ring buffer size. We prefer to allocate the
- * largest possible size, scaling down the size by half until it
+ * Allocate the largest possible size, scaling down the size by half until it
  * reaches a minimum limit (1M), beyond which we give up.
  */
 static struct etr_buf *
@@ -1268,33 +1266,24 @@ alloc_etr_buf(struct tmc_drvdata *drvdata, struct perf_event *event,
 	unsigned long size;
 
 	node = (event->cpu == -1) ? NUMA_NO_NODE : cpu_to_node(event->cpu);
-	/*
-	 * Try to match the perf ring buffer size if it is larger
-	 * than the size requested via sysfs.
-	 */
-	if ((nr_pages << PAGE_SHIFT) > drvdata->size) {
-		etr_buf = tmc_alloc_etr_buf(drvdata, ((ssize_t)nr_pages << PAGE_SHIFT),
-					    0, node, NULL);
-		if (!IS_ERR(etr_buf))
-			goto done;
-	}
+
+	/* Use the minimum limit if the required size is smaller */
+	size = (unsigned long)nr_pages << PAGE_SHIFT;
+	if (size < TMC_ETR_PERF_MIN_BUF_SIZE)
+		size = TMC_ETR_PERF_MIN_BUF_SIZE;
 
 	/*
-	 * Else switch to configured size for this ETR
-	 * and scale down until we hit the minimum limit.
+	 * Try to allocate the required size for this ETR, if failed scale
+	 * down until we hit the minimum limit.
 	 */
-	size = drvdata->size;
 	do {
 		etr_buf = tmc_alloc_etr_buf(drvdata, size, 0, node, NULL);
 		if (!IS_ERR(etr_buf))
-			goto done;
+			return etr_buf;
 		size /= 2;
 	} while (size >= TMC_ETR_PERF_MIN_BUF_SIZE);
 
 	return ERR_PTR(-ENOMEM);
-
-done:
-	return etr_buf;
 }
 
 static struct etr_buf *
-- 
2.33.0

Re: [PATCH 3/4] coresight: tmc: refactor the tmc-etr mode setting

[Jie Gan]

On 4/18/2025 1:58 PM, Junhao He wrote:
> When trying to run perf and sysfs mode simultaneously, the WARN_ON()
> in tmc_etr_enable_hw() is triggered sometimes:
> 
>   WARNING: CPU: 42 PID: 3911571 at drivers/hwtracing/coresight/coresight-tmc-etr.c:1060 tmc_etr_enable_hw+0xc0/0xd8 [coresight_tmc]
>   [..snip..]
>   Call trace:
>    tmc_etr_enable_hw+0xc0/0xd8 [coresight_tmc] (P)
>    tmc_enable_etr_sink+0x11c/0x250 [coresight_tmc] (L)
>    tmc_enable_etr_sink+0x11c/0x250 [coresight_tmc]
>    coresight_enable_path+0x1c8/0x218 [coresight]
>    coresight_enable_sysfs+0xa4/0x228 [coresight]
>    enable_source_store+0x58/0xa8 [coresight]
>    dev_attr_store+0x20/0x40
>    sysfs_kf_write+0x4c/0x68
>    kernfs_fop_write_iter+0x120/0x1b8
>    vfs_write+0x2c8/0x388
>    ksys_write+0x74/0x108
>    __arm64_sys_write+0x24/0x38
>    el0_svc_common.constprop.0+0x64/0x148
>    do_el0_svc+0x24/0x38
>    el0_svc+0x3c/0x130
>    el0t_64_sync_handler+0xc8/0xd0
>    el0t_64_sync+0x1ac/0x1b0
>   ---[ end trace 0000000000000000 ]---
> 
> Since the sysfs buffer allocation and the hardware enablement is not
> in the same critical region, it's possible to race with the perf
> 
> mode:
>    [sysfs mode]                   [perf mode]
>    tmc_etr_get_sysfs_buffer()
>      spin_lock(&drvdata->spinlock)
>      [sysfs buffer allocation]
>      spin_unlock(&drvdata->spinlock)
>                                   spin_lock(&drvdata->spinlock)
>                                   tmc_etr_enable_hw()
>                                     drvdata->etr_buf = etr_perf->etr_buf
>                                   spin_unlock(&drvdata->spinlock)
>   spin_lock(&drvdata->spinlock)
>   tmc_etr_enable_hw()
>     WARN_ON(drvdata->etr_buf) // WARN sicne etr_buf initialized at
>                                  the perf side
>    spin_unlock(&drvdata->spinlock)
> 
> To resolve this, configure the tmc-etr mode before invoking
> `enable_perf()` or sysfs interfaces. Prior to mode configuration,
> explicitly check if the tmc-etr sink is already enabled in a
> different mode to prevent race conditions between mode transitions.
> Furthermore, enforce spinlock protection around the critical
> sections to serialize concurrent accesses from sysfs and perf
> subsystems.

Is any issue observed during this race condition?

The etr_buf is checked before assign it to sysfs_buf or perf_buf.
In my understanding, the warning raised to indicate the etr already 
enabled with sysfs mode or perf mode, so terminate current enable 
session, right?

Thanks,
Jie

> 
> Fixes: 296b01fd106e ("coresight: Refactor out buffer allocation function for ETR")
> Reported-by: Yicong Yang <yangyicong@hisilicon.com>
> Closes: https://lore.kernel.org/linux-arm-kernel/20241202092419.11777-2-yangyicong@huawei.com/
> Signed-off-by: Junhao He <hejunhao3@huawei.com>
> ---
>   .../hwtracing/coresight/coresight-tmc-etr.c   | 77 +++++++++++--------
>   1 file changed, 47 insertions(+), 30 deletions(-)
> 
> diff --git a/drivers/hwtracing/coresight/coresight-tmc-etr.c b/drivers/hwtracing/coresight/coresight-tmc-etr.c
> index a48bb85d0e7f..3d94d64cacaa 100644
> --- a/drivers/hwtracing/coresight/coresight-tmc-etr.c
> +++ b/drivers/hwtracing/coresight/coresight-tmc-etr.c
> @@ -1190,11 +1190,6 @@ static struct etr_buf *tmc_etr_get_sysfs_buffer(struct coresight_device *csdev)
>   		spin_lock_irqsave(&drvdata->spinlock, flags);
>   	}
>   
> -	if (drvdata->reading || coresight_get_mode(csdev) == CS_MODE_PERF) {
> -		ret = -EBUSY;
> -		goto out;
> -	}
> -
>   	/*
>   	 * If we don't have a buffer or it doesn't match the requested size,
>   	 * use the buffer allocated above. Otherwise reuse the existing buffer.
> @@ -1205,7 +1200,6 @@ static struct etr_buf *tmc_etr_get_sysfs_buffer(struct coresight_device *csdev)
>   		drvdata->sysfs_buf = new_buf;
>   	}
>   
> -out:
>   	spin_unlock_irqrestore(&drvdata->spinlock, flags);
>   
>   	/* Free memory outside the spinlock if need be */
> @@ -1216,7 +1210,7 @@ static struct etr_buf *tmc_etr_get_sysfs_buffer(struct coresight_device *csdev)
>   
>   static int tmc_enable_etr_sink_sysfs(struct coresight_device *csdev)
>   {
> -	int ret = 0;
> +	int ret;
>   	unsigned long flags;
>   	struct tmc_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
>   	struct etr_buf *sysfs_buf = tmc_etr_get_sysfs_buffer(csdev);
> @@ -1226,23 +1220,10 @@ static int tmc_enable_etr_sink_sysfs(struct coresight_device *csdev)
>   
>   	spin_lock_irqsave(&drvdata->spinlock, flags);
>   
> -	/*
> -	 * In sysFS mode we can have multiple writers per sink.  Since this
> -	 * sink is already enabled no memory is needed and the HW need not be
> -	 * touched, even if the buffer size has changed.
> -	 */
> -	if (coresight_get_mode(csdev) == CS_MODE_SYSFS) {
> -		csdev->refcnt++;
> -		goto out;
> -	}
> -
>   	ret = tmc_etr_enable_hw(drvdata, sysfs_buf);
> -	if (!ret) {
> -		coresight_set_mode(csdev, CS_MODE_SYSFS);
> +	if (!ret)
>   		csdev->refcnt++;
> -	}
>   
> -out:
>   	spin_unlock_irqrestore(&drvdata->spinlock, flags);
>   
>   	if (!ret)
> @@ -1652,11 +1633,6 @@ static int tmc_enable_etr_sink_perf(struct coresight_device *csdev, void *data)
>   	struct etr_perf_buffer *etr_perf = etm_perf_sink_config(handle);
>   
>   	spin_lock_irqsave(&drvdata->spinlock, flags);
> -	 /* Don't use this sink if it is already claimed by sysFS */
> -	if (coresight_get_mode(csdev) == CS_MODE_SYSFS) {
> -		rc = -EBUSY;
> -		goto unlock_out;
> -	}
>   
>   	if (WARN_ON(!etr_perf || !etr_perf->etr_buf)) {
>   		rc = -EINVAL;
> @@ -1685,7 +1661,6 @@ static int tmc_enable_etr_sink_perf(struct coresight_device *csdev, void *data)
>   	if (!rc) {
>   		/* Associate with monitored process. */
>   		drvdata->pid = pid;
> -		coresight_set_mode(csdev, CS_MODE_PERF);
>   		drvdata->perf_buf = etr_perf->etr_buf;
>   		csdev->refcnt++;
>   	}
> @@ -1698,14 +1673,56 @@ static int tmc_enable_etr_sink_perf(struct coresight_device *csdev, void *data)
>   static int tmc_enable_etr_sink(struct coresight_device *csdev,
>   			       enum cs_mode mode, void *data)
>   {
> +	struct tmc_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
> +	enum cs_mode old_mode;
> +	int rc;
> +
> +	scoped_guard(spinlock_irqsave, &drvdata->spinlock) {
> +		old_mode = coresight_get_mode(csdev);
> +		if (old_mode != CS_MODE_DISABLED && old_mode != mode)
> +			return -EBUSY;
> +
> +		if (drvdata->reading)
> +			return -EBUSY;
> +
> +		/*
> +		 * In sysFS mode we can have multiple writers per sink. Since this
> +		 * sink is already enabled no memory is needed and the HW need not be
> +		 * touched, even if the buffer size has changed.
> +		 */
> +		if (old_mode == CS_MODE_SYSFS) {
> +			csdev->refcnt++;
> +			return 0;
> +		}
> +
> +		/*
> +		 * minor note:
> +		 * When sysfs-task1 get locked, it setup the mode first. Then
> +		 * sysfs-task2 gets locked，it will directly return success even
> +		 * when the tmc-etr is not enabled at this moment. Ultimately,
> +		 * sysfs-task1 will still successfully enable tmc-etr.
> +		 * This is a transient state and does not cause an anomaly.
> +		 */
> +		coresight_set_mode(csdev, mode);
> +	}
> +
>   	switch (mode) {
>   	case CS_MODE_SYSFS:
> -		return tmc_enable_etr_sink_sysfs(csdev);
> +		rc = tmc_enable_etr_sink_sysfs(csdev);
> +		break;
>   	case CS_MODE_PERF:
> -		return tmc_enable_etr_sink_perf(csdev, data);
> +		rc = tmc_enable_etr_sink_perf(csdev, data);
> +		break;
>   	default:
> -		return -EINVAL;
> +		rc = -EINVAL;
>   	}
> +
> +	scoped_guard(spinlock_irqsave, &drvdata->spinlock) {
> +		if (rc && old_mode != mode)
> +			coresight_set_mode(csdev, old_mode);
> +	}
> +
> +	return rc;
>   }
>   
>   static int tmc_disable_etr_sink(struct coresight_device *csdev)

Re: [PATCH 3/4] coresight: tmc: refactor the tmc-etr mode setting

[hejunhao]

On 2025/4/18 15:12, Jie Gan wrote:
>
>
> On 4/18/2025 1:58 PM, Junhao He wrote:
>> When trying to run perf and sysfs mode simultaneously, the WARN_ON()
>> in tmc_etr_enable_hw() is triggered sometimes:
>>
>>   WARNING: CPU: 42 PID: 3911571 at 
>> drivers/hwtracing/coresight/coresight-tmc-etr.c:1060 
>> tmc_etr_enable_hw+0xc0/0xd8 [coresight_tmc]
>>   [..snip..]
>>   Call trace:
>>    tmc_etr_enable_hw+0xc0/0xd8 [coresight_tmc] (P)
>>    tmc_enable_etr_sink+0x11c/0x250 [coresight_tmc] (L)
>>    tmc_enable_etr_sink+0x11c/0x250 [coresight_tmc]
>>    coresight_enable_path+0x1c8/0x218 [coresight]
>>    coresight_enable_sysfs+0xa4/0x228 [coresight]
>>    enable_source_store+0x58/0xa8 [coresight]
>>    dev_attr_store+0x20/0x40
>>    sysfs_kf_write+0x4c/0x68
>>    kernfs_fop_write_iter+0x120/0x1b8
>>    vfs_write+0x2c8/0x388
>>    ksys_write+0x74/0x108
>>    __arm64_sys_write+0x24/0x38
>>    el0_svc_common.constprop.0+0x64/0x148
>>    do_el0_svc+0x24/0x38
>>    el0_svc+0x3c/0x130
>>    el0t_64_sync_handler+0xc8/0xd0
>>    el0t_64_sync+0x1ac/0x1b0
>>   ---[ end trace 0000000000000000 ]---
>>
>> Since the sysfs buffer allocation and the hardware enablement is not
>> in the same critical region, it's possible to race with the perf
>>
>> mode:
>>    [sysfs mode]                   [perf mode]
>>    tmc_etr_get_sysfs_buffer()
>>      spin_lock(&drvdata->spinlock)
>>      [sysfs buffer allocation]
>>      spin_unlock(&drvdata->spinlock)
>> spin_lock(&drvdata->spinlock)
>>                                   tmc_etr_enable_hw()
>>                                     drvdata->etr_buf = etr_perf->etr_buf
>> spin_unlock(&drvdata->spinlock)
>>   spin_lock(&drvdata->spinlock)
>>   tmc_etr_enable_hw()
>>     WARN_ON(drvdata->etr_buf) // WARN sicne etr_buf initialized at
>>                                  the perf side
>>    spin_unlock(&drvdata->spinlock)
>>
>> To resolve this, configure the tmc-etr mode before invoking
>> `enable_perf()` or sysfs interfaces. Prior to mode configuration,
>> explicitly check if the tmc-etr sink is already enabled in a
>> different mode to prevent race conditions between mode transitions.
>> Furthermore, enforce spinlock protection around the critical
>> sections to serialize concurrent accesses from sysfs and perf
>> subsystems.
>
> Is any issue observed during this race condition?
>
> The etr_buf is checked before assign it to sysfs_buf or perf_buf.
> In my understanding, the warning raised to indicate the etr already 
> enabled with sysfs mode or perf mode, so terminate current enable 
> session, right?
>
> Thanks,
> Jie
>

The sysfs task is executing the process to enable tmc-etr and has allocated
a sysfs_buf. However, the sysfs spinlock does not ensure atomicity of the
entire enable_etr_sink_sysfs() function execution. In this scenario, the 
perf
session may preemptively call tmc_etr_enable_hw() to enable tmc-etr.

Consequently, during race conditions, perf always prioritizes execution, and
the sysfs_buf remains un-released, leading to memory leaks. Then this 
triggers
a WARN_ON(drvdata->etr_buf) warning.

This patch also addresses the issue described in [1], simplifying the 
setup and
checks for "mode".

[1] Closes: 
https://lore.kernel.org/linux-arm-kernel/20241202092419.11777-2-yangyicong@huawei.com/

Best regards,
Junhao.

>>
>> Fixes: 296b01fd106e ("coresight: Refactor out buffer allocation 
>> function for ETR")
>> Reported-by: Yicong Yang <yangyicong@hisilicon.com>
>> Closes: 
>> https://lore.kernel.org/linux-arm-kernel/20241202092419.11777-2-yangyicong@huawei.com/
>> Signed-off-by: Junhao He <hejunhao3@huawei.com>
>> ---
>>   .../hwtracing/coresight/coresight-tmc-etr.c   | 77 +++++++++++--------
>>   1 file changed, 47 insertions(+), 30 deletions(-)
>>
>> diff --git a/drivers/hwtracing/coresight/coresight-tmc-etr.c 
>> b/drivers/hwtracing/coresight/coresight-tmc-etr.c
>> index a48bb85d0e7f..3d94d64cacaa 100644
>> --- a/drivers/hwtracing/coresight/coresight-tmc-etr.c
>> +++ b/drivers/hwtracing/coresight/coresight-tmc-etr.c
>> @@ -1190,11 +1190,6 @@ static struct etr_buf 
>> *tmc_etr_get_sysfs_buffer(struct coresight_device *csdev)
>>           spin_lock_irqsave(&drvdata->spinlock, flags);
>>       }
>>   -    if (drvdata->reading || coresight_get_mode(csdev) == 
>> CS_MODE_PERF) {
>> -        ret = -EBUSY;
>> -        goto out;
>> -    }
>> -
>>       /*
>>        * If we don't have a buffer or it doesn't match the requested 
>> size,
>>        * use the buffer allocated above. Otherwise reuse the existing 
>> buffer.
>> @@ -1205,7 +1200,6 @@ static struct etr_buf 
>> *tmc_etr_get_sysfs_buffer(struct coresight_device *csdev)
>>           drvdata->sysfs_buf = new_buf;
>>       }
>>   -out:
>>       spin_unlock_irqrestore(&drvdata->spinlock, flags);
>>         /* Free memory outside the spinlock if need be */
>> @@ -1216,7 +1210,7 @@ static struct etr_buf 
>> *tmc_etr_get_sysfs_buffer(struct coresight_device *csdev)
>>     static int tmc_enable_etr_sink_sysfs(struct coresight_device *csdev)
>>   {
>> -    int ret = 0;
>> +    int ret;
>>       unsigned long flags;
>>       struct tmc_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
>>       struct etr_buf *sysfs_buf = tmc_etr_get_sysfs_buffer(csdev);
>> @@ -1226,23 +1220,10 @@ static int tmc_enable_etr_sink_sysfs(struct 
>> coresight_device *csdev)
>>         spin_lock_irqsave(&drvdata->spinlock, flags);
>>   -    /*
>> -     * In sysFS mode we can have multiple writers per sink. Since this
>> -     * sink is already enabled no memory is needed and the HW need 
>> not be
>> -     * touched, even if the buffer size has changed.
>> -     */
>> -    if (coresight_get_mode(csdev) == CS_MODE_SYSFS) {
>> -        csdev->refcnt++;
>> -        goto out;
>> -    }
>> -
>>       ret = tmc_etr_enable_hw(drvdata, sysfs_buf);
>> -    if (!ret) {
>> -        coresight_set_mode(csdev, CS_MODE_SYSFS);
>> +    if (!ret)
>>           csdev->refcnt++;
>> -    }
>>   -out:
>>       spin_unlock_irqrestore(&drvdata->spinlock, flags);
>>         if (!ret)
>> @@ -1652,11 +1633,6 @@ static int tmc_enable_etr_sink_perf(struct 
>> coresight_device *csdev, void *data)
>>       struct etr_perf_buffer *etr_perf = etm_perf_sink_config(handle);
>>         spin_lock_irqsave(&drvdata->spinlock, flags);
>> -     /* Don't use this sink if it is already claimed by sysFS */
>> -    if (coresight_get_mode(csdev) == CS_MODE_SYSFS) {
>> -        rc = -EBUSY;
>> -        goto unlock_out;
>> -    }
>>         if (WARN_ON(!etr_perf || !etr_perf->etr_buf)) {
>>           rc = -EINVAL;
>> @@ -1685,7 +1661,6 @@ static int tmc_enable_etr_sink_perf(struct 
>> coresight_device *csdev, void *data)
>>       if (!rc) {
>>           /* Associate with monitored process. */
>>           drvdata->pid = pid;
>> -        coresight_set_mode(csdev, CS_MODE_PERF);
>>           drvdata->perf_buf = etr_perf->etr_buf;
>>           csdev->refcnt++;
>>       }
>> @@ -1698,14 +1673,56 @@ static int tmc_enable_etr_sink_perf(struct 
>> coresight_device *csdev, void *data)
>>   static int tmc_enable_etr_sink(struct coresight_device *csdev,
>>                      enum cs_mode mode, void *data)
>>   {
>> +    struct tmc_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
>> +    enum cs_mode old_mode;
>> +    int rc;
>> +
>> +    scoped_guard(spinlock_irqsave, &drvdata->spinlock) {
>> +        old_mode = coresight_get_mode(csdev);
>> +        if (old_mode != CS_MODE_DISABLED && old_mode != mode)
>> +            return -EBUSY;
>> +
>> +        if (drvdata->reading)
>> +            return -EBUSY;
>> +
>> +        /*
>> +         * In sysFS mode we can have multiple writers per sink. 
>> Since this
>> +         * sink is already enabled no memory is needed and the HW 
>> need not be
>> +         * touched, even if the buffer size has changed.
>> +         */
>> +        if (old_mode == CS_MODE_SYSFS) {
>> +            csdev->refcnt++;
>> +            return 0;
>> +        }
>> +
>> +        /*
>> +         * minor note:
>> +         * When sysfs-task1 get locked, it setup the mode first. Then
>> +         * sysfs-task2 gets locked，it will directly return success 
>> even
>> +         * when the tmc-etr is not enabled at this moment. Ultimately,
>> +         * sysfs-task1 will still successfully enable tmc-etr.
>> +         * This is a transient state and does not cause an anomaly.
>> +         */
>> +        coresight_set_mode(csdev, mode);
>> +    }
>> +
>>       switch (mode) {
>>       case CS_MODE_SYSFS:
>> -        return tmc_enable_etr_sink_sysfs(csdev);
>> +        rc = tmc_enable_etr_sink_sysfs(csdev);
>> +        break;
>>       case CS_MODE_PERF:
>> -        return tmc_enable_etr_sink_perf(csdev, data);
>> +        rc = tmc_enable_etr_sink_perf(csdev, data);
>> +        break;
>>       default:
>> -        return -EINVAL;
>> +        rc = -EINVAL;
>>       }
>> +
>> +    scoped_guard(spinlock_irqsave, &drvdata->spinlock) {
>> +        if (rc && old_mode != mode)
>> +            coresight_set_mode(csdev, old_mode);
>> +    }
>> +
>> +    return rc;
>>   }
>>     static int tmc_disable_etr_sink(struct coresight_device *csdev)
>
> .
>

Re: [PATCH 3/4] coresight: tmc: refactor the tmc-etr mode setting

[Jie Gan]

On 4/22/2025 8:52 PM, hejunhao wrote:
> 
> On 2025/4/18 15:12, Jie Gan wrote:
>>
>>
>> On 4/18/2025 1:58 PM, Junhao He wrote:
>>> When trying to run perf and sysfs mode simultaneously, the WARN_ON()
>>> in tmc_etr_enable_hw() is triggered sometimes:
>>>
>>>   WARNING: CPU: 42 PID: 3911571 at drivers/hwtracing/coresight/ 
>>> coresight-tmc-etr.c:1060 tmc_etr_enable_hw+0xc0/0xd8 [coresight_tmc]
>>>   [..snip..]
>>>   Call trace:
>>>    tmc_etr_enable_hw+0xc0/0xd8 [coresight_tmc] (P)
>>>    tmc_enable_etr_sink+0x11c/0x250 [coresight_tmc] (L)
>>>    tmc_enable_etr_sink+0x11c/0x250 [coresight_tmc]
>>>    coresight_enable_path+0x1c8/0x218 [coresight]
>>>    coresight_enable_sysfs+0xa4/0x228 [coresight]
>>>    enable_source_store+0x58/0xa8 [coresight]
>>>    dev_attr_store+0x20/0x40
>>>    sysfs_kf_write+0x4c/0x68
>>>    kernfs_fop_write_iter+0x120/0x1b8
>>>    vfs_write+0x2c8/0x388
>>>    ksys_write+0x74/0x108
>>>    __arm64_sys_write+0x24/0x38
>>>    el0_svc_common.constprop.0+0x64/0x148
>>>    do_el0_svc+0x24/0x38
>>>    el0_svc+0x3c/0x130
>>>    el0t_64_sync_handler+0xc8/0xd0
>>>    el0t_64_sync+0x1ac/0x1b0
>>>   ---[ end trace 0000000000000000 ]---
>>>
>>> Since the sysfs buffer allocation and the hardware enablement is not
>>> in the same critical region, it's possible to race with the perf
>>>
>>> mode:
>>>    [sysfs mode]                   [perf mode]
>>>    tmc_etr_get_sysfs_buffer()
>>>      spin_lock(&drvdata->spinlock)
>>>      [sysfs buffer allocation]
>>>      spin_unlock(&drvdata->spinlock)
>>> spin_lock(&drvdata->spinlock)
>>>                                   tmc_etr_enable_hw()
>>>                                     drvdata->etr_buf = etr_perf->etr_buf
>>> spin_unlock(&drvdata->spinlock)
>>>   spin_lock(&drvdata->spinlock)
>>>   tmc_etr_enable_hw()
>>>     WARN_ON(drvdata->etr_buf) // WARN sicne etr_buf initialized at
>>>                                  the perf side
>>>    spin_unlock(&drvdata->spinlock)
>>>
>>> To resolve this, configure the tmc-etr mode before invoking
>>> `enable_perf()` or sysfs interfaces. Prior to mode configuration,
>>> explicitly check if the tmc-etr sink is already enabled in a
>>> different mode to prevent race conditions between mode transitions.
>>> Furthermore, enforce spinlock protection around the critical
>>> sections to serialize concurrent accesses from sysfs and perf
>>> subsystems.
>>
>> Is any issue observed during this race condition?
>>
>> The etr_buf is checked before assign it to sysfs_buf or perf_buf.
>> In my understanding, the warning raised to indicate the etr already 
>> enabled with sysfs mode or perf mode, so terminate current enable 
>> session, right?
>>
>> Thanks,
>> Jie
>>
> 
> The sysfs task is executing the process to enable tmc-etr and has allocated
> a sysfs_buf. However, the sysfs spinlock does not ensure atomicity of the
> entire enable_etr_sink_sysfs() function execution. In this scenario, the 
> perf
> session may preemptively call tmc_etr_enable_hw() to enable tmc-etr.
> 
> Consequently, during race conditions, perf always prioritizes execution, 
> and
> the sysfs_buf remains un-released, leading to memory leaks. Then this 
> triggers
> a WARN_ON(drvdata->etr_buf) warning.

The pre-allocated sysfs_buf is stored in drvdata->sysfs_buf. And I think 
it's fine to remain unreleased until the ETR is enabled with sysfs mode.
The ETR will not setup a new sysfs_buf if drvdata->sysfs_buf already 
exists. But in other words, it definitely consumes some 
memory(especailly allocated a large size of sysfs_buf) for a while or 
forever if we dont enable the ETR with sysfs mode in the future?

I think this is the issue try to address?

Thanks,
Jie

> 
> This patch also addresses the issue described in [1], simplifying the 
> setup and
> checks for "mode".
> 
> [1] Closes: https://lore.kernel.org/linux-arm- 
> kernel/20241202092419.11777-2-yangyicong@huawei.com/
> 
> Best regards,
> Junhao.
> 
>>>
>>> Fixes: 296b01fd106e ("coresight: Refactor out buffer allocation 
>>> function for ETR")
>>> Reported-by: Yicong Yang <yangyicong@hisilicon.com>
>>> Closes: https://lore.kernel.org/linux-arm- 
>>> kernel/20241202092419.11777-2-yangyicong@huawei.com/
>>> Signed-off-by: Junhao He <hejunhao3@huawei.com>
>>> ---
>>>   .../hwtracing/coresight/coresight-tmc-etr.c   | 77 +++++++++++--------
>>>   1 file changed, 47 insertions(+), 30 deletions(-)
>>>
>>> diff --git a/drivers/hwtracing/coresight/coresight-tmc-etr.c b/ 
>>> drivers/hwtracing/coresight/coresight-tmc-etr.c
>>> index a48bb85d0e7f..3d94d64cacaa 100644
>>> --- a/drivers/hwtracing/coresight/coresight-tmc-etr.c
>>> +++ b/drivers/hwtracing/coresight/coresight-tmc-etr.c
>>> @@ -1190,11 +1190,6 @@ static struct etr_buf 
>>> *tmc_etr_get_sysfs_buffer(struct coresight_device *csdev)
>>>           spin_lock_irqsave(&drvdata->spinlock, flags);
>>>       }
>>>   -    if (drvdata->reading || coresight_get_mode(csdev) == 
>>> CS_MODE_PERF) {
>>> -        ret = -EBUSY;
>>> -        goto out;
>>> -    }
>>> -
>>>       /*
>>>        * If we don't have a buffer or it doesn't match the requested 
>>> size,
>>>        * use the buffer allocated above. Otherwise reuse the existing 
>>> buffer.
>>> @@ -1205,7 +1200,6 @@ static struct etr_buf 
>>> *tmc_etr_get_sysfs_buffer(struct coresight_device *csdev)
>>>           drvdata->sysfs_buf = new_buf;
>>>       }
>>>   -out:
>>>       spin_unlock_irqrestore(&drvdata->spinlock, flags);
>>>         /* Free memory outside the spinlock if need be */
>>> @@ -1216,7 +1210,7 @@ static struct etr_buf 
>>> *tmc_etr_get_sysfs_buffer(struct coresight_device *csdev)
>>>     static int tmc_enable_etr_sink_sysfs(struct coresight_device *csdev)
>>>   {
>>> -    int ret = 0;
>>> +    int ret;
>>>       unsigned long flags;
>>>       struct tmc_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
>>>       struct etr_buf *sysfs_buf = tmc_etr_get_sysfs_buffer(csdev);
>>> @@ -1226,23 +1220,10 @@ static int tmc_enable_etr_sink_sysfs(struct 
>>> coresight_device *csdev)
>>>         spin_lock_irqsave(&drvdata->spinlock, flags);
>>>   -    /*
>>> -     * In sysFS mode we can have multiple writers per sink. Since this
>>> -     * sink is already enabled no memory is needed and the HW need 
>>> not be
>>> -     * touched, even if the buffer size has changed.
>>> -     */
>>> -    if (coresight_get_mode(csdev) == CS_MODE_SYSFS) {
>>> -        csdev->refcnt++;
>>> -        goto out;
>>> -    }
>>> -
>>>       ret = tmc_etr_enable_hw(drvdata, sysfs_buf);
>>> -    if (!ret) {
>>> -        coresight_set_mode(csdev, CS_MODE_SYSFS);
>>> +    if (!ret)
>>>           csdev->refcnt++;
>>> -    }
>>>   -out:
>>>       spin_unlock_irqrestore(&drvdata->spinlock, flags);
>>>         if (!ret)
>>> @@ -1652,11 +1633,6 @@ static int tmc_enable_etr_sink_perf(struct 
>>> coresight_device *csdev, void *data)
>>>       struct etr_perf_buffer *etr_perf = etm_perf_sink_config(handle);
>>>         spin_lock_irqsave(&drvdata->spinlock, flags);
>>> -     /* Don't use this sink if it is already claimed by sysFS */
>>> -    if (coresight_get_mode(csdev) == CS_MODE_SYSFS) {
>>> -        rc = -EBUSY;
>>> -        goto unlock_out;
>>> -    }
>>>         if (WARN_ON(!etr_perf || !etr_perf->etr_buf)) {
>>>           rc = -EINVAL;
>>> @@ -1685,7 +1661,6 @@ static int tmc_enable_etr_sink_perf(struct 
>>> coresight_device *csdev, void *data)
>>>       if (!rc) {
>>>           /* Associate with monitored process. */
>>>           drvdata->pid = pid;
>>> -        coresight_set_mode(csdev, CS_MODE_PERF);
>>>           drvdata->perf_buf = etr_perf->etr_buf;
>>>           csdev->refcnt++;
>>>       }
>>> @@ -1698,14 +1673,56 @@ static int tmc_enable_etr_sink_perf(struct 
>>> coresight_device *csdev, void *data)
>>>   static int tmc_enable_etr_sink(struct coresight_device *csdev,
>>>                      enum cs_mode mode, void *data)
>>>   {
>>> +    struct tmc_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
>>> +    enum cs_mode old_mode;
>>> +    int rc;
>>> +
>>> +    scoped_guard(spinlock_irqsave, &drvdata->spinlock) {
>>> +        old_mode = coresight_get_mode(csdev);
>>> +        if (old_mode != CS_MODE_DISABLED && old_mode != mode)
>>> +            return -EBUSY;
>>> +
>>> +        if (drvdata->reading)
>>> +            return -EBUSY;
>>> +
>>> +        /*
>>> +         * In sysFS mode we can have multiple writers per sink. 
>>> Since this
>>> +         * sink is already enabled no memory is needed and the HW 
>>> need not be
>>> +         * touched, even if the buffer size has changed.
>>> +         */
>>> +        if (old_mode == CS_MODE_SYSFS) {
>>> +            csdev->refcnt++;
>>> +            return 0;
>>> +        }
>>> +
>>> +        /*
>>> +         * minor note:
>>> +         * When sysfs-task1 get locked, it setup the mode first. Then
>>> +         * sysfs-task2 gets locked，it will directly return success 
>>> even
>>> +         * when the tmc-etr is not enabled at this moment. Ultimately,
>>> +         * sysfs-task1 will still successfully enable tmc-etr.
>>> +         * This is a transient state and does not cause an anomaly.
>>> +         */
>>> +        coresight_set_mode(csdev, mode);
>>> +    }
>>> +
>>>       switch (mode) {
>>>       case CS_MODE_SYSFS:
>>> -        return tmc_enable_etr_sink_sysfs(csdev);
>>> +        rc = tmc_enable_etr_sink_sysfs(csdev);
>>> +        break;
>>>       case CS_MODE_PERF:
>>> -        return tmc_enable_etr_sink_perf(csdev, data);
>>> +        rc = tmc_enable_etr_sink_perf(csdev, data);
>>> +        break;
>>>       default:
>>> -        return -EINVAL;
>>> +        rc = -EINVAL;
>>>       }
>>> +
>>> +    scoped_guard(spinlock_irqsave, &drvdata->spinlock) {
>>> +        if (rc && old_mode != mode)
>>> +            coresight_set_mode(csdev, old_mode);
>>> +    }
>>> +
>>> +    return rc;
>>>   }
>>>     static int tmc_disable_etr_sink(struct coresight_device *csdev)
>>
>> .
>>
> 

Re: [PATCH 3/4] coresight: tmc: refactor the tmc-etr mode setting

[hejunhao]

On 2025/4/23 10:07, Jie Gan wrote:
>
>
> On 4/22/2025 8:52 PM, hejunhao wrote:
>>
>> On 2025/4/18 15:12, Jie Gan wrote:
>>>
>>>
>>> On 4/18/2025 1:58 PM, Junhao He wrote:
>>>> When trying to run perf and sysfs mode simultaneously, the WARN_ON()
>>>> in tmc_etr_enable_hw() is triggered sometimes:
>>>>
>>>>   WARNING: CPU: 42 PID: 3911571 at drivers/hwtracing/coresight/ 
>>>> coresight-tmc-etr.c:1060 tmc_etr_enable_hw+0xc0/0xd8 [coresight_tmc]
>>>>   [..snip..]
>>>>   Call trace:
>>>>    tmc_etr_enable_hw+0xc0/0xd8 [coresight_tmc] (P)
>>>>    tmc_enable_etr_sink+0x11c/0x250 [coresight_tmc] (L)
>>>>    tmc_enable_etr_sink+0x11c/0x250 [coresight_tmc]
>>>>    coresight_enable_path+0x1c8/0x218 [coresight]
>>>>    coresight_enable_sysfs+0xa4/0x228 [coresight]
>>>>    enable_source_store+0x58/0xa8 [coresight]
>>>>    dev_attr_store+0x20/0x40
>>>>    sysfs_kf_write+0x4c/0x68
>>>>    kernfs_fop_write_iter+0x120/0x1b8
>>>>    vfs_write+0x2c8/0x388
>>>>    ksys_write+0x74/0x108
>>>>    __arm64_sys_write+0x24/0x38
>>>>    el0_svc_common.constprop.0+0x64/0x148
>>>>    do_el0_svc+0x24/0x38
>>>>    el0_svc+0x3c/0x130
>>>>    el0t_64_sync_handler+0xc8/0xd0
>>>>    el0t_64_sync+0x1ac/0x1b0
>>>>   ---[ end trace 0000000000000000 ]---
>>>>
>>>> Since the sysfs buffer allocation and the hardware enablement is not
>>>> in the same critical region, it's possible to race with the perf
>>>>
>>>> mode:
>>>>    [sysfs mode]                   [perf mode]
>>>>    tmc_etr_get_sysfs_buffer()
>>>>      spin_lock(&drvdata->spinlock)
>>>>      [sysfs buffer allocation]
>>>>      spin_unlock(&drvdata->spinlock)
>>>> spin_lock(&drvdata->spinlock)
>>>>                                   tmc_etr_enable_hw()
>>>>                                     drvdata->etr_buf = 
>>>> etr_perf->etr_buf
>>>> spin_unlock(&drvdata->spinlock)
>>>>   spin_lock(&drvdata->spinlock)
>>>>   tmc_etr_enable_hw()
>>>>     WARN_ON(drvdata->etr_buf) // WARN sicne etr_buf initialized at
>>>>                                  the perf side
>>>>    spin_unlock(&drvdata->spinlock)
>>>>
>>>> To resolve this, configure the tmc-etr mode before invoking
>>>> `enable_perf()` or sysfs interfaces. Prior to mode configuration,
>>>> explicitly check if the tmc-etr sink is already enabled in a
>>>> different mode to prevent race conditions between mode transitions.
>>>> Furthermore, enforce spinlock protection around the critical
>>>> sections to serialize concurrent accesses from sysfs and perf
>>>> subsystems.
>>>
>>> Is any issue observed during this race condition?
>>>
>>> The etr_buf is checked before assign it to sysfs_buf or perf_buf.
>>> In my understanding, the warning raised to indicate the etr already 
>>> enabled with sysfs mode or perf mode, so terminate current enable 
>>> session, right?
>>>
>>> Thanks,
>>> Jie
>>>
>>
>> The sysfs task is executing the process to enable tmc-etr and has 
>> allocated
>> a sysfs_buf. However, the sysfs spinlock does not ensure atomicity of 
>> the
>> entire enable_etr_sink_sysfs() function execution. In this scenario, 
>> the perf
>> session may preemptively call tmc_etr_enable_hw() to enable tmc-etr.
>>
>> Consequently, during race conditions, perf always prioritizes 
>> execution, and
>> the sysfs_buf remains un-released, leading to memory leaks. Then this 
>> triggers
>> a WARN_ON(drvdata->etr_buf) warning.
>
> The pre-allocated sysfs_buf is stored in drvdata->sysfs_buf. And I 
> think it's fine to remain unreleased until the ETR is enabled with 
> sysfs mode.

Yes, you're right. The tmc-etr is retaining the sysfs_buf, so no memory 
leak occurs.

> The ETR will not setup a new sysfs_buf if drvdata->sysfs_buf already 
> exists. But in other words, it definitely consumes some 
> memory(especailly allocated a large size of sysfs_buf) for a while or 
> forever if we dont enable the ETR with sysfs mode in the future?
>
> I think this is the issue try to address?
>

Patch 3 of patchset [1] is visible, serving as the basis, and you can 
understand what the patch aims to achieve.

This patch aims to avoid race conditions, as warnings can lead to 
concerns about potential hidden bugs, such
as getting out of sync. I may need to update the commit to include the 
commit log for patch 1's fix in patchset [1].

[1]: 
https://lore.kernel.org/linux-arm-kernel/20241202092419.11777-4-yangyicong@huawei.com/

Best regards,
Junhao.

> Thanks,
> Jie
>
>>
>> This patch also addresses the issue described in [1], simplifying the 
>> setup and
>> checks for "mode".
>>
>> [1] Closes: https://lore.kernel.org/linux-arm- 
>> kernel/20241202092419.11777-2-yangyicong@huawei.com/
>>
>> Best regards,
>> Junhao.
>>
>>>>
>>>> Fixes: 296b01fd106e ("coresight: Refactor out buffer allocation 
>>>> function for ETR")
>>>> Reported-by: Yicong Yang <yangyicong@hisilicon.com>
>>>> Closes: https://lore.kernel.org/linux-arm- 
>>>> kernel/20241202092419.11777-2-yangyicong@huawei.com/
>>>> Signed-off-by: Junhao He <hejunhao3@huawei.com>
>>>> ---
>>>>   .../hwtracing/coresight/coresight-tmc-etr.c   | 77 
>>>> +++++++++++--------
>>>>   1 file changed, 47 insertions(+), 30 deletions(-)
>>>>
>>>> diff --git a/drivers/hwtracing/coresight/coresight-tmc-etr.c b/ 
>>>> drivers/hwtracing/coresight/coresight-tmc-etr.c
>>>> index a48bb85d0e7f..3d94d64cacaa 100644
>>>> --- a/drivers/hwtracing/coresight/coresight-tmc-etr.c
>>>> +++ b/drivers/hwtracing/coresight/coresight-tmc-etr.c
>>>> @@ -1190,11 +1190,6 @@ static struct etr_buf 
>>>> *tmc_etr_get_sysfs_buffer(struct coresight_device *csdev)
>>>>           spin_lock_irqsave(&drvdata->spinlock, flags);
>>>>       }
>>>>   -    if (drvdata->reading || coresight_get_mode(csdev) == 
>>>> CS_MODE_PERF) {
>>>> -        ret = -EBUSY;
>>>> -        goto out;
>>>> -    }
>>>> -
>>>>       /*
>>>>        * If we don't have a buffer or it doesn't match the 
>>>> requested size,
>>>>        * use the buffer allocated above. Otherwise reuse the 
>>>> existing buffer.
>>>> @@ -1205,7 +1200,6 @@ static struct etr_buf 
>>>> *tmc_etr_get_sysfs_buffer(struct coresight_device *csdev)
>>>>           drvdata->sysfs_buf = new_buf;
>>>>       }
>>>>   -out:
>>>>       spin_unlock_irqrestore(&drvdata->spinlock, flags);
>>>>         /* Free memory outside the spinlock if need be */
>>>> @@ -1216,7 +1210,7 @@ static struct etr_buf 
>>>> *tmc_etr_get_sysfs_buffer(struct coresight_device *csdev)
>>>>     static int tmc_enable_etr_sink_sysfs(struct coresight_device 
>>>> *csdev)
>>>>   {
>>>> -    int ret = 0;
>>>> +    int ret;
>>>>       unsigned long flags;
>>>>       struct tmc_drvdata *drvdata = 
>>>> dev_get_drvdata(csdev->dev.parent);
>>>>       struct etr_buf *sysfs_buf = tmc_etr_get_sysfs_buffer(csdev);
>>>> @@ -1226,23 +1220,10 @@ static int tmc_enable_etr_sink_sysfs(struct 
>>>> coresight_device *csdev)
>>>>         spin_lock_irqsave(&drvdata->spinlock, flags);
>>>>   -    /*
>>>> -     * In sysFS mode we can have multiple writers per sink. Since 
>>>> this
>>>> -     * sink is already enabled no memory is needed and the HW need 
>>>> not be
>>>> -     * touched, even if the buffer size has changed.
>>>> -     */
>>>> -    if (coresight_get_mode(csdev) == CS_MODE_SYSFS) {
>>>> -        csdev->refcnt++;
>>>> -        goto out;
>>>> -    }
>>>> -
>>>>       ret = tmc_etr_enable_hw(drvdata, sysfs_buf);
>>>> -    if (!ret) {
>>>> -        coresight_set_mode(csdev, CS_MODE_SYSFS);
>>>> +    if (!ret)
>>>>           csdev->refcnt++;
>>>> -    }
>>>>   -out:
>>>>       spin_unlock_irqrestore(&drvdata->spinlock, flags);
>>>>         if (!ret)
>>>> @@ -1652,11 +1633,6 @@ static int tmc_enable_etr_sink_perf(struct 
>>>> coresight_device *csdev, void *data)
>>>>       struct etr_perf_buffer *etr_perf = etm_perf_sink_config(handle);
>>>>         spin_lock_irqsave(&drvdata->spinlock, flags);
>>>> -     /* Don't use this sink if it is already claimed by sysFS */
>>>> -    if (coresight_get_mode(csdev) == CS_MODE_SYSFS) {
>>>> -        rc = -EBUSY;
>>>> -        goto unlock_out;
>>>> -    }
>>>>         if (WARN_ON(!etr_perf || !etr_perf->etr_buf)) {
>>>>           rc = -EINVAL;
>>>> @@ -1685,7 +1661,6 @@ static int tmc_enable_etr_sink_perf(struct 
>>>> coresight_device *csdev, void *data)
>>>>       if (!rc) {
>>>>           /* Associate with monitored process. */
>>>>           drvdata->pid = pid;
>>>> -        coresight_set_mode(csdev, CS_MODE_PERF);
>>>>           drvdata->perf_buf = etr_perf->etr_buf;
>>>>           csdev->refcnt++;
>>>>       }
>>>> @@ -1698,14 +1673,56 @@ static int tmc_enable_etr_sink_perf(struct 
>>>> coresight_device *csdev, void *data)
>>>>   static int tmc_enable_etr_sink(struct coresight_device *csdev,
>>>>                      enum cs_mode mode, void *data)
>>>>   {
>>>> +    struct tmc_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
>>>> +    enum cs_mode old_mode;
>>>> +    int rc;
>>>> +
>>>> +    scoped_guard(spinlock_irqsave, &drvdata->spinlock) {
>>>> +        old_mode = coresight_get_mode(csdev);
>>>> +        if (old_mode != CS_MODE_DISABLED && old_mode != mode)
>>>> +            return -EBUSY;
>>>> +
>>>> +        if (drvdata->reading)
>>>> +            return -EBUSY;
>>>> +
>>>> +        /*
>>>> +         * In sysFS mode we can have multiple writers per sink. 
>>>> Since this
>>>> +         * sink is already enabled no memory is needed and the HW 
>>>> need not be
>>>> +         * touched, even if the buffer size has changed.
>>>> +         */
>>>> +        if (old_mode == CS_MODE_SYSFS) {
>>>> +            csdev->refcnt++;
>>>> +            return 0;
>>>> +        }
>>>> +
>>>> +        /*
>>>> +         * minor note:
>>>> +         * When sysfs-task1 get locked, it setup the mode first. Then
>>>> +         * sysfs-task2 gets locked，it will directly return success 
>>>> even
>>>> +         * when the tmc-etr is not enabled at this moment. 
>>>> Ultimately,
>>>> +         * sysfs-task1 will still successfully enable tmc-etr.
>>>> +         * This is a transient state and does not cause an anomaly.
>>>> +         */
>>>> +        coresight_set_mode(csdev, mode);
>>>> +    }
>>>> +
>>>>       switch (mode) {
>>>>       case CS_MODE_SYSFS:
>>>> -        return tmc_enable_etr_sink_sysfs(csdev);
>>>> +        rc = tmc_enable_etr_sink_sysfs(csdev);
>>>> +        break;
>>>>       case CS_MODE_PERF:
>>>> -        return tmc_enable_etr_sink_perf(csdev, data);
>>>> +        rc = tmc_enable_etr_sink_perf(csdev, data);
>>>> +        break;
>>>>       default:
>>>> -        return -EINVAL;
>>>> +        rc = -EINVAL;
>>>>       }
>>>> +
>>>> +    scoped_guard(spinlock_irqsave, &drvdata->spinlock) {
>>>> +        if (rc && old_mode != mode)
>>>> +            coresight_set_mode(csdev, old_mode);
>>>> +    }
>>>> +
>>>> +    return rc;
>>>>   }
>>>>     static int tmc_disable_etr_sink(struct coresight_device *csdev)
>>>
>>> .
>>>
>>
>
> .
>