* [PATCH v4 0/2] RISC-V IOMMU HPM support
@ 2026-07-14 8:36 ` Zong Li
0 siblings, 0 replies; 14+ messages in thread
From: Zong Li @ 2026-07-14 8:36 UTC (permalink / raw)
To: tjeznach, joro, will, robin.murphy, pjw, palmer, aou, alex,
mark.rutland, andrew.jones, guoren, david.laight.linux,
zhangzhanpeng.jasper, iommu, linux-riscv, linux-kernel,
linux-perf-users
Cc: Zong Li
This series implements support for the RISC-V IOMMU hardware performance
monitor.
The RISC-V IOMMU PMU driver is implemented as an auxiliary device driver
created by the parent RISC-V IOMMU driver. Therefore, the child driver
can obtain resources and information from the parent device, such as
the MMIO base address and IRQ number.
Changed in v3:
- Rebased oto v7.2-rc3
- Use hi_lo_writeq/readq to access register
- Pick comments from sashiko-bot as follows
- Set IRQ CPU affinity
- Remove IRQF_ONESHOT flag when request irq
- Adjust cycle event check by checking event_id field only
- Fix bug for group events verificaiton
- Fix KASAN issue about casting 32-bit variable to unsigned long pointer
- Clear IPSR pending bit before starting counter
- Clear OF bit in event selector register in irq handler
- Release irq by devm instead of explicit free_irq
Changed in v2:
- Rebased onto v7.2-rc1
- Use hi-lo-hi mechanism to read counter.
Suggested by Guo Ren and David Laight
Changed in v1:
- Rebased onto v6.19-rc8
- Pick all suggestions and feedbacks from v1 series
- Add cpu hotplug implementation to avoid race enablement
- Move PMU-related definition from header to c file
- Change PMU driver to auxiliary device driver
Changed in RFC:
- Rebase onto v6.13-rc7
- Clear interrupt pending before handling interrupt
- Fix the counter value issue caused by OF bit in the cycle counter.
- Invoke riscv_iommu_hpm_disable() instead of riscv_iommu_pmu_uninit()
in riscv_iommu_remove()
Zong Li (2):
drivers/perf: riscv-iommu: add risc-v iommu pmu driver
iommu/riscv: create a auxiliary device for HPM
drivers/iommu/riscv/Kconfig | 1 +
drivers/iommu/riscv/iommu-bits.h | 61 ---
drivers/iommu/riscv/iommu.c | 19 +
drivers/perf/Kconfig | 12 +
drivers/perf/Makefile | 1 +
drivers/perf/riscv_iommu_pmu.c | 722 +++++++++++++++++++++++++++++++
6 files changed, 755 insertions(+), 61 deletions(-)
create mode 100644 drivers/perf/riscv_iommu_pmu.c
--
2.43.7
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v4 0/2] RISC-V IOMMU HPM support
@ 2026-07-14 8:36 ` Zong Li
0 siblings, 0 replies; 14+ messages in thread
From: Zong Li @ 2026-07-14 8:36 UTC (permalink / raw)
To: tjeznach, joro, will, robin.murphy, pjw, palmer, aou, alex,
mark.rutland, andrew.jones, guoren, david.laight.linux,
zhangzhanpeng.jasper, iommu, linux-riscv, linux-kernel,
linux-perf-users
Cc: Zong Li
This series implements support for the RISC-V IOMMU hardware performance
monitor.
The RISC-V IOMMU PMU driver is implemented as an auxiliary device driver
created by the parent RISC-V IOMMU driver. Therefore, the child driver
can obtain resources and information from the parent device, such as
the MMIO base address and IRQ number.
Changed in v3:
- Rebased oto v7.2-rc3
- Use hi_lo_writeq/readq to access register
- Pick comments from sashiko-bot as follows
- Set IRQ CPU affinity
- Remove IRQF_ONESHOT flag when request irq
- Adjust cycle event check by checking event_id field only
- Fix bug for group events verificaiton
- Fix KASAN issue about casting 32-bit variable to unsigned long pointer
- Clear IPSR pending bit before starting counter
- Clear OF bit in event selector register in irq handler
- Release irq by devm instead of explicit free_irq
Changed in v2:
- Rebased onto v7.2-rc1
- Use hi-lo-hi mechanism to read counter.
Suggested by Guo Ren and David Laight
Changed in v1:
- Rebased onto v6.19-rc8
- Pick all suggestions and feedbacks from v1 series
- Add cpu hotplug implementation to avoid race enablement
- Move PMU-related definition from header to c file
- Change PMU driver to auxiliary device driver
Changed in RFC:
- Rebase onto v6.13-rc7
- Clear interrupt pending before handling interrupt
- Fix the counter value issue caused by OF bit in the cycle counter.
- Invoke riscv_iommu_hpm_disable() instead of riscv_iommu_pmu_uninit()
in riscv_iommu_remove()
Zong Li (2):
drivers/perf: riscv-iommu: add risc-v iommu pmu driver
iommu/riscv: create a auxiliary device for HPM
drivers/iommu/riscv/Kconfig | 1 +
drivers/iommu/riscv/iommu-bits.h | 61 ---
drivers/iommu/riscv/iommu.c | 19 +
drivers/perf/Kconfig | 12 +
drivers/perf/Makefile | 1 +
drivers/perf/riscv_iommu_pmu.c | 722 +++++++++++++++++++++++++++++++
6 files changed, 755 insertions(+), 61 deletions(-)
create mode 100644 drivers/perf/riscv_iommu_pmu.c
--
2.43.7
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v4 1/2] drivers/perf: riscv-iommu: add risc-v iommu pmu driver
2026-07-14 8:36 ` Zong Li
@ 2026-07-14 8:36 ` Zong Li
-1 siblings, 0 replies; 14+ messages in thread
From: Zong Li @ 2026-07-14 8:36 UTC (permalink / raw)
To: tjeznach, joro, will, robin.murphy, pjw, palmer, aou, alex,
mark.rutland, andrew.jones, guoren, david.laight.linux,
zhangzhanpeng.jasper, iommu, linux-riscv, linux-kernel,
linux-perf-users
Cc: Zong Li
Add a new driver to support the RISC-V IOMMU PMU. This is an auxiliary
device driver created by the parent RISC-V IOMMU driver.
The RISC-V IOMMU PMU separates the cycle counter from the event counters.
The cycle counter is not associated with iohpmevt0, so a software-defined
cycle event is required for the perf subsystem.
The number and width of the counters are hardware-implemented and must
be detected at runtime.
The performance monitor provides counters with filtering support to
collect events for specific device ID/process ID, or GSCID/PSCID.
According to RISC-V IOMMU specification Chapter 6:
Whether an 8 byte access to an IOMMU register is single-copy atomic is
UNSPECIFIED. Use two separate 4 byte accesses for hardware
compatibility.
PMU-related definitions are moved into the perf driver, where they are
used exclusively.
Suggested-by: David Laight <david.laight.linux@gmail.com>
Suggested-by: Guo Ren <guoren@kernel.org>
Link: https://lore.kernel.org/linux-riscv/20260618143634.7f3dd6c5@pumpkin/
Signed-off-by: Zong Li <zong.li@sifive.com>
---
drivers/iommu/riscv/iommu-bits.h | 61 ---
drivers/perf/Kconfig | 12 +
drivers/perf/Makefile | 1 +
drivers/perf/riscv_iommu_pmu.c | 722 +++++++++++++++++++++++++++++++
4 files changed, 735 insertions(+), 61 deletions(-)
create mode 100644 drivers/perf/riscv_iommu_pmu.c
diff --git a/drivers/iommu/riscv/iommu-bits.h b/drivers/iommu/riscv/iommu-bits.h
index f2ef9bd3cde9..6b5de913a032 100644
--- a/drivers/iommu/riscv/iommu-bits.h
+++ b/drivers/iommu/riscv/iommu-bits.h
@@ -192,67 +192,6 @@ enum riscv_iommu_ddtp_modes {
#define RISCV_IOMMU_IPSR_PMIP BIT(RISCV_IOMMU_INTR_PM)
#define RISCV_IOMMU_IPSR_PIP BIT(RISCV_IOMMU_INTR_PQ)
-/* 5.19 Performance monitoring counter overflow status (32bits) */
-#define RISCV_IOMMU_REG_IOCOUNTOVF 0x0058
-#define RISCV_IOMMU_IOCOUNTOVF_CY BIT(0)
-#define RISCV_IOMMU_IOCOUNTOVF_HPM GENMASK_ULL(31, 1)
-
-/* 5.20 Performance monitoring counter inhibits (32bits) */
-#define RISCV_IOMMU_REG_IOCOUNTINH 0x005C
-#define RISCV_IOMMU_IOCOUNTINH_CY BIT(0)
-#define RISCV_IOMMU_IOCOUNTINH_HPM GENMASK(31, 1)
-
-/* 5.21 Performance monitoring cycles counter (64bits) */
-#define RISCV_IOMMU_REG_IOHPMCYCLES 0x0060
-#define RISCV_IOMMU_IOHPMCYCLES_COUNTER GENMASK_ULL(62, 0)
-#define RISCV_IOMMU_IOHPMCYCLES_OF BIT_ULL(63)
-
-/* 5.22 Performance monitoring event counters (31 * 64bits) */
-#define RISCV_IOMMU_REG_IOHPMCTR_BASE 0x0068
-#define RISCV_IOMMU_REG_IOHPMCTR(_n) (RISCV_IOMMU_REG_IOHPMCTR_BASE + ((_n) * 0x8))
-
-/* 5.23 Performance monitoring event selectors (31 * 64bits) */
-#define RISCV_IOMMU_REG_IOHPMEVT_BASE 0x0160
-#define RISCV_IOMMU_REG_IOHPMEVT(_n) (RISCV_IOMMU_REG_IOHPMEVT_BASE + ((_n) * 0x8))
-#define RISCV_IOMMU_IOHPMEVT_EVENTID GENMASK_ULL(14, 0)
-#define RISCV_IOMMU_IOHPMEVT_DMASK BIT_ULL(15)
-#define RISCV_IOMMU_IOHPMEVT_PID_PSCID GENMASK_ULL(35, 16)
-#define RISCV_IOMMU_IOHPMEVT_DID_GSCID GENMASK_ULL(59, 36)
-#define RISCV_IOMMU_IOHPMEVT_PV_PSCV BIT_ULL(60)
-#define RISCV_IOMMU_IOHPMEVT_DV_GSCV BIT_ULL(61)
-#define RISCV_IOMMU_IOHPMEVT_IDT BIT_ULL(62)
-#define RISCV_IOMMU_IOHPMEVT_OF BIT_ULL(63)
-
-/* Number of defined performance-monitoring event selectors */
-#define RISCV_IOMMU_IOHPMEVT_CNT 31
-
-/**
- * enum riscv_iommu_hpmevent_id - Performance-monitoring event identifier
- *
- * @RISCV_IOMMU_HPMEVENT_INVALID: Invalid event, do not count
- * @RISCV_IOMMU_HPMEVENT_URQ: Untranslated requests
- * @RISCV_IOMMU_HPMEVENT_TRQ: Translated requests
- * @RISCV_IOMMU_HPMEVENT_ATS_RQ: ATS translation requests
- * @RISCV_IOMMU_HPMEVENT_TLB_MISS: TLB misses
- * @RISCV_IOMMU_HPMEVENT_DD_WALK: Device directory walks
- * @RISCV_IOMMU_HPMEVENT_PD_WALK: Process directory walks
- * @RISCV_IOMMU_HPMEVENT_S_VS_WALKS: First-stage page table walks
- * @RISCV_IOMMU_HPMEVENT_G_WALKS: Second-stage page table walks
- * @RISCV_IOMMU_HPMEVENT_MAX: Value to denote maximum Event IDs
- */
-enum riscv_iommu_hpmevent_id {
- RISCV_IOMMU_HPMEVENT_INVALID = 0,
- RISCV_IOMMU_HPMEVENT_URQ = 1,
- RISCV_IOMMU_HPMEVENT_TRQ = 2,
- RISCV_IOMMU_HPMEVENT_ATS_RQ = 3,
- RISCV_IOMMU_HPMEVENT_TLB_MISS = 4,
- RISCV_IOMMU_HPMEVENT_DD_WALK = 5,
- RISCV_IOMMU_HPMEVENT_PD_WALK = 6,
- RISCV_IOMMU_HPMEVENT_S_VS_WALKS = 7,
- RISCV_IOMMU_HPMEVENT_G_WALKS = 8,
- RISCV_IOMMU_HPMEVENT_MAX = 9
-};
-
/* 5.24 Translation request IOVA (64bits) */
#define RISCV_IOMMU_REG_TR_REQ_IOVA 0x0258
#define RISCV_IOMMU_TR_REQ_IOVA_VPN GENMASK_ULL(63, 12)
diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig
index 245e7bb763b9..8cce6c2ea626 100644
--- a/drivers/perf/Kconfig
+++ b/drivers/perf/Kconfig
@@ -105,6 +105,18 @@ config RISCV_PMU_SBI
full perf feature support i.e. counter overflow, privilege mode
filtering, counter configuration.
+config RISCV_IOMMU_PMU
+ depends on RISCV || COMPILE_TEST
+ depends on RISCV_IOMMU
+ bool "RISC-V IOMMU Hardware Performance Monitor"
+ default y
+ help
+ Say Y if you want to use the RISC-V IOMMU performance monitor
+ implementation. The performance monitor is an optional hardware
+ feature, and whether it is actually enabled depends on IOMMU
+ hardware support. If the underlying hardware does not implement
+ the PMU, this option will have no effect.
+
config STARFIVE_STARLINK_PMU
depends on ARCH_STARFIVE || COMPILE_TEST
depends on 64BIT
diff --git a/drivers/perf/Makefile b/drivers/perf/Makefile
index eb8a022dad9a..90c75f3c0ac1 100644
--- a/drivers/perf/Makefile
+++ b/drivers/perf/Makefile
@@ -20,6 +20,7 @@ obj-$(CONFIG_QCOM_L3_PMU) += qcom_l3_pmu.o
obj-$(CONFIG_RISCV_PMU) += riscv_pmu.o
obj-$(CONFIG_RISCV_PMU_LEGACY) += riscv_pmu_legacy.o
obj-$(CONFIG_RISCV_PMU_SBI) += riscv_pmu_sbi.o
+obj-$(CONFIG_RISCV_IOMMU_PMU) += riscv_iommu_pmu.o
obj-$(CONFIG_STARFIVE_STARLINK_PMU) += starfive_starlink_pmu.o
obj-$(CONFIG_THUNDERX2_PMU) += thunderx2_pmu.o
obj-$(CONFIG_XGENE_PMU) += xgene_pmu.o
diff --git a/drivers/perf/riscv_iommu_pmu.c b/drivers/perf/riscv_iommu_pmu.c
new file mode 100644
index 000000000000..eb331d4633e1
--- /dev/null
+++ b/drivers/perf/riscv_iommu_pmu.c
@@ -0,0 +1,722 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026 SiFive
+ *
+ * Authors
+ * Zong Li <zong.li@sifive.com>
+ */
+
+#include <linux/auxiliary_bus.h>
+#include <linux/io-64-nonatomic-hi-lo.h>
+#include <linux/perf_event.h>
+
+#include "../iommu/riscv/iommu.h"
+
+/* 5.19 Performance monitoring counter overflow status (32bits) */
+#define RISCV_IOMMU_REG_IOCOUNTOVF 0x0058
+#define RISCV_IOMMU_IOCOUNTOVF_CY BIT(0)
+#define RISCV_IOMMU_IOCOUNTOVF_HPM GENMASK_ULL(31, 1)
+
+/* 5.20 Performance monitoring counter inhibits (32bits) */
+#define RISCV_IOMMU_REG_IOCOUNTINH 0x005C
+#define RISCV_IOMMU_IOCOUNTINH_CY BIT(0)
+#define RISCV_IOMMU_IOCOUNTINH_HPM GENMASK(31, 0)
+
+/* 5.21 Performance monitoring cycles counter (64bits) */
+#define RISCV_IOMMU_REG_IOHPMCYCLES 0x0060
+#define RISCV_IOMMU_IOHPMCYCLES_COUNTER GENMASK_ULL(62, 0)
+#define RISCV_IOMMU_IOHPMCYCLES_OF BIT_ULL(63)
+#define RISCV_IOMMU_REG_IOHPMCTR(_n) (RISCV_IOMMU_REG_IOHPMCYCLES + ((_n) * 0x8))
+
+/* 5.22 Performance monitoring event counters (31 * 64bits) */
+#define RISCV_IOMMU_REG_IOHPMCTR_BASE 0x0068
+#define RISCV_IOMMU_IOHPMCTR_COUNTER GENMASK_ULL(63, 0)
+
+/* 5.23 Performance monitoring event selectors (31 * 64bits) */
+#define RISCV_IOMMU_REG_IOHPMEVT_BASE 0x0160
+#define RISCV_IOMMU_REG_IOHPMEVT(_n) (RISCV_IOMMU_REG_IOHPMEVT_BASE + ((_n) * 0x8))
+#define RISCV_IOMMU_IOHPMEVT_EVENTID GENMASK_ULL(14, 0)
+#define RISCV_IOMMU_IOHPMEVT_DMASK BIT_ULL(15)
+#define RISCV_IOMMU_IOHPMEVT_PID_PSCID GENMASK_ULL(35, 16)
+#define RISCV_IOMMU_IOHPMEVT_DID_GSCID GENMASK_ULL(59, 36)
+#define RISCV_IOMMU_IOHPMEVT_PV_PSCV BIT_ULL(60)
+#define RISCV_IOMMU_IOHPMEVT_DV_GSCV BIT_ULL(61)
+#define RISCV_IOMMU_IOHPMEVT_IDT BIT_ULL(62)
+#define RISCV_IOMMU_IOHPMEVT_OF BIT_ULL(63)
+#define RISCV_IOMMU_IOHPMEVT_EVENT GENMASK_ULL(62, 0)
+
+/* The total number of counters is 31 event counters plus 1 cycle counter */
+#define RISCV_IOMMU_HPM_COUNTER_NUM 32
+
+static int cpuhp_state;
+
+/**
+ * enum riscv_iommu_hpmevent_id - Performance-monitoring event identifier
+ *
+ * @RISCV_IOMMU_HPMEVENT_CYCLE: Clock cycle counter
+ * @RISCV_IOMMU_HPMEVENT_URQ: Untranslated requests
+ * @RISCV_IOMMU_HPMEVENT_TRQ: Translated requests
+ * @RISCV_IOMMU_HPMEVENT_ATS_RQ: ATS translation requests
+ * @RISCV_IOMMU_HPMEVENT_TLB_MISS: TLB misses
+ * @RISCV_IOMMU_HPMEVENT_DD_WALK: Device directory walks
+ * @RISCV_IOMMU_HPMEVENT_PD_WALK: Process directory walks
+ * @RISCV_IOMMU_HPMEVENT_S_VS_WALKS: First-stage page table walks
+ * @RISCV_IOMMU_HPMEVENT_G_WALKS: Second-stage page table walks
+ * @RISCV_IOMMU_HPMEVENT_MAX: Value to denote maximum Event IDs
+ *
+ * The specification does not define an event ID for counting the
+ * number of clock cycles, meaning there is no associated 'iohpmevt0'.
+ * Event ID 0 is an invalid event and does not overlap with any valid
+ * event ID. Let's repurpose ID 0 as the cycle for perf, the cycle
+ * event is not actually written into any register, it serves solely
+ * as an identifier.
+ */
+enum riscv_iommu_hpmevent_id {
+ RISCV_IOMMU_HPMEVENT_CYCLE = 0,
+ RISCV_IOMMU_HPMEVENT_URQ = 1,
+ RISCV_IOMMU_HPMEVENT_TRQ = 2,
+ RISCV_IOMMU_HPMEVENT_ATS_RQ = 3,
+ RISCV_IOMMU_HPMEVENT_TLB_MISS = 4,
+ RISCV_IOMMU_HPMEVENT_DD_WALK = 5,
+ RISCV_IOMMU_HPMEVENT_PD_WALK = 6,
+ RISCV_IOMMU_HPMEVENT_S_VS_WALKS = 7,
+ RISCV_IOMMU_HPMEVENT_G_WALKS = 8,
+ RISCV_IOMMU_HPMEVENT_MAX = 9
+};
+
+struct riscv_iommu_pmu {
+ struct pmu pmu;
+ struct hlist_node node;
+ void __iomem *reg;
+ unsigned int on_cpu;
+ unsigned int irq;
+ int num_counters;
+ u64 cycle_cntr_mask;
+ u64 event_cntr_mask;
+ struct perf_event *events[RISCV_IOMMU_HPM_COUNTER_NUM];
+ DECLARE_BITMAP(used_counters, RISCV_IOMMU_HPM_COUNTER_NUM);
+ u32 hi_prev[RISCV_IOMMU_HPM_COUNTER_NUM];
+ u32 lo_prev[RISCV_IOMMU_HPM_COUNTER_NUM];
+};
+
+#define to_riscv_iommu_pmu(p) (container_of(p, struct riscv_iommu_pmu, pmu))
+
+#define RISCV_IOMMU_PMU_ATTR_EXTRACTOR(_name, _mask) \
+ static inline u32 get_##_name(struct perf_event *event) \
+ { \
+ return FIELD_GET(_mask, event->attr.config); \
+ } \
+
+RISCV_IOMMU_PMU_ATTR_EXTRACTOR(event, RISCV_IOMMU_IOHPMEVT_EVENTID);
+RISCV_IOMMU_PMU_ATTR_EXTRACTOR(partial_matching, RISCV_IOMMU_IOHPMEVT_DMASK);
+RISCV_IOMMU_PMU_ATTR_EXTRACTOR(pid_pscid, RISCV_IOMMU_IOHPMEVT_PID_PSCID);
+RISCV_IOMMU_PMU_ATTR_EXTRACTOR(did_gscid, RISCV_IOMMU_IOHPMEVT_DID_GSCID);
+RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_pid_pscid, RISCV_IOMMU_IOHPMEVT_PV_PSCV);
+RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_did_gscid, RISCV_IOMMU_IOHPMEVT_DV_GSCV);
+RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_id_type, RISCV_IOMMU_IOHPMEVT_IDT);
+
+/* Formats */
+PMU_FORMAT_ATTR(event, "config:0-14");
+PMU_FORMAT_ATTR(partial_matching, "config:15");
+PMU_FORMAT_ATTR(pid_pscid, "config:16-35");
+PMU_FORMAT_ATTR(did_gscid, "config:36-59");
+PMU_FORMAT_ATTR(filter_pid_pscid, "config:60");
+PMU_FORMAT_ATTR(filter_did_gscid, "config:61");
+PMU_FORMAT_ATTR(filter_id_type, "config:62");
+
+static struct attribute *riscv_iommu_pmu_formats[] = {
+ &format_attr_event.attr,
+ &format_attr_partial_matching.attr,
+ &format_attr_pid_pscid.attr,
+ &format_attr_did_gscid.attr,
+ &format_attr_filter_pid_pscid.attr,
+ &format_attr_filter_did_gscid.attr,
+ &format_attr_filter_id_type.attr,
+ NULL,
+};
+
+static const struct attribute_group riscv_iommu_pmu_format_group = {
+ .name = "format",
+ .attrs = riscv_iommu_pmu_formats,
+};
+
+/* Events */
+static ssize_t riscv_iommu_pmu_event_show(struct device *dev,
+ struct device_attribute *attr,
+ char *page)
+{
+ struct perf_pmu_events_attr *pmu_attr;
+
+ pmu_attr = container_of(attr, struct perf_pmu_events_attr, attr);
+
+ return sysfs_emit(page, "event=0x%02llx\n", pmu_attr->id);
+}
+
+#define RISCV_IOMMU_PMU_EVENT_ATTR(name, id) \
+ PMU_EVENT_ATTR_ID(name, riscv_iommu_pmu_event_show, id)
+
+static struct attribute *riscv_iommu_pmu_events[] = {
+ RISCV_IOMMU_PMU_EVENT_ATTR(cycle, RISCV_IOMMU_HPMEVENT_CYCLE),
+ RISCV_IOMMU_PMU_EVENT_ATTR(untranslated_req, RISCV_IOMMU_HPMEVENT_URQ),
+ RISCV_IOMMU_PMU_EVENT_ATTR(translated_req, RISCV_IOMMU_HPMEVENT_TRQ),
+ RISCV_IOMMU_PMU_EVENT_ATTR(ats_trans_req, RISCV_IOMMU_HPMEVENT_ATS_RQ),
+ RISCV_IOMMU_PMU_EVENT_ATTR(tlb_miss, RISCV_IOMMU_HPMEVENT_TLB_MISS),
+ RISCV_IOMMU_PMU_EVENT_ATTR(ddt_walks, RISCV_IOMMU_HPMEVENT_DD_WALK),
+ RISCV_IOMMU_PMU_EVENT_ATTR(pdt_walks, RISCV_IOMMU_HPMEVENT_PD_WALK),
+ RISCV_IOMMU_PMU_EVENT_ATTR(s_vs_pt_walks, RISCV_IOMMU_HPMEVENT_S_VS_WALKS),
+ RISCV_IOMMU_PMU_EVENT_ATTR(g_pt_walks, RISCV_IOMMU_HPMEVENT_G_WALKS),
+ NULL,
+};
+
+static const struct attribute_group riscv_iommu_pmu_events_group = {
+ .name = "events",
+ .attrs = riscv_iommu_pmu_events,
+};
+
+/* cpumask */
+static ssize_t riscv_iommu_cpumask_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(dev_get_drvdata(dev));
+
+ return cpumap_print_to_pagebuf(true, buf, cpumask_of(pmu->on_cpu));
+}
+
+static struct device_attribute riscv_iommu_cpumask_attr =
+ __ATTR(cpumask, 0444, riscv_iommu_cpumask_show, NULL);
+
+static struct attribute *riscv_iommu_cpumask_attrs[] = {
+ &riscv_iommu_cpumask_attr.attr,
+ NULL
+};
+
+static const struct attribute_group riscv_iommu_pmu_cpumask_group = {
+ .attrs = riscv_iommu_cpumask_attrs,
+};
+
+static const struct attribute_group *riscv_iommu_pmu_attr_grps[] = {
+ &riscv_iommu_pmu_cpumask_group,
+ &riscv_iommu_pmu_format_group,
+ &riscv_iommu_pmu_events_group,
+ NULL,
+};
+
+/*
+ * Register access wrapper
+ *
+ * According to RISC-V IOMMU specification Chapter 6:
+ * A 4 byte access to an IOMMU register must be single-copy atomic.
+ * Whether an 8 byte access to an IOMMU register is single-copy atomic is UNSPECIFIED
+ *
+ * Use two separate 4 byte accesses for hardware compatibility
+ */
+static u64 riscv_iommu_pmu_readq(void __iomem *addr)
+{
+ return hi_lo_readq(addr);
+}
+
+static void riscv_iommu_pmu_writeq(u64 value, void __iomem *addr)
+{
+ hi_lo_writeq(value, addr);
+}
+
+/* PMU Operations */
+static void riscv_iommu_pmu_set_counter(struct riscv_iommu_pmu *pmu, u32 idx,
+ u64 value)
+{
+ u64 counter_mask = idx ? pmu->event_cntr_mask : pmu->cycle_cntr_mask;
+
+ riscv_iommu_pmu_writeq(value & counter_mask, pmu->reg + RISCV_IOMMU_REG_IOHPMCTR(idx));
+}
+
+/*
+ * As stated in the RISC-V IOMMU Specification, Chapter 6:
+ * Whether an 8 byte access to an IOMMU register is single-copy atomic
+ * is UNSPECIFIED, and such an access may appear, internally to the
+ * IOMMU, as if two separate 4 byte accesses - first to the high half
+ * and second to the low half - were performed
+ *
+ * To make sure the driver works correctly on different hardware,
+ * the software will always use two 4-byte access for the counter.
+ *
+ * This function implements the hi-lo-hi pattern to detect and handle
+ * wraparound during the read operation:
+ * 1. Read high half (hi)
+ * 2. Read low half (lo)
+ * 3. Check if low half wrapped or high half changed:
+ * - If lo <= lo_prev: possible wraparound occurred
+ * - If hi != hi_prev: high half changed during read
+ * 4. If wraparound detected, re-read high half and assume low half is 0
+ * 5. Update previous values for next read
+ */
+static u64 riscv_iommu_pmu_get_counter(struct riscv_iommu_pmu *pmu, u32 idx)
+{
+ void __iomem *addr = pmu->reg + RISCV_IOMMU_REG_IOHPMCTR(idx);
+ u64 value, counter_mask = idx ? pmu->event_cntr_mask : pmu->cycle_cntr_mask;
+ u32 hi, lo;
+
+ hi = readl(addr + 4);
+ lo = readl(addr);
+
+ if (lo <= pmu->lo_prev[idx] || hi != pmu->hi_prev[idx]) {
+ u32 hi_tmp = readl(addr + 4);
+
+ /*
+ * If hi changes, then hi+1:0 must have happened while
+ * the code was running so it is a safe return value
+ */
+ if (hi_tmp != hi) {
+ hi = hi_tmp;
+ lo = 0;
+ }
+ pmu->lo_prev[idx] = ~0u;
+ pmu->hi_prev[idx] = hi;
+ }
+ pmu->lo_prev[idx] = lo;
+
+ value = (((u64)hi << 32) | lo) & counter_mask;
+
+ /* The bit 63 of cycle counter (i.e., idx == 0) is OF bit */
+ return idx ? value : (value & ~RISCV_IOMMU_IOHPMCYCLES_OF);
+}
+
+static bool is_cycle_event(u64 event)
+{
+ return FIELD_GET(RISCV_IOMMU_IOHPMEVT_EVENTID, event) ==
+ RISCV_IOMMU_HPMEVENT_CYCLE;
+}
+
+static void riscv_iommu_pmu_set_event(struct riscv_iommu_pmu *pmu, u32 idx,
+ u64 value)
+{
+ /* There is no associtated IOHPMEVT0 for IOHPMCYCLES */
+ if (is_cycle_event(value))
+ return;
+
+ /* Event counter start from idx 1 */
+ riscv_iommu_pmu_writeq(FIELD_GET(RISCV_IOMMU_IOHPMEVT_EVENT, value),
+ pmu->reg + RISCV_IOMMU_REG_IOHPMEVT(idx - 1));
+}
+
+static void riscv_iommu_pmu_enable_counter(struct riscv_iommu_pmu *pmu, u32 idx)
+{
+ void __iomem *addr = pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH;
+ u32 value = readl(addr);
+
+ writel(value & ~BIT(idx), addr);
+}
+
+static void riscv_iommu_pmu_disable_counter(struct riscv_iommu_pmu *pmu, u32 idx)
+{
+ void __iomem *addr = pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH;
+ u32 value = readl(addr);
+
+ writel(value | BIT(idx), addr);
+}
+
+static void riscv_iommu_pmu_clear_ovf(struct perf_event *event, u32 idx)
+{
+ struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
+ u64 value;
+
+ /* Counter is disabled here, making it safe to read and write registers */
+ if (is_cycle_event(get_event(event))) {
+ value = riscv_iommu_pmu_readq(pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES) &
+ ~RISCV_IOMMU_IOHPMCYCLES_OF;
+ riscv_iommu_pmu_writeq(value, pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES);
+ } else {
+ /* Event counter start from idx 1 */
+ value = riscv_iommu_pmu_readq(pmu->reg + RISCV_IOMMU_REG_IOHPMEVT(idx - 1)) &
+ ~RISCV_IOMMU_IOHPMEVT_OF;
+ riscv_iommu_pmu_writeq(value, pmu->reg + RISCV_IOMMU_REG_IOHPMEVT(idx - 1));
+ }
+}
+static void riscv_iommu_pmu_start_all(struct riscv_iommu_pmu *pmu)
+{
+ void __iomem *addr = pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH;
+ u32 used_cntr = 0;
+
+ /* The performance-monitoring counter inhibits is a 32-bit WARL register */
+ bitmap_to_arr32(&used_cntr, pmu->used_counters, pmu->num_counters);
+
+ writel(~used_cntr, addr);
+}
+
+static void riscv_iommu_pmu_stop_all(struct riscv_iommu_pmu *pmu)
+{
+ writel(GENMASK_ULL(pmu->num_counters - 1, 0),
+ pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH);
+}
+
+/* PMU APIs */
+static void riscv_iommu_pmu_set_period(struct perf_event *event)
+{
+ struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
+ struct hw_perf_event *hwc = &event->hw;
+ u64 counter_mask = hwc->idx ? pmu->event_cntr_mask : pmu->cycle_cntr_mask;
+ u64 period;
+
+ /*
+ * Limit the maximum period to prevent the counter value
+ * from overtaking the one we are about to program.
+ * In effect we are reducing max_period to account for
+ * interrupt latency (and we are being very conservative).
+ */
+ period = counter_mask >> 1;
+ riscv_iommu_pmu_set_counter(pmu, hwc->idx, period);
+ local64_set(&hwc->prev_count, period);
+}
+
+static int riscv_iommu_pmu_event_init(struct perf_event *event)
+{
+ struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
+ struct hw_perf_event *hwc = &event->hw;
+ struct perf_event *sibling;
+ int total_event_counters = pmu->num_counters - 1;
+ int counters = 0;
+
+ if (event->attr.type != event->pmu->type)
+ return -ENOENT;
+
+ if (hwc->sample_period)
+ return -EOPNOTSUPP;
+
+ if (event->cpu < 0)
+ return -EOPNOTSUPP;
+
+ event->cpu = pmu->on_cpu;
+
+ hwc->idx = -1;
+ hwc->config = event->attr.config;
+
+ if (event->group_leader == event)
+ return 0;
+
+ if (!is_cycle_event(get_event(event->group_leader)))
+ if (++counters > total_event_counters)
+ return -EINVAL;
+
+ for_each_sibling_event(sibling, event->group_leader) {
+ if (is_cycle_event(get_event(sibling)) || is_software_event(sibling))
+ continue;
+
+ if (sibling->pmu != event->pmu)
+ return -EINVAL;
+
+ if (++counters > total_event_counters)
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static void riscv_iommu_pmu_update(struct perf_event *event)
+{
+ struct hw_perf_event *hwc = &event->hw;
+ struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
+ u64 delta, prev, now;
+ u32 idx = hwc->idx;
+ u64 counter_mask = idx ? pmu->event_cntr_mask : pmu->cycle_cntr_mask;
+
+ do {
+ prev = local64_read(&hwc->prev_count);
+ now = riscv_iommu_pmu_get_counter(pmu, idx);
+ } while (local64_cmpxchg(&hwc->prev_count, prev, now) != prev);
+
+ delta = (now - prev) & counter_mask;
+ local64_add(delta, &event->count);
+}
+
+static void riscv_iommu_pmu_start(struct perf_event *event, int flags)
+{
+ struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
+ struct hw_perf_event *hwc = &event->hw;
+
+ if (WARN_ON_ONCE(!(event->hw.state & PERF_HES_STOPPED)))
+ return;
+
+ if (flags & PERF_EF_RELOAD)
+ WARN_ON_ONCE(!(event->hw.state & PERF_HES_UPTODATE));
+
+ hwc->state = 0;
+ riscv_iommu_pmu_set_period(event);
+ riscv_iommu_pmu_set_event(pmu, hwc->idx, hwc->config);
+ riscv_iommu_pmu_enable_counter(pmu, hwc->idx);
+
+ perf_event_update_userpage(event);
+}
+
+static void riscv_iommu_pmu_stop(struct perf_event *event, int flags)
+{
+ struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
+ struct hw_perf_event *hwc = &event->hw;
+ int idx = hwc->idx;
+
+ if (hwc->state & PERF_HES_STOPPED)
+ return;
+
+ riscv_iommu_pmu_disable_counter(pmu, idx);
+
+ if ((flags & PERF_EF_UPDATE) && !(hwc->state & PERF_HES_UPTODATE))
+ riscv_iommu_pmu_update(event);
+
+ hwc->state |= PERF_HES_STOPPED | PERF_HES_UPTODATE;
+}
+
+static int riscv_iommu_pmu_add(struct perf_event *event, int flags)
+{
+ struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
+ struct hw_perf_event *hwc = &event->hw;
+ unsigned int num_counters = pmu->num_counters;
+ int idx;
+
+ /* Reserve index zero for iohpmcycles */
+ if (is_cycle_event(get_event(event)))
+ idx = 0;
+ else
+ idx = find_next_zero_bit(pmu->used_counters, num_counters, 1);
+
+ /* All event counters or cycle counter are in use */
+ if (idx == num_counters || pmu->events[idx])
+ return -EAGAIN;
+
+ set_bit(idx, pmu->used_counters);
+
+ pmu->events[idx] = event;
+ hwc->idx = idx;
+ hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
+ local64_set(&hwc->prev_count, 0);
+
+ if (flags & PERF_EF_START)
+ riscv_iommu_pmu_start(event, flags);
+
+ /* Propagate changes to the userspace mapping. */
+ perf_event_update_userpage(event);
+
+ return 0;
+}
+
+static void riscv_iommu_pmu_read(struct perf_event *event)
+{
+ riscv_iommu_pmu_update(event);
+}
+
+static void riscv_iommu_pmu_del(struct perf_event *event, int flags)
+{
+ struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
+ struct hw_perf_event *hwc = &event->hw;
+ int idx = hwc->idx;
+
+ riscv_iommu_pmu_stop(event, PERF_EF_UPDATE);
+ pmu->events[idx] = NULL;
+ clear_bit(idx, pmu->used_counters);
+
+ perf_event_update_userpage(event);
+}
+
+static int riscv_iommu_pmu_online_cpu(unsigned int cpu, struct hlist_node *node)
+{
+ struct riscv_iommu_pmu *iommu_pmu;
+
+ iommu_pmu = hlist_entry_safe(node, struct riscv_iommu_pmu, node);
+
+ if (iommu_pmu->on_cpu == -1)
+ iommu_pmu->on_cpu = cpu;
+
+ return 0;
+}
+
+static int riscv_iommu_pmu_offline_cpu(unsigned int cpu, struct hlist_node *node)
+{
+ struct riscv_iommu_pmu *iommu_pmu;
+ unsigned int target_cpu;
+
+ iommu_pmu = hlist_entry_safe(node, struct riscv_iommu_pmu, node);
+
+ if (cpu != iommu_pmu->on_cpu)
+ return 0;
+
+ iommu_pmu->on_cpu = -1;
+
+ target_cpu = cpumask_any_but(cpu_online_mask, cpu);
+ if (target_cpu >= nr_cpu_ids)
+ return 0;
+
+ perf_pmu_migrate_context(&iommu_pmu->pmu, cpu, target_cpu);
+ iommu_pmu->on_cpu = target_cpu;
+ WARN_ON(irq_set_affinity(iommu_pmu->irq, cpumask_of(target_cpu)));
+
+ return 0;
+}
+
+static irqreturn_t riscv_iommu_pmu_irq_handler(int irq, void *dev_id)
+{
+ struct riscv_iommu_pmu *pmu = (struct riscv_iommu_pmu *)dev_id;
+ DECLARE_BITMAP(ovf_bitmap, RISCV_IOMMU_HPM_COUNTER_NUM);
+ u32 ovf, idx;
+
+ /* Check whether this interrupt is for PMU */
+ if (!(readl_relaxed(pmu->reg + RISCV_IOMMU_REG_IPSR) & RISCV_IOMMU_IPSR_PMIP))
+ return IRQ_NONE;
+
+ /* Process PMU IRQ */
+ riscv_iommu_pmu_stop_all(pmu);
+
+ ovf = readl(pmu->reg + RISCV_IOMMU_REG_IOCOUNTOVF);
+ if (!ovf)
+ return IRQ_NONE;
+
+ bitmap_from_u64(ovf_bitmap, ovf);
+ for_each_set_bit(idx, ovf_bitmap, pmu->num_counters) {
+ struct perf_event *event = pmu->events[idx];
+
+ if (WARN_ON_ONCE(!event))
+ continue;
+
+ riscv_iommu_pmu_update(event);
+ riscv_iommu_pmu_set_period(event);
+ riscv_iommu_pmu_clear_ovf(event, idx);
+ }
+
+ /* Clear performance monitoring interrupt pending bit */
+ writel_relaxed(RISCV_IOMMU_IPSR_PMIP, pmu->reg + RISCV_IOMMU_REG_IPSR);
+
+ riscv_iommu_pmu_start_all(pmu);
+
+ return IRQ_HANDLED;
+}
+
+static unsigned int riscv_iommu_pmu_get_irq_num(struct riscv_iommu_device *iommu)
+{
+ /* Reuse ICVEC.CIV mask for all interrupt vectors mapping */
+ int vec = (iommu->icvec >> (RISCV_IOMMU_INTR_PM * 4)) & RISCV_IOMMU_ICVEC_CIV;
+
+ return iommu->irqs[vec];
+}
+
+static int riscv_iommu_pmu_request_irq(struct riscv_iommu_device *iommu,
+ struct riscv_iommu_pmu *pmu)
+{
+ return devm_request_irq(iommu->dev, pmu->irq, riscv_iommu_pmu_irq_handler,
+ IRQF_NOBALANCING | IRQF_SHARED, dev_name(iommu->dev), pmu);
+}
+
+static int riscv_iommu_pmu_probe(struct auxiliary_device *auxdev,
+ const struct auxiliary_device_id *id)
+{
+ struct riscv_iommu_device *iommu_dev = dev_get_platdata(&auxdev->dev);
+ struct riscv_iommu_pmu *iommu_pmu;
+ void __iomem *addr;
+ char *name;
+ int ret;
+
+ iommu_pmu = devm_kzalloc(&auxdev->dev, sizeof(*iommu_pmu), GFP_KERNEL);
+ if (!iommu_pmu)
+ return -ENOMEM;
+
+ iommu_pmu->reg = iommu_dev->reg;
+
+ /* Counter number and width are hardware-implemented. Detect them by write 1s */
+ addr = iommu_pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH;
+ writel(RISCV_IOMMU_IOCOUNTINH_HPM, addr);
+ iommu_pmu->num_counters = hweight32(readl(addr));
+
+ addr = iommu_pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES;
+ riscv_iommu_pmu_writeq(RISCV_IOMMU_IOHPMCYCLES_COUNTER, addr);
+ iommu_pmu->cycle_cntr_mask = riscv_iommu_pmu_readq(addr);
+
+ /* Assume the width of all event counters are the same */
+ addr = iommu_pmu->reg + RISCV_IOMMU_REG_IOHPMCTR_BASE;
+ riscv_iommu_pmu_writeq(RISCV_IOMMU_IOHPMCTR_COUNTER, addr);
+ iommu_pmu->event_cntr_mask = riscv_iommu_pmu_readq(addr);
+
+ iommu_pmu->pmu = (struct pmu) {
+ .module = THIS_MODULE,
+ .parent = &auxdev->dev,
+ .task_ctx_nr = perf_invalid_context,
+ .event_init = riscv_iommu_pmu_event_init,
+ .add = riscv_iommu_pmu_add,
+ .del = riscv_iommu_pmu_del,
+ .start = riscv_iommu_pmu_start,
+ .stop = riscv_iommu_pmu_stop,
+ .read = riscv_iommu_pmu_read,
+ .attr_groups = riscv_iommu_pmu_attr_grps,
+ .capabilities = PERF_PMU_CAP_NO_EXCLUDE,
+ };
+
+ auxiliary_set_drvdata(auxdev, iommu_pmu);
+
+ name = devm_kasprintf(&auxdev->dev, GFP_KERNEL,
+ "riscv_iommu_pmu_%s", dev_name(iommu_dev->dev));
+ if (!name) {
+ dev_err(&auxdev->dev, "Failed to create name riscv_iommu_pmu_%s\n",
+ dev_name(iommu_dev->dev));
+ return -ENOMEM;
+ }
+
+ /* Bind all events to the same cpu context to avoid race enabling */
+ iommu_pmu->on_cpu = raw_smp_processor_id();
+ iommu_pmu->irq = riscv_iommu_pmu_get_irq_num(iommu_dev);
+ WARN_ON(irq_set_affinity(iommu_pmu->irq, cpumask_of(iommu_pmu->on_cpu)));
+
+ ret = riscv_iommu_pmu_request_irq(iommu_dev, iommu_pmu);
+ if (ret) {
+ dev_err(&auxdev->dev, "Failed to request irq %s: %d\n", name, ret);
+ return ret;
+ }
+
+ ret = cpuhp_state_add_instance_nocalls(cpuhp_state, &iommu_pmu->node);
+ if (ret) {
+ dev_err(&auxdev->dev, "Failed to register hotplug %s: %d\n", name, ret);
+ return ret;
+ }
+
+ ret = perf_pmu_register(&iommu_pmu->pmu, name, -1);
+ if (ret) {
+ dev_err(&auxdev->dev, "Failed to registe %s: %d\n", name, ret);
+ goto err_unregister;
+ }
+
+ dev_info(&auxdev->dev, "%s: Registered with %d counters\n",
+ name, iommu_pmu->num_counters);
+
+ return 0;
+
+err_unregister:
+ cpuhp_state_remove_instance_nocalls(cpuhp_state, &iommu_pmu->node);
+ return ret;
+}
+
+static const struct auxiliary_device_id riscv_iommu_pmu_id_table[] = {
+ { .name = "iommu.pmu" },
+ {}
+};
+MODULE_DEVICE_TABLE(auxiliary, riscv_iommu_pmu_id_table);
+
+static struct auxiliary_driver iommu_pmu_driver = {
+ .probe = riscv_iommu_pmu_probe,
+ .id_table = riscv_iommu_pmu_id_table,
+};
+
+static int __init riscv_iommu_pmu_init(void)
+{
+ int ret;
+
+ cpuhp_state = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN,
+ "perf/riscv/iommu:online",
+ riscv_iommu_pmu_online_cpu,
+ riscv_iommu_pmu_offline_cpu);
+ if (cpuhp_state < 0)
+ return cpuhp_state;
+
+ ret = auxiliary_driver_register(&iommu_pmu_driver);
+ if (ret)
+ cpuhp_remove_multi_state(cpuhp_state);
+
+ return ret;
+}
+module_init(riscv_iommu_pmu_init);
+
+MODULE_DESCRIPTION("RISC-V IOMMU PMU");
+MODULE_LICENSE("GPL");
--
2.43.7
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v4 1/2] drivers/perf: riscv-iommu: add risc-v iommu pmu driver
@ 2026-07-14 8:36 ` Zong Li
0 siblings, 0 replies; 14+ messages in thread
From: Zong Li @ 2026-07-14 8:36 UTC (permalink / raw)
To: tjeznach, joro, will, robin.murphy, pjw, palmer, aou, alex,
mark.rutland, andrew.jones, guoren, david.laight.linux,
zhangzhanpeng.jasper, iommu, linux-riscv, linux-kernel,
linux-perf-users
Cc: Zong Li
Add a new driver to support the RISC-V IOMMU PMU. This is an auxiliary
device driver created by the parent RISC-V IOMMU driver.
The RISC-V IOMMU PMU separates the cycle counter from the event counters.
The cycle counter is not associated with iohpmevt0, so a software-defined
cycle event is required for the perf subsystem.
The number and width of the counters are hardware-implemented and must
be detected at runtime.
The performance monitor provides counters with filtering support to
collect events for specific device ID/process ID, or GSCID/PSCID.
According to RISC-V IOMMU specification Chapter 6:
Whether an 8 byte access to an IOMMU register is single-copy atomic is
UNSPECIFIED. Use two separate 4 byte accesses for hardware
compatibility.
PMU-related definitions are moved into the perf driver, where they are
used exclusively.
Suggested-by: David Laight <david.laight.linux@gmail.com>
Suggested-by: Guo Ren <guoren@kernel.org>
Link: https://lore.kernel.org/linux-riscv/20260618143634.7f3dd6c5@pumpkin/
Signed-off-by: Zong Li <zong.li@sifive.com>
---
drivers/iommu/riscv/iommu-bits.h | 61 ---
drivers/perf/Kconfig | 12 +
drivers/perf/Makefile | 1 +
drivers/perf/riscv_iommu_pmu.c | 722 +++++++++++++++++++++++++++++++
4 files changed, 735 insertions(+), 61 deletions(-)
create mode 100644 drivers/perf/riscv_iommu_pmu.c
diff --git a/drivers/iommu/riscv/iommu-bits.h b/drivers/iommu/riscv/iommu-bits.h
index f2ef9bd3cde9..6b5de913a032 100644
--- a/drivers/iommu/riscv/iommu-bits.h
+++ b/drivers/iommu/riscv/iommu-bits.h
@@ -192,67 +192,6 @@ enum riscv_iommu_ddtp_modes {
#define RISCV_IOMMU_IPSR_PMIP BIT(RISCV_IOMMU_INTR_PM)
#define RISCV_IOMMU_IPSR_PIP BIT(RISCV_IOMMU_INTR_PQ)
-/* 5.19 Performance monitoring counter overflow status (32bits) */
-#define RISCV_IOMMU_REG_IOCOUNTOVF 0x0058
-#define RISCV_IOMMU_IOCOUNTOVF_CY BIT(0)
-#define RISCV_IOMMU_IOCOUNTOVF_HPM GENMASK_ULL(31, 1)
-
-/* 5.20 Performance monitoring counter inhibits (32bits) */
-#define RISCV_IOMMU_REG_IOCOUNTINH 0x005C
-#define RISCV_IOMMU_IOCOUNTINH_CY BIT(0)
-#define RISCV_IOMMU_IOCOUNTINH_HPM GENMASK(31, 1)
-
-/* 5.21 Performance monitoring cycles counter (64bits) */
-#define RISCV_IOMMU_REG_IOHPMCYCLES 0x0060
-#define RISCV_IOMMU_IOHPMCYCLES_COUNTER GENMASK_ULL(62, 0)
-#define RISCV_IOMMU_IOHPMCYCLES_OF BIT_ULL(63)
-
-/* 5.22 Performance monitoring event counters (31 * 64bits) */
-#define RISCV_IOMMU_REG_IOHPMCTR_BASE 0x0068
-#define RISCV_IOMMU_REG_IOHPMCTR(_n) (RISCV_IOMMU_REG_IOHPMCTR_BASE + ((_n) * 0x8))
-
-/* 5.23 Performance monitoring event selectors (31 * 64bits) */
-#define RISCV_IOMMU_REG_IOHPMEVT_BASE 0x0160
-#define RISCV_IOMMU_REG_IOHPMEVT(_n) (RISCV_IOMMU_REG_IOHPMEVT_BASE + ((_n) * 0x8))
-#define RISCV_IOMMU_IOHPMEVT_EVENTID GENMASK_ULL(14, 0)
-#define RISCV_IOMMU_IOHPMEVT_DMASK BIT_ULL(15)
-#define RISCV_IOMMU_IOHPMEVT_PID_PSCID GENMASK_ULL(35, 16)
-#define RISCV_IOMMU_IOHPMEVT_DID_GSCID GENMASK_ULL(59, 36)
-#define RISCV_IOMMU_IOHPMEVT_PV_PSCV BIT_ULL(60)
-#define RISCV_IOMMU_IOHPMEVT_DV_GSCV BIT_ULL(61)
-#define RISCV_IOMMU_IOHPMEVT_IDT BIT_ULL(62)
-#define RISCV_IOMMU_IOHPMEVT_OF BIT_ULL(63)
-
-/* Number of defined performance-monitoring event selectors */
-#define RISCV_IOMMU_IOHPMEVT_CNT 31
-
-/**
- * enum riscv_iommu_hpmevent_id - Performance-monitoring event identifier
- *
- * @RISCV_IOMMU_HPMEVENT_INVALID: Invalid event, do not count
- * @RISCV_IOMMU_HPMEVENT_URQ: Untranslated requests
- * @RISCV_IOMMU_HPMEVENT_TRQ: Translated requests
- * @RISCV_IOMMU_HPMEVENT_ATS_RQ: ATS translation requests
- * @RISCV_IOMMU_HPMEVENT_TLB_MISS: TLB misses
- * @RISCV_IOMMU_HPMEVENT_DD_WALK: Device directory walks
- * @RISCV_IOMMU_HPMEVENT_PD_WALK: Process directory walks
- * @RISCV_IOMMU_HPMEVENT_S_VS_WALKS: First-stage page table walks
- * @RISCV_IOMMU_HPMEVENT_G_WALKS: Second-stage page table walks
- * @RISCV_IOMMU_HPMEVENT_MAX: Value to denote maximum Event IDs
- */
-enum riscv_iommu_hpmevent_id {
- RISCV_IOMMU_HPMEVENT_INVALID = 0,
- RISCV_IOMMU_HPMEVENT_URQ = 1,
- RISCV_IOMMU_HPMEVENT_TRQ = 2,
- RISCV_IOMMU_HPMEVENT_ATS_RQ = 3,
- RISCV_IOMMU_HPMEVENT_TLB_MISS = 4,
- RISCV_IOMMU_HPMEVENT_DD_WALK = 5,
- RISCV_IOMMU_HPMEVENT_PD_WALK = 6,
- RISCV_IOMMU_HPMEVENT_S_VS_WALKS = 7,
- RISCV_IOMMU_HPMEVENT_G_WALKS = 8,
- RISCV_IOMMU_HPMEVENT_MAX = 9
-};
-
/* 5.24 Translation request IOVA (64bits) */
#define RISCV_IOMMU_REG_TR_REQ_IOVA 0x0258
#define RISCV_IOMMU_TR_REQ_IOVA_VPN GENMASK_ULL(63, 12)
diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig
index 245e7bb763b9..8cce6c2ea626 100644
--- a/drivers/perf/Kconfig
+++ b/drivers/perf/Kconfig
@@ -105,6 +105,18 @@ config RISCV_PMU_SBI
full perf feature support i.e. counter overflow, privilege mode
filtering, counter configuration.
+config RISCV_IOMMU_PMU
+ depends on RISCV || COMPILE_TEST
+ depends on RISCV_IOMMU
+ bool "RISC-V IOMMU Hardware Performance Monitor"
+ default y
+ help
+ Say Y if you want to use the RISC-V IOMMU performance monitor
+ implementation. The performance monitor is an optional hardware
+ feature, and whether it is actually enabled depends on IOMMU
+ hardware support. If the underlying hardware does not implement
+ the PMU, this option will have no effect.
+
config STARFIVE_STARLINK_PMU
depends on ARCH_STARFIVE || COMPILE_TEST
depends on 64BIT
diff --git a/drivers/perf/Makefile b/drivers/perf/Makefile
index eb8a022dad9a..90c75f3c0ac1 100644
--- a/drivers/perf/Makefile
+++ b/drivers/perf/Makefile
@@ -20,6 +20,7 @@ obj-$(CONFIG_QCOM_L3_PMU) += qcom_l3_pmu.o
obj-$(CONFIG_RISCV_PMU) += riscv_pmu.o
obj-$(CONFIG_RISCV_PMU_LEGACY) += riscv_pmu_legacy.o
obj-$(CONFIG_RISCV_PMU_SBI) += riscv_pmu_sbi.o
+obj-$(CONFIG_RISCV_IOMMU_PMU) += riscv_iommu_pmu.o
obj-$(CONFIG_STARFIVE_STARLINK_PMU) += starfive_starlink_pmu.o
obj-$(CONFIG_THUNDERX2_PMU) += thunderx2_pmu.o
obj-$(CONFIG_XGENE_PMU) += xgene_pmu.o
diff --git a/drivers/perf/riscv_iommu_pmu.c b/drivers/perf/riscv_iommu_pmu.c
new file mode 100644
index 000000000000..eb331d4633e1
--- /dev/null
+++ b/drivers/perf/riscv_iommu_pmu.c
@@ -0,0 +1,722 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026 SiFive
+ *
+ * Authors
+ * Zong Li <zong.li@sifive.com>
+ */
+
+#include <linux/auxiliary_bus.h>
+#include <linux/io-64-nonatomic-hi-lo.h>
+#include <linux/perf_event.h>
+
+#include "../iommu/riscv/iommu.h"
+
+/* 5.19 Performance monitoring counter overflow status (32bits) */
+#define RISCV_IOMMU_REG_IOCOUNTOVF 0x0058
+#define RISCV_IOMMU_IOCOUNTOVF_CY BIT(0)
+#define RISCV_IOMMU_IOCOUNTOVF_HPM GENMASK_ULL(31, 1)
+
+/* 5.20 Performance monitoring counter inhibits (32bits) */
+#define RISCV_IOMMU_REG_IOCOUNTINH 0x005C
+#define RISCV_IOMMU_IOCOUNTINH_CY BIT(0)
+#define RISCV_IOMMU_IOCOUNTINH_HPM GENMASK(31, 0)
+
+/* 5.21 Performance monitoring cycles counter (64bits) */
+#define RISCV_IOMMU_REG_IOHPMCYCLES 0x0060
+#define RISCV_IOMMU_IOHPMCYCLES_COUNTER GENMASK_ULL(62, 0)
+#define RISCV_IOMMU_IOHPMCYCLES_OF BIT_ULL(63)
+#define RISCV_IOMMU_REG_IOHPMCTR(_n) (RISCV_IOMMU_REG_IOHPMCYCLES + ((_n) * 0x8))
+
+/* 5.22 Performance monitoring event counters (31 * 64bits) */
+#define RISCV_IOMMU_REG_IOHPMCTR_BASE 0x0068
+#define RISCV_IOMMU_IOHPMCTR_COUNTER GENMASK_ULL(63, 0)
+
+/* 5.23 Performance monitoring event selectors (31 * 64bits) */
+#define RISCV_IOMMU_REG_IOHPMEVT_BASE 0x0160
+#define RISCV_IOMMU_REG_IOHPMEVT(_n) (RISCV_IOMMU_REG_IOHPMEVT_BASE + ((_n) * 0x8))
+#define RISCV_IOMMU_IOHPMEVT_EVENTID GENMASK_ULL(14, 0)
+#define RISCV_IOMMU_IOHPMEVT_DMASK BIT_ULL(15)
+#define RISCV_IOMMU_IOHPMEVT_PID_PSCID GENMASK_ULL(35, 16)
+#define RISCV_IOMMU_IOHPMEVT_DID_GSCID GENMASK_ULL(59, 36)
+#define RISCV_IOMMU_IOHPMEVT_PV_PSCV BIT_ULL(60)
+#define RISCV_IOMMU_IOHPMEVT_DV_GSCV BIT_ULL(61)
+#define RISCV_IOMMU_IOHPMEVT_IDT BIT_ULL(62)
+#define RISCV_IOMMU_IOHPMEVT_OF BIT_ULL(63)
+#define RISCV_IOMMU_IOHPMEVT_EVENT GENMASK_ULL(62, 0)
+
+/* The total number of counters is 31 event counters plus 1 cycle counter */
+#define RISCV_IOMMU_HPM_COUNTER_NUM 32
+
+static int cpuhp_state;
+
+/**
+ * enum riscv_iommu_hpmevent_id - Performance-monitoring event identifier
+ *
+ * @RISCV_IOMMU_HPMEVENT_CYCLE: Clock cycle counter
+ * @RISCV_IOMMU_HPMEVENT_URQ: Untranslated requests
+ * @RISCV_IOMMU_HPMEVENT_TRQ: Translated requests
+ * @RISCV_IOMMU_HPMEVENT_ATS_RQ: ATS translation requests
+ * @RISCV_IOMMU_HPMEVENT_TLB_MISS: TLB misses
+ * @RISCV_IOMMU_HPMEVENT_DD_WALK: Device directory walks
+ * @RISCV_IOMMU_HPMEVENT_PD_WALK: Process directory walks
+ * @RISCV_IOMMU_HPMEVENT_S_VS_WALKS: First-stage page table walks
+ * @RISCV_IOMMU_HPMEVENT_G_WALKS: Second-stage page table walks
+ * @RISCV_IOMMU_HPMEVENT_MAX: Value to denote maximum Event IDs
+ *
+ * The specification does not define an event ID for counting the
+ * number of clock cycles, meaning there is no associated 'iohpmevt0'.
+ * Event ID 0 is an invalid event and does not overlap with any valid
+ * event ID. Let's repurpose ID 0 as the cycle for perf, the cycle
+ * event is not actually written into any register, it serves solely
+ * as an identifier.
+ */
+enum riscv_iommu_hpmevent_id {
+ RISCV_IOMMU_HPMEVENT_CYCLE = 0,
+ RISCV_IOMMU_HPMEVENT_URQ = 1,
+ RISCV_IOMMU_HPMEVENT_TRQ = 2,
+ RISCV_IOMMU_HPMEVENT_ATS_RQ = 3,
+ RISCV_IOMMU_HPMEVENT_TLB_MISS = 4,
+ RISCV_IOMMU_HPMEVENT_DD_WALK = 5,
+ RISCV_IOMMU_HPMEVENT_PD_WALK = 6,
+ RISCV_IOMMU_HPMEVENT_S_VS_WALKS = 7,
+ RISCV_IOMMU_HPMEVENT_G_WALKS = 8,
+ RISCV_IOMMU_HPMEVENT_MAX = 9
+};
+
+struct riscv_iommu_pmu {
+ struct pmu pmu;
+ struct hlist_node node;
+ void __iomem *reg;
+ unsigned int on_cpu;
+ unsigned int irq;
+ int num_counters;
+ u64 cycle_cntr_mask;
+ u64 event_cntr_mask;
+ struct perf_event *events[RISCV_IOMMU_HPM_COUNTER_NUM];
+ DECLARE_BITMAP(used_counters, RISCV_IOMMU_HPM_COUNTER_NUM);
+ u32 hi_prev[RISCV_IOMMU_HPM_COUNTER_NUM];
+ u32 lo_prev[RISCV_IOMMU_HPM_COUNTER_NUM];
+};
+
+#define to_riscv_iommu_pmu(p) (container_of(p, struct riscv_iommu_pmu, pmu))
+
+#define RISCV_IOMMU_PMU_ATTR_EXTRACTOR(_name, _mask) \
+ static inline u32 get_##_name(struct perf_event *event) \
+ { \
+ return FIELD_GET(_mask, event->attr.config); \
+ } \
+
+RISCV_IOMMU_PMU_ATTR_EXTRACTOR(event, RISCV_IOMMU_IOHPMEVT_EVENTID);
+RISCV_IOMMU_PMU_ATTR_EXTRACTOR(partial_matching, RISCV_IOMMU_IOHPMEVT_DMASK);
+RISCV_IOMMU_PMU_ATTR_EXTRACTOR(pid_pscid, RISCV_IOMMU_IOHPMEVT_PID_PSCID);
+RISCV_IOMMU_PMU_ATTR_EXTRACTOR(did_gscid, RISCV_IOMMU_IOHPMEVT_DID_GSCID);
+RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_pid_pscid, RISCV_IOMMU_IOHPMEVT_PV_PSCV);
+RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_did_gscid, RISCV_IOMMU_IOHPMEVT_DV_GSCV);
+RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_id_type, RISCV_IOMMU_IOHPMEVT_IDT);
+
+/* Formats */
+PMU_FORMAT_ATTR(event, "config:0-14");
+PMU_FORMAT_ATTR(partial_matching, "config:15");
+PMU_FORMAT_ATTR(pid_pscid, "config:16-35");
+PMU_FORMAT_ATTR(did_gscid, "config:36-59");
+PMU_FORMAT_ATTR(filter_pid_pscid, "config:60");
+PMU_FORMAT_ATTR(filter_did_gscid, "config:61");
+PMU_FORMAT_ATTR(filter_id_type, "config:62");
+
+static struct attribute *riscv_iommu_pmu_formats[] = {
+ &format_attr_event.attr,
+ &format_attr_partial_matching.attr,
+ &format_attr_pid_pscid.attr,
+ &format_attr_did_gscid.attr,
+ &format_attr_filter_pid_pscid.attr,
+ &format_attr_filter_did_gscid.attr,
+ &format_attr_filter_id_type.attr,
+ NULL,
+};
+
+static const struct attribute_group riscv_iommu_pmu_format_group = {
+ .name = "format",
+ .attrs = riscv_iommu_pmu_formats,
+};
+
+/* Events */
+static ssize_t riscv_iommu_pmu_event_show(struct device *dev,
+ struct device_attribute *attr,
+ char *page)
+{
+ struct perf_pmu_events_attr *pmu_attr;
+
+ pmu_attr = container_of(attr, struct perf_pmu_events_attr, attr);
+
+ return sysfs_emit(page, "event=0x%02llx\n", pmu_attr->id);
+}
+
+#define RISCV_IOMMU_PMU_EVENT_ATTR(name, id) \
+ PMU_EVENT_ATTR_ID(name, riscv_iommu_pmu_event_show, id)
+
+static struct attribute *riscv_iommu_pmu_events[] = {
+ RISCV_IOMMU_PMU_EVENT_ATTR(cycle, RISCV_IOMMU_HPMEVENT_CYCLE),
+ RISCV_IOMMU_PMU_EVENT_ATTR(untranslated_req, RISCV_IOMMU_HPMEVENT_URQ),
+ RISCV_IOMMU_PMU_EVENT_ATTR(translated_req, RISCV_IOMMU_HPMEVENT_TRQ),
+ RISCV_IOMMU_PMU_EVENT_ATTR(ats_trans_req, RISCV_IOMMU_HPMEVENT_ATS_RQ),
+ RISCV_IOMMU_PMU_EVENT_ATTR(tlb_miss, RISCV_IOMMU_HPMEVENT_TLB_MISS),
+ RISCV_IOMMU_PMU_EVENT_ATTR(ddt_walks, RISCV_IOMMU_HPMEVENT_DD_WALK),
+ RISCV_IOMMU_PMU_EVENT_ATTR(pdt_walks, RISCV_IOMMU_HPMEVENT_PD_WALK),
+ RISCV_IOMMU_PMU_EVENT_ATTR(s_vs_pt_walks, RISCV_IOMMU_HPMEVENT_S_VS_WALKS),
+ RISCV_IOMMU_PMU_EVENT_ATTR(g_pt_walks, RISCV_IOMMU_HPMEVENT_G_WALKS),
+ NULL,
+};
+
+static const struct attribute_group riscv_iommu_pmu_events_group = {
+ .name = "events",
+ .attrs = riscv_iommu_pmu_events,
+};
+
+/* cpumask */
+static ssize_t riscv_iommu_cpumask_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(dev_get_drvdata(dev));
+
+ return cpumap_print_to_pagebuf(true, buf, cpumask_of(pmu->on_cpu));
+}
+
+static struct device_attribute riscv_iommu_cpumask_attr =
+ __ATTR(cpumask, 0444, riscv_iommu_cpumask_show, NULL);
+
+static struct attribute *riscv_iommu_cpumask_attrs[] = {
+ &riscv_iommu_cpumask_attr.attr,
+ NULL
+};
+
+static const struct attribute_group riscv_iommu_pmu_cpumask_group = {
+ .attrs = riscv_iommu_cpumask_attrs,
+};
+
+static const struct attribute_group *riscv_iommu_pmu_attr_grps[] = {
+ &riscv_iommu_pmu_cpumask_group,
+ &riscv_iommu_pmu_format_group,
+ &riscv_iommu_pmu_events_group,
+ NULL,
+};
+
+/*
+ * Register access wrapper
+ *
+ * According to RISC-V IOMMU specification Chapter 6:
+ * A 4 byte access to an IOMMU register must be single-copy atomic.
+ * Whether an 8 byte access to an IOMMU register is single-copy atomic is UNSPECIFIED
+ *
+ * Use two separate 4 byte accesses for hardware compatibility
+ */
+static u64 riscv_iommu_pmu_readq(void __iomem *addr)
+{
+ return hi_lo_readq(addr);
+}
+
+static void riscv_iommu_pmu_writeq(u64 value, void __iomem *addr)
+{
+ hi_lo_writeq(value, addr);
+}
+
+/* PMU Operations */
+static void riscv_iommu_pmu_set_counter(struct riscv_iommu_pmu *pmu, u32 idx,
+ u64 value)
+{
+ u64 counter_mask = idx ? pmu->event_cntr_mask : pmu->cycle_cntr_mask;
+
+ riscv_iommu_pmu_writeq(value & counter_mask, pmu->reg + RISCV_IOMMU_REG_IOHPMCTR(idx));
+}
+
+/*
+ * As stated in the RISC-V IOMMU Specification, Chapter 6:
+ * Whether an 8 byte access to an IOMMU register is single-copy atomic
+ * is UNSPECIFIED, and such an access may appear, internally to the
+ * IOMMU, as if two separate 4 byte accesses - first to the high half
+ * and second to the low half - were performed
+ *
+ * To make sure the driver works correctly on different hardware,
+ * the software will always use two 4-byte access for the counter.
+ *
+ * This function implements the hi-lo-hi pattern to detect and handle
+ * wraparound during the read operation:
+ * 1. Read high half (hi)
+ * 2. Read low half (lo)
+ * 3. Check if low half wrapped or high half changed:
+ * - If lo <= lo_prev: possible wraparound occurred
+ * - If hi != hi_prev: high half changed during read
+ * 4. If wraparound detected, re-read high half and assume low half is 0
+ * 5. Update previous values for next read
+ */
+static u64 riscv_iommu_pmu_get_counter(struct riscv_iommu_pmu *pmu, u32 idx)
+{
+ void __iomem *addr = pmu->reg + RISCV_IOMMU_REG_IOHPMCTR(idx);
+ u64 value, counter_mask = idx ? pmu->event_cntr_mask : pmu->cycle_cntr_mask;
+ u32 hi, lo;
+
+ hi = readl(addr + 4);
+ lo = readl(addr);
+
+ if (lo <= pmu->lo_prev[idx] || hi != pmu->hi_prev[idx]) {
+ u32 hi_tmp = readl(addr + 4);
+
+ /*
+ * If hi changes, then hi+1:0 must have happened while
+ * the code was running so it is a safe return value
+ */
+ if (hi_tmp != hi) {
+ hi = hi_tmp;
+ lo = 0;
+ }
+ pmu->lo_prev[idx] = ~0u;
+ pmu->hi_prev[idx] = hi;
+ }
+ pmu->lo_prev[idx] = lo;
+
+ value = (((u64)hi << 32) | lo) & counter_mask;
+
+ /* The bit 63 of cycle counter (i.e., idx == 0) is OF bit */
+ return idx ? value : (value & ~RISCV_IOMMU_IOHPMCYCLES_OF);
+}
+
+static bool is_cycle_event(u64 event)
+{
+ return FIELD_GET(RISCV_IOMMU_IOHPMEVT_EVENTID, event) ==
+ RISCV_IOMMU_HPMEVENT_CYCLE;
+}
+
+static void riscv_iommu_pmu_set_event(struct riscv_iommu_pmu *pmu, u32 idx,
+ u64 value)
+{
+ /* There is no associtated IOHPMEVT0 for IOHPMCYCLES */
+ if (is_cycle_event(value))
+ return;
+
+ /* Event counter start from idx 1 */
+ riscv_iommu_pmu_writeq(FIELD_GET(RISCV_IOMMU_IOHPMEVT_EVENT, value),
+ pmu->reg + RISCV_IOMMU_REG_IOHPMEVT(idx - 1));
+}
+
+static void riscv_iommu_pmu_enable_counter(struct riscv_iommu_pmu *pmu, u32 idx)
+{
+ void __iomem *addr = pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH;
+ u32 value = readl(addr);
+
+ writel(value & ~BIT(idx), addr);
+}
+
+static void riscv_iommu_pmu_disable_counter(struct riscv_iommu_pmu *pmu, u32 idx)
+{
+ void __iomem *addr = pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH;
+ u32 value = readl(addr);
+
+ writel(value | BIT(idx), addr);
+}
+
+static void riscv_iommu_pmu_clear_ovf(struct perf_event *event, u32 idx)
+{
+ struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
+ u64 value;
+
+ /* Counter is disabled here, making it safe to read and write registers */
+ if (is_cycle_event(get_event(event))) {
+ value = riscv_iommu_pmu_readq(pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES) &
+ ~RISCV_IOMMU_IOHPMCYCLES_OF;
+ riscv_iommu_pmu_writeq(value, pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES);
+ } else {
+ /* Event counter start from idx 1 */
+ value = riscv_iommu_pmu_readq(pmu->reg + RISCV_IOMMU_REG_IOHPMEVT(idx - 1)) &
+ ~RISCV_IOMMU_IOHPMEVT_OF;
+ riscv_iommu_pmu_writeq(value, pmu->reg + RISCV_IOMMU_REG_IOHPMEVT(idx - 1));
+ }
+}
+static void riscv_iommu_pmu_start_all(struct riscv_iommu_pmu *pmu)
+{
+ void __iomem *addr = pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH;
+ u32 used_cntr = 0;
+
+ /* The performance-monitoring counter inhibits is a 32-bit WARL register */
+ bitmap_to_arr32(&used_cntr, pmu->used_counters, pmu->num_counters);
+
+ writel(~used_cntr, addr);
+}
+
+static void riscv_iommu_pmu_stop_all(struct riscv_iommu_pmu *pmu)
+{
+ writel(GENMASK_ULL(pmu->num_counters - 1, 0),
+ pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH);
+}
+
+/* PMU APIs */
+static void riscv_iommu_pmu_set_period(struct perf_event *event)
+{
+ struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
+ struct hw_perf_event *hwc = &event->hw;
+ u64 counter_mask = hwc->idx ? pmu->event_cntr_mask : pmu->cycle_cntr_mask;
+ u64 period;
+
+ /*
+ * Limit the maximum period to prevent the counter value
+ * from overtaking the one we are about to program.
+ * In effect we are reducing max_period to account for
+ * interrupt latency (and we are being very conservative).
+ */
+ period = counter_mask >> 1;
+ riscv_iommu_pmu_set_counter(pmu, hwc->idx, period);
+ local64_set(&hwc->prev_count, period);
+}
+
+static int riscv_iommu_pmu_event_init(struct perf_event *event)
+{
+ struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
+ struct hw_perf_event *hwc = &event->hw;
+ struct perf_event *sibling;
+ int total_event_counters = pmu->num_counters - 1;
+ int counters = 0;
+
+ if (event->attr.type != event->pmu->type)
+ return -ENOENT;
+
+ if (hwc->sample_period)
+ return -EOPNOTSUPP;
+
+ if (event->cpu < 0)
+ return -EOPNOTSUPP;
+
+ event->cpu = pmu->on_cpu;
+
+ hwc->idx = -1;
+ hwc->config = event->attr.config;
+
+ if (event->group_leader == event)
+ return 0;
+
+ if (!is_cycle_event(get_event(event->group_leader)))
+ if (++counters > total_event_counters)
+ return -EINVAL;
+
+ for_each_sibling_event(sibling, event->group_leader) {
+ if (is_cycle_event(get_event(sibling)) || is_software_event(sibling))
+ continue;
+
+ if (sibling->pmu != event->pmu)
+ return -EINVAL;
+
+ if (++counters > total_event_counters)
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static void riscv_iommu_pmu_update(struct perf_event *event)
+{
+ struct hw_perf_event *hwc = &event->hw;
+ struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
+ u64 delta, prev, now;
+ u32 idx = hwc->idx;
+ u64 counter_mask = idx ? pmu->event_cntr_mask : pmu->cycle_cntr_mask;
+
+ do {
+ prev = local64_read(&hwc->prev_count);
+ now = riscv_iommu_pmu_get_counter(pmu, idx);
+ } while (local64_cmpxchg(&hwc->prev_count, prev, now) != prev);
+
+ delta = (now - prev) & counter_mask;
+ local64_add(delta, &event->count);
+}
+
+static void riscv_iommu_pmu_start(struct perf_event *event, int flags)
+{
+ struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
+ struct hw_perf_event *hwc = &event->hw;
+
+ if (WARN_ON_ONCE(!(event->hw.state & PERF_HES_STOPPED)))
+ return;
+
+ if (flags & PERF_EF_RELOAD)
+ WARN_ON_ONCE(!(event->hw.state & PERF_HES_UPTODATE));
+
+ hwc->state = 0;
+ riscv_iommu_pmu_set_period(event);
+ riscv_iommu_pmu_set_event(pmu, hwc->idx, hwc->config);
+ riscv_iommu_pmu_enable_counter(pmu, hwc->idx);
+
+ perf_event_update_userpage(event);
+}
+
+static void riscv_iommu_pmu_stop(struct perf_event *event, int flags)
+{
+ struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
+ struct hw_perf_event *hwc = &event->hw;
+ int idx = hwc->idx;
+
+ if (hwc->state & PERF_HES_STOPPED)
+ return;
+
+ riscv_iommu_pmu_disable_counter(pmu, idx);
+
+ if ((flags & PERF_EF_UPDATE) && !(hwc->state & PERF_HES_UPTODATE))
+ riscv_iommu_pmu_update(event);
+
+ hwc->state |= PERF_HES_STOPPED | PERF_HES_UPTODATE;
+}
+
+static int riscv_iommu_pmu_add(struct perf_event *event, int flags)
+{
+ struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
+ struct hw_perf_event *hwc = &event->hw;
+ unsigned int num_counters = pmu->num_counters;
+ int idx;
+
+ /* Reserve index zero for iohpmcycles */
+ if (is_cycle_event(get_event(event)))
+ idx = 0;
+ else
+ idx = find_next_zero_bit(pmu->used_counters, num_counters, 1);
+
+ /* All event counters or cycle counter are in use */
+ if (idx == num_counters || pmu->events[idx])
+ return -EAGAIN;
+
+ set_bit(idx, pmu->used_counters);
+
+ pmu->events[idx] = event;
+ hwc->idx = idx;
+ hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
+ local64_set(&hwc->prev_count, 0);
+
+ if (flags & PERF_EF_START)
+ riscv_iommu_pmu_start(event, flags);
+
+ /* Propagate changes to the userspace mapping. */
+ perf_event_update_userpage(event);
+
+ return 0;
+}
+
+static void riscv_iommu_pmu_read(struct perf_event *event)
+{
+ riscv_iommu_pmu_update(event);
+}
+
+static void riscv_iommu_pmu_del(struct perf_event *event, int flags)
+{
+ struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
+ struct hw_perf_event *hwc = &event->hw;
+ int idx = hwc->idx;
+
+ riscv_iommu_pmu_stop(event, PERF_EF_UPDATE);
+ pmu->events[idx] = NULL;
+ clear_bit(idx, pmu->used_counters);
+
+ perf_event_update_userpage(event);
+}
+
+static int riscv_iommu_pmu_online_cpu(unsigned int cpu, struct hlist_node *node)
+{
+ struct riscv_iommu_pmu *iommu_pmu;
+
+ iommu_pmu = hlist_entry_safe(node, struct riscv_iommu_pmu, node);
+
+ if (iommu_pmu->on_cpu == -1)
+ iommu_pmu->on_cpu = cpu;
+
+ return 0;
+}
+
+static int riscv_iommu_pmu_offline_cpu(unsigned int cpu, struct hlist_node *node)
+{
+ struct riscv_iommu_pmu *iommu_pmu;
+ unsigned int target_cpu;
+
+ iommu_pmu = hlist_entry_safe(node, struct riscv_iommu_pmu, node);
+
+ if (cpu != iommu_pmu->on_cpu)
+ return 0;
+
+ iommu_pmu->on_cpu = -1;
+
+ target_cpu = cpumask_any_but(cpu_online_mask, cpu);
+ if (target_cpu >= nr_cpu_ids)
+ return 0;
+
+ perf_pmu_migrate_context(&iommu_pmu->pmu, cpu, target_cpu);
+ iommu_pmu->on_cpu = target_cpu;
+ WARN_ON(irq_set_affinity(iommu_pmu->irq, cpumask_of(target_cpu)));
+
+ return 0;
+}
+
+static irqreturn_t riscv_iommu_pmu_irq_handler(int irq, void *dev_id)
+{
+ struct riscv_iommu_pmu *pmu = (struct riscv_iommu_pmu *)dev_id;
+ DECLARE_BITMAP(ovf_bitmap, RISCV_IOMMU_HPM_COUNTER_NUM);
+ u32 ovf, idx;
+
+ /* Check whether this interrupt is for PMU */
+ if (!(readl_relaxed(pmu->reg + RISCV_IOMMU_REG_IPSR) & RISCV_IOMMU_IPSR_PMIP))
+ return IRQ_NONE;
+
+ /* Process PMU IRQ */
+ riscv_iommu_pmu_stop_all(pmu);
+
+ ovf = readl(pmu->reg + RISCV_IOMMU_REG_IOCOUNTOVF);
+ if (!ovf)
+ return IRQ_NONE;
+
+ bitmap_from_u64(ovf_bitmap, ovf);
+ for_each_set_bit(idx, ovf_bitmap, pmu->num_counters) {
+ struct perf_event *event = pmu->events[idx];
+
+ if (WARN_ON_ONCE(!event))
+ continue;
+
+ riscv_iommu_pmu_update(event);
+ riscv_iommu_pmu_set_period(event);
+ riscv_iommu_pmu_clear_ovf(event, idx);
+ }
+
+ /* Clear performance monitoring interrupt pending bit */
+ writel_relaxed(RISCV_IOMMU_IPSR_PMIP, pmu->reg + RISCV_IOMMU_REG_IPSR);
+
+ riscv_iommu_pmu_start_all(pmu);
+
+ return IRQ_HANDLED;
+}
+
+static unsigned int riscv_iommu_pmu_get_irq_num(struct riscv_iommu_device *iommu)
+{
+ /* Reuse ICVEC.CIV mask for all interrupt vectors mapping */
+ int vec = (iommu->icvec >> (RISCV_IOMMU_INTR_PM * 4)) & RISCV_IOMMU_ICVEC_CIV;
+
+ return iommu->irqs[vec];
+}
+
+static int riscv_iommu_pmu_request_irq(struct riscv_iommu_device *iommu,
+ struct riscv_iommu_pmu *pmu)
+{
+ return devm_request_irq(iommu->dev, pmu->irq, riscv_iommu_pmu_irq_handler,
+ IRQF_NOBALANCING | IRQF_SHARED, dev_name(iommu->dev), pmu);
+}
+
+static int riscv_iommu_pmu_probe(struct auxiliary_device *auxdev,
+ const struct auxiliary_device_id *id)
+{
+ struct riscv_iommu_device *iommu_dev = dev_get_platdata(&auxdev->dev);
+ struct riscv_iommu_pmu *iommu_pmu;
+ void __iomem *addr;
+ char *name;
+ int ret;
+
+ iommu_pmu = devm_kzalloc(&auxdev->dev, sizeof(*iommu_pmu), GFP_KERNEL);
+ if (!iommu_pmu)
+ return -ENOMEM;
+
+ iommu_pmu->reg = iommu_dev->reg;
+
+ /* Counter number and width are hardware-implemented. Detect them by write 1s */
+ addr = iommu_pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH;
+ writel(RISCV_IOMMU_IOCOUNTINH_HPM, addr);
+ iommu_pmu->num_counters = hweight32(readl(addr));
+
+ addr = iommu_pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES;
+ riscv_iommu_pmu_writeq(RISCV_IOMMU_IOHPMCYCLES_COUNTER, addr);
+ iommu_pmu->cycle_cntr_mask = riscv_iommu_pmu_readq(addr);
+
+ /* Assume the width of all event counters are the same */
+ addr = iommu_pmu->reg + RISCV_IOMMU_REG_IOHPMCTR_BASE;
+ riscv_iommu_pmu_writeq(RISCV_IOMMU_IOHPMCTR_COUNTER, addr);
+ iommu_pmu->event_cntr_mask = riscv_iommu_pmu_readq(addr);
+
+ iommu_pmu->pmu = (struct pmu) {
+ .module = THIS_MODULE,
+ .parent = &auxdev->dev,
+ .task_ctx_nr = perf_invalid_context,
+ .event_init = riscv_iommu_pmu_event_init,
+ .add = riscv_iommu_pmu_add,
+ .del = riscv_iommu_pmu_del,
+ .start = riscv_iommu_pmu_start,
+ .stop = riscv_iommu_pmu_stop,
+ .read = riscv_iommu_pmu_read,
+ .attr_groups = riscv_iommu_pmu_attr_grps,
+ .capabilities = PERF_PMU_CAP_NO_EXCLUDE,
+ };
+
+ auxiliary_set_drvdata(auxdev, iommu_pmu);
+
+ name = devm_kasprintf(&auxdev->dev, GFP_KERNEL,
+ "riscv_iommu_pmu_%s", dev_name(iommu_dev->dev));
+ if (!name) {
+ dev_err(&auxdev->dev, "Failed to create name riscv_iommu_pmu_%s\n",
+ dev_name(iommu_dev->dev));
+ return -ENOMEM;
+ }
+
+ /* Bind all events to the same cpu context to avoid race enabling */
+ iommu_pmu->on_cpu = raw_smp_processor_id();
+ iommu_pmu->irq = riscv_iommu_pmu_get_irq_num(iommu_dev);
+ WARN_ON(irq_set_affinity(iommu_pmu->irq, cpumask_of(iommu_pmu->on_cpu)));
+
+ ret = riscv_iommu_pmu_request_irq(iommu_dev, iommu_pmu);
+ if (ret) {
+ dev_err(&auxdev->dev, "Failed to request irq %s: %d\n", name, ret);
+ return ret;
+ }
+
+ ret = cpuhp_state_add_instance_nocalls(cpuhp_state, &iommu_pmu->node);
+ if (ret) {
+ dev_err(&auxdev->dev, "Failed to register hotplug %s: %d\n", name, ret);
+ return ret;
+ }
+
+ ret = perf_pmu_register(&iommu_pmu->pmu, name, -1);
+ if (ret) {
+ dev_err(&auxdev->dev, "Failed to registe %s: %d\n", name, ret);
+ goto err_unregister;
+ }
+
+ dev_info(&auxdev->dev, "%s: Registered with %d counters\n",
+ name, iommu_pmu->num_counters);
+
+ return 0;
+
+err_unregister:
+ cpuhp_state_remove_instance_nocalls(cpuhp_state, &iommu_pmu->node);
+ return ret;
+}
+
+static const struct auxiliary_device_id riscv_iommu_pmu_id_table[] = {
+ { .name = "iommu.pmu" },
+ {}
+};
+MODULE_DEVICE_TABLE(auxiliary, riscv_iommu_pmu_id_table);
+
+static struct auxiliary_driver iommu_pmu_driver = {
+ .probe = riscv_iommu_pmu_probe,
+ .id_table = riscv_iommu_pmu_id_table,
+};
+
+static int __init riscv_iommu_pmu_init(void)
+{
+ int ret;
+
+ cpuhp_state = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN,
+ "perf/riscv/iommu:online",
+ riscv_iommu_pmu_online_cpu,
+ riscv_iommu_pmu_offline_cpu);
+ if (cpuhp_state < 0)
+ return cpuhp_state;
+
+ ret = auxiliary_driver_register(&iommu_pmu_driver);
+ if (ret)
+ cpuhp_remove_multi_state(cpuhp_state);
+
+ return ret;
+}
+module_init(riscv_iommu_pmu_init);
+
+MODULE_DESCRIPTION("RISC-V IOMMU PMU");
+MODULE_LICENSE("GPL");
--
2.43.7
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v4 2/2] iommu/riscv: create a auxiliary device for HPM
2026-07-14 8:36 ` Zong Li
@ 2026-07-14 8:36 ` Zong Li
-1 siblings, 0 replies; 14+ messages in thread
From: Zong Li @ 2026-07-14 8:36 UTC (permalink / raw)
To: tjeznach, joro, will, robin.murphy, pjw, palmer, aou, alex,
mark.rutland, andrew.jones, guoren, david.laight.linux,
zhangzhanpeng.jasper, iommu, linux-riscv, linux-kernel,
linux-perf-users
Cc: Zong Li, Samuel Holland
Create an auxiliary device for HPM when the IOMMU supports a
hardware performance monitor.
Use the physical address of the device directory table as the
unique ID for auxiliary device in each IOMMU instance
Reviewed-by: Guo Ren <guoren@kernel.org>
Suggested-by: Samuel Holland <samuel.holland@sifive.com>
Signed-off-by: Zong Li <zong.li@sifive.com>
---
drivers/iommu/riscv/Kconfig | 1 +
drivers/iommu/riscv/iommu.c | 19 +++++++++++++++++++
2 files changed, 20 insertions(+)
diff --git a/drivers/iommu/riscv/Kconfig b/drivers/iommu/riscv/Kconfig
index b86e5ab94183..8025bf0fb67f 100644
--- a/drivers/iommu/riscv/Kconfig
+++ b/drivers/iommu/riscv/Kconfig
@@ -10,6 +10,7 @@ config RISCV_IOMMU
select GENERIC_PT
select IOMMU_PT
select IOMMU_PT_RISCV64
+ select AUXILIARY_BUS
help
Support for implementations of the RISC-V IOMMU architecture that
complements the RISC-V MMU capabilities, providing similar address
diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c
index cec3ddd7ab10..186c22c3ea6c 100644
--- a/drivers/iommu/riscv/iommu.c
+++ b/drivers/iommu/riscv/iommu.c
@@ -14,6 +14,7 @@
#include <linux/acpi.h>
#include <linux/acpi_rimt.h>
+#include <linux/auxiliary_bus.h>
#include <linux/compiler.h>
#include <linux/crash_dump.h>
#include <linux/init.h>
@@ -565,6 +566,21 @@ static irqreturn_t riscv_iommu_fltq_process(int irq, void *data)
return IRQ_HANDLED;
}
+/*
+ * IOMMU Hardware performance monitor
+ */
+static int riscv_iommu_hpm_enable(struct riscv_iommu_device *iommu)
+{
+ struct auxiliary_device *auxdev;
+
+ auxdev = __devm_auxiliary_device_create(iommu->dev, KBUILD_MODNAME,
+ "pmu", iommu, iommu->ddt_phys);
+ if (!auxdev)
+ return -ENODEV;
+
+ return 0;
+}
+
/* Lookup and initialize device context info structure. */
static struct riscv_iommu_dc *riscv_iommu_get_dc(struct riscv_iommu_device *iommu,
unsigned int devid)
@@ -1613,6 +1629,9 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu)
goto err_remove_sysfs;
}
+ if (iommu->caps & RISCV_IOMMU_CAPABILITIES_HPM)
+ riscv_iommu_hpm_enable(iommu);
+
return 0;
err_remove_sysfs:
--
2.43.7
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v4 2/2] iommu/riscv: create a auxiliary device for HPM
@ 2026-07-14 8:36 ` Zong Li
0 siblings, 0 replies; 14+ messages in thread
From: Zong Li @ 2026-07-14 8:36 UTC (permalink / raw)
To: tjeznach, joro, will, robin.murphy, pjw, palmer, aou, alex,
mark.rutland, andrew.jones, guoren, david.laight.linux,
zhangzhanpeng.jasper, iommu, linux-riscv, linux-kernel,
linux-perf-users
Cc: Zong Li, Samuel Holland
Create an auxiliary device for HPM when the IOMMU supports a
hardware performance monitor.
Use the physical address of the device directory table as the
unique ID for auxiliary device in each IOMMU instance
Reviewed-by: Guo Ren <guoren@kernel.org>
Suggested-by: Samuel Holland <samuel.holland@sifive.com>
Signed-off-by: Zong Li <zong.li@sifive.com>
---
drivers/iommu/riscv/Kconfig | 1 +
drivers/iommu/riscv/iommu.c | 19 +++++++++++++++++++
2 files changed, 20 insertions(+)
diff --git a/drivers/iommu/riscv/Kconfig b/drivers/iommu/riscv/Kconfig
index b86e5ab94183..8025bf0fb67f 100644
--- a/drivers/iommu/riscv/Kconfig
+++ b/drivers/iommu/riscv/Kconfig
@@ -10,6 +10,7 @@ config RISCV_IOMMU
select GENERIC_PT
select IOMMU_PT
select IOMMU_PT_RISCV64
+ select AUXILIARY_BUS
help
Support for implementations of the RISC-V IOMMU architecture that
complements the RISC-V MMU capabilities, providing similar address
diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c
index cec3ddd7ab10..186c22c3ea6c 100644
--- a/drivers/iommu/riscv/iommu.c
+++ b/drivers/iommu/riscv/iommu.c
@@ -14,6 +14,7 @@
#include <linux/acpi.h>
#include <linux/acpi_rimt.h>
+#include <linux/auxiliary_bus.h>
#include <linux/compiler.h>
#include <linux/crash_dump.h>
#include <linux/init.h>
@@ -565,6 +566,21 @@ static irqreturn_t riscv_iommu_fltq_process(int irq, void *data)
return IRQ_HANDLED;
}
+/*
+ * IOMMU Hardware performance monitor
+ */
+static int riscv_iommu_hpm_enable(struct riscv_iommu_device *iommu)
+{
+ struct auxiliary_device *auxdev;
+
+ auxdev = __devm_auxiliary_device_create(iommu->dev, KBUILD_MODNAME,
+ "pmu", iommu, iommu->ddt_phys);
+ if (!auxdev)
+ return -ENODEV;
+
+ return 0;
+}
+
/* Lookup and initialize device context info structure. */
static struct riscv_iommu_dc *riscv_iommu_get_dc(struct riscv_iommu_device *iommu,
unsigned int devid)
@@ -1613,6 +1629,9 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu)
goto err_remove_sysfs;
}
+ if (iommu->caps & RISCV_IOMMU_CAPABILITIES_HPM)
+ riscv_iommu_hpm_enable(iommu);
+
return 0;
err_remove_sysfs:
--
2.43.7
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCH v4 1/2] drivers/perf: riscv-iommu: add risc-v iommu pmu driver
2026-07-14 8:36 ` Zong Li
(?)
@ 2026-07-14 8:53 ` sashiko-bot
-1 siblings, 0 replies; 14+ messages in thread
From: sashiko-bot @ 2026-07-14 8:53 UTC (permalink / raw)
To: Zong Li; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 8 potential issue(s) to consider:
- [High] Missing `.remove` callback in the auxiliary driver and incorrect `devm` usage tie resources to the wrong lifecycles, causing Use-After-Free on device unbind.
- [High] The PMU IRQ handler fails to properly clear the interrupt source in multiple edge cases, leading to a permanent PMU lockup or an infinite interrupt storm.
- [High] The stateful counter read optimization in `riscv_iommu_pmu_get_counter` overwrites its state, causing subsequent hardware wraparounds to be completely missed.
- [High] The interrupt handler blindly restarts all allocated counters, incorrectly unpausing events that were explicitly stopped by the perf core.
- [High] Stack buffer overflow on 32-bit systems due to writing a 64-bit value into a 32-bit bitmap array.
- [Medium] Group validation logic in `riscv_iommu_pmu_event_init` fails to enforce hardware constraints and allows invalid cross-PMU combinations.
- [Medium] Wrapping expected `irq_set_affinity` failures in `WARN_ON` causes unnecessary kernel stack traces.
- [Medium] Incorrect determination of `num_counters` using `hweight32` makes the highest counter inaccessible if lower counters are omitted.
--
--- Patch [1]: [PATCH v4 1/2] drivers/perf: riscv-iommu: add risc-v iommu pmu driver ---
Note: The format of this report has been altered to provide a free-form summary
due to system recitation restrictions. No original patch code is quoted.
commit f8f3c1f0f8ae563726a542519ac8eef772529601
Author: Zong Li <zong.li@sifive.com>
Subject: drivers/perf: riscv-iommu: add risc-v iommu pmu driver
This commit adds a new auxiliary device driver to support the risc-v iommu pmu.
It implements the performance monitoring counters, including hardware detection,
filtering support, and event configuration.
Link: https://lore.kernel.org/linux-riscv/20260618143634.7f3dd6c5@pumpkin/
[Severity: High]
Does the iommu_pmu_driver auxiliary driver leak resources or cause a
use-after-free on unbind?
The driver definition is missing a remove callback to unregister the pmu and
remove the cpu hotplug instance. Furthermore, riscv_iommu_pmu_request_irq()
binds the interrupt using devm_request_irq() against the parent iommu device
rather than the auxiliary device. If a user unbinds the pmu driver via sysfs,
the struct will be freed, but the interrupt and hotplug callbacks will remain
active with dangling pointers.
[Severity: High]
Will the pmu interrupt handler cause an interrupt storm or permanent lockup on
edge cases?
In riscv_iommu_pmu_irq_handler(), if the overflow register reads as zero, the
function returns early without clearing the interrupt pending bit in the status
register and without restarting the counters. Similarly, if an event is missing
from the array, the loop uses continue and skips clearing the hardware overflow
bit. Both cases leave the interrupt pending indefinitely.
[Severity: Medium]
Does the event initialization logic properly validate groups?
In riscv_iommu_pmu_event_init(), when iterating over siblings in a group, the
code explicitly skips checking if the sibling's pmu matches the event's pmu
when the sibling is a cycle event. Additionally, the loop counts the siblings
but seems to miss accounting for the new event itself when checking against
total_event_counters, which could allow exceeding hardware limits or creating
invalid cross-pmu groups.
[Severity: Medium]
Is it intentional to trigger a kernel warning for expected affinity failures?
In both riscv_iommu_pmu_probe() and riscv_iommu_pmu_offline_cpu(), the calls
to irq_set_affinity() are wrapped in a warning macro. On systems without
multiprocessor support or with interrupt controllers that do not support
affinity routing, this function safely returns an error. Wrapping this in a
warning causes unnecessary stack traces for valid hardware configurations.
[Severity: High]
Can the counter read optimization miss hardware wraparounds?
In riscv_iommu_pmu_get_counter(), when a wraparound is detected, the code sets
the previous low value to a sentinel of all ones to force the slow path on the
next read. However, this sentinel value is immediately overwritten
unconditionally by the actual low value read, which is zero. If another wrap
occurs before the next read, the check for whether the new low value is less
than or equal to the previous low value will fail, and the wraparound will be
missed.
[Severity: Medium]
Does using hweight32() to determine the number of counters cause issues with
sparse counter implementations?
In riscv_iommu_pmu_probe(), the number of counters is derived by counting the
set bits in the inhibit register using hweight32(). If the hardware omits
some lower counters but implements higher ones, hweight32() returns a count
that is strictly less than the highest implemented index. This means the bitmap
operations will use a bound that cuts off the highest available counters,
leaving them inaccessible. Should this use fls() on the bitmask instead?
[Severity: High]
Does the interrupt handler accidentally unpause stopped events?
In riscv_iommu_pmu_start_all(), the code clears the inhibit bits for every
counter that is marked in used_counters. However, used_counters only tracks
whether a counter has been allocated to an event, not whether that event is
currently active or explicitly stopped by the perf core. If an event was
stopped for throttling or a context switch, an interrupt from another event
will cause the stopped event to unconditionally resume counting.
[Severity: High]
Can the overflow bitmap trigger a stack buffer overflow on 32-bit architectures?
In riscv_iommu_pmu_irq_handler(), ovf_bitmap is declared with a size of 32
bits. On 32-bit systems, this allocates exactly 4 bytes on the stack. The
function then calls bitmap_from_u64() to copy the overflow register into it,
which unconditionally writes 64 bits (8 bytes), overwriting 4 bytes of adjacent
stack memory.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260714083625.1083606-1-zong.li@sifive.com?part=1
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v4 2/2] iommu/riscv: create a auxiliary device for HPM
2026-07-14 8:36 ` Zong Li
(?)
@ 2026-07-14 9:03 ` sashiko-bot
-1 siblings, 0 replies; 14+ messages in thread
From: sashiko-bot @ 2026-07-14 9:03 UTC (permalink / raw)
To: Zong Li; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] Data truncation, negative sysfs ID, and information leak when passing physical address to auxiliary device ID.
- [High] Teardown asymmetry causing Use-After-Disable and guaranteed Use-After-Free of PMU structure.
- [Medium] Overly generic auxiliary device namespace risking collisions.
--
--- Patch [2]: [PATCH v4 2/2] iommu/riscv: create a auxiliary device for HPM ---
commit b51f5eeba53f86d299b4eda15a076df1b60764b8
Author: Zong Li <zong.li@sifive.com>
iommu/riscv: create a auxiliary device for HPM
This commit creates an auxiliary device for the hardware performance monitor
when supported by the IOMMU. It uses the physical address of the device
directory table as the unique ID for the auxiliary device.
> diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c
> --- a/drivers/iommu/riscv/iommu.c
> +++ b/drivers/iommu/riscv/iommu.c
[ ... ]
> @@ -565,6 +566,21 @@ static irqreturn_t riscv_iommu_fltq_process(int irq, void *data)
> return IRQ_HANDLED;
> }
>
> +/*
> + * IOMMU Hardware performance monitor
> + */
> +static int riscv_iommu_hpm_enable(struct riscv_iommu_device *iommu)
> +{
> + struct auxiliary_device *auxdev;
> +
> + auxdev = __devm_auxiliary_device_create(iommu->dev, KBUILD_MODNAME,
> + "pmu", iommu, iommu->ddt_phys);
[Severity: High]
Will this cause truncation and potentially negative sysfs IDs?
The fifth argument to __devm_auxiliary_device_create() is an int id, but we
are passing iommu->ddt_phys which is a 64-bit dma_addr_t.
If bit 31 of the physical address is set (which is common for RAM on RISC-V),
the truncated integer becomes negative, resulting in malformed sysfs
directories.
Could this also cause sysfs naming collisions if two IOMMUs share the same
lower 32-bits of their physical address?
[Severity: Medium]
Could KBUILD_MODNAME result in an overly generic namespace for the auxiliary
bus?
Since this source file is compiled as iommu.o, KBUILD_MODNAME evaluates to
"iommu", creating an auxiliary device named "iommu.pmu.<id>".
If another IOMMU driver also registers a "pmu" auxiliary device using
KBUILD_MODNAME, would they collide on the bus, or could the riscv_iommu_pmu
driver incorrectly bind to the wrong device in a multi-arch build?
[Severity: High]
Does using a devm-managed allocation here create a teardown asymmetry with
the manual hardware shutdown in the remove path?
Because devm cleanup executes after the driver's remove callback returns,
the auxiliary device will remain active while the IOMMU hardware is disabled:
riscv_iommu_remove()
...
riscv_iommu_iodir_set_mode(iommu, RISCV_IOMMU_DDTP_IOMMU_MODE_OFF);
...
<driver remove returns>
<devm cleanup destroys the auxiliary device>
Does this leave a window where an active devm-managed PMU interrupt could fire
and attempt to access disabled hardware?
> + if (!auxdev)
> + return -ENODEV;
> +
> + return 0;
> +}
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260714083625.1083606-1-zong.li@sifive.com?part=2
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v4 1/2] drivers/perf: riscv-iommu: add risc-v iommu pmu driver
2026-07-14 8:36 ` Zong Li
@ 2026-07-14 11:22 ` Guo Ren
-1 siblings, 0 replies; 14+ messages in thread
From: Guo Ren @ 2026-07-14 11:22 UTC (permalink / raw)
To: Zong Li
Cc: tjeznach, joro, will, robin.murphy, pjw, palmer, aou, alex,
mark.rutland, andrew.jones, david.laight.linux,
zhangzhanpeng.jasper, iommu, linux-riscv, linux-kernel,
linux-perf-users
On Tue, Jul 14, 2026 at 4:36 PM Zong Li <zong.li@sifive.com> wrote:
>
> Add a new driver to support the RISC-V IOMMU PMU. This is an auxiliary
> device driver created by the parent RISC-V IOMMU driver.
>
> The RISC-V IOMMU PMU separates the cycle counter from the event counters.
> The cycle counter is not associated with iohpmevt0, so a software-defined
> cycle event is required for the perf subsystem.
>
> The number and width of the counters are hardware-implemented and must
> be detected at runtime.
>
> The performance monitor provides counters with filtering support to
> collect events for specific device ID/process ID, or GSCID/PSCID.
>
> According to RISC-V IOMMU specification Chapter 6:
> Whether an 8 byte access to an IOMMU register is single-copy atomic is
> UNSPECIFIED. Use two separate 4 byte accesses for hardware
> compatibility.
>
> PMU-related definitions are moved into the perf driver, where they are
> used exclusively.
>
> Suggested-by: David Laight <david.laight.linux@gmail.com>
> Suggested-by: Guo Ren <guoren@kernel.org>
> Link: https://lore.kernel.org/linux-riscv/20260618143634.7f3dd6c5@pumpkin/
> Signed-off-by: Zong Li <zong.li@sifive.com>
> ---
> drivers/iommu/riscv/iommu-bits.h | 61 ---
> drivers/perf/Kconfig | 12 +
> drivers/perf/Makefile | 1 +
> drivers/perf/riscv_iommu_pmu.c | 722 +++++++++++++++++++++++++++++++
> 4 files changed, 735 insertions(+), 61 deletions(-)
> create mode 100644 drivers/perf/riscv_iommu_pmu.c
>
> diff --git a/drivers/iommu/riscv/iommu-bits.h b/drivers/iommu/riscv/iommu-bits.h
> index f2ef9bd3cde9..6b5de913a032 100644
> --- a/drivers/iommu/riscv/iommu-bits.h
> +++ b/drivers/iommu/riscv/iommu-bits.h
> @@ -192,67 +192,6 @@ enum riscv_iommu_ddtp_modes {
> #define RISCV_IOMMU_IPSR_PMIP BIT(RISCV_IOMMU_INTR_PM)
> #define RISCV_IOMMU_IPSR_PIP BIT(RISCV_IOMMU_INTR_PQ)
>
> -/* 5.19 Performance monitoring counter overflow status (32bits) */
> -#define RISCV_IOMMU_REG_IOCOUNTOVF 0x0058
> -#define RISCV_IOMMU_IOCOUNTOVF_CY BIT(0)
> -#define RISCV_IOMMU_IOCOUNTOVF_HPM GENMASK_ULL(31, 1)
> -
> -/* 5.20 Performance monitoring counter inhibits (32bits) */
> -#define RISCV_IOMMU_REG_IOCOUNTINH 0x005C
> -#define RISCV_IOMMU_IOCOUNTINH_CY BIT(0)
> -#define RISCV_IOMMU_IOCOUNTINH_HPM GENMASK(31, 1)
> -
> -/* 5.21 Performance monitoring cycles counter (64bits) */
> -#define RISCV_IOMMU_REG_IOHPMCYCLES 0x0060
> -#define RISCV_IOMMU_IOHPMCYCLES_COUNTER GENMASK_ULL(62, 0)
> -#define RISCV_IOMMU_IOHPMCYCLES_OF BIT_ULL(63)
> -
> -/* 5.22 Performance monitoring event counters (31 * 64bits) */
> -#define RISCV_IOMMU_REG_IOHPMCTR_BASE 0x0068
> -#define RISCV_IOMMU_REG_IOHPMCTR(_n) (RISCV_IOMMU_REG_IOHPMCTR_BASE + ((_n) * 0x8))
> -
> -/* 5.23 Performance monitoring event selectors (31 * 64bits) */
> -#define RISCV_IOMMU_REG_IOHPMEVT_BASE 0x0160
> -#define RISCV_IOMMU_REG_IOHPMEVT(_n) (RISCV_IOMMU_REG_IOHPMEVT_BASE + ((_n) * 0x8))
> -#define RISCV_IOMMU_IOHPMEVT_EVENTID GENMASK_ULL(14, 0)
> -#define RISCV_IOMMU_IOHPMEVT_DMASK BIT_ULL(15)
> -#define RISCV_IOMMU_IOHPMEVT_PID_PSCID GENMASK_ULL(35, 16)
> -#define RISCV_IOMMU_IOHPMEVT_DID_GSCID GENMASK_ULL(59, 36)
> -#define RISCV_IOMMU_IOHPMEVT_PV_PSCV BIT_ULL(60)
> -#define RISCV_IOMMU_IOHPMEVT_DV_GSCV BIT_ULL(61)
> -#define RISCV_IOMMU_IOHPMEVT_IDT BIT_ULL(62)
> -#define RISCV_IOMMU_IOHPMEVT_OF BIT_ULL(63)
> -
> -/* Number of defined performance-monitoring event selectors */
> -#define RISCV_IOMMU_IOHPMEVT_CNT 31
> -
> -/**
> - * enum riscv_iommu_hpmevent_id - Performance-monitoring event identifier
> - *
> - * @RISCV_IOMMU_HPMEVENT_INVALID: Invalid event, do not count
> - * @RISCV_IOMMU_HPMEVENT_URQ: Untranslated requests
> - * @RISCV_IOMMU_HPMEVENT_TRQ: Translated requests
> - * @RISCV_IOMMU_HPMEVENT_ATS_RQ: ATS translation requests
> - * @RISCV_IOMMU_HPMEVENT_TLB_MISS: TLB misses
> - * @RISCV_IOMMU_HPMEVENT_DD_WALK: Device directory walks
> - * @RISCV_IOMMU_HPMEVENT_PD_WALK: Process directory walks
> - * @RISCV_IOMMU_HPMEVENT_S_VS_WALKS: First-stage page table walks
> - * @RISCV_IOMMU_HPMEVENT_G_WALKS: Second-stage page table walks
> - * @RISCV_IOMMU_HPMEVENT_MAX: Value to denote maximum Event IDs
> - */
> -enum riscv_iommu_hpmevent_id {
> - RISCV_IOMMU_HPMEVENT_INVALID = 0,
> - RISCV_IOMMU_HPMEVENT_URQ = 1,
> - RISCV_IOMMU_HPMEVENT_TRQ = 2,
> - RISCV_IOMMU_HPMEVENT_ATS_RQ = 3,
> - RISCV_IOMMU_HPMEVENT_TLB_MISS = 4,
> - RISCV_IOMMU_HPMEVENT_DD_WALK = 5,
> - RISCV_IOMMU_HPMEVENT_PD_WALK = 6,
> - RISCV_IOMMU_HPMEVENT_S_VS_WALKS = 7,
> - RISCV_IOMMU_HPMEVENT_G_WALKS = 8,
> - RISCV_IOMMU_HPMEVENT_MAX = 9
> -};
> -
> /* 5.24 Translation request IOVA (64bits) */
> #define RISCV_IOMMU_REG_TR_REQ_IOVA 0x0258
> #define RISCV_IOMMU_TR_REQ_IOVA_VPN GENMASK_ULL(63, 12)
> diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig
> index 245e7bb763b9..8cce6c2ea626 100644
> --- a/drivers/perf/Kconfig
> +++ b/drivers/perf/Kconfig
> @@ -105,6 +105,18 @@ config RISCV_PMU_SBI
> full perf feature support i.e. counter overflow, privilege mode
> filtering, counter configuration.
>
> +config RISCV_IOMMU_PMU
> + depends on RISCV || COMPILE_TEST
> + depends on RISCV_IOMMU
> + bool "RISC-V IOMMU Hardware Performance Monitor"
> + default y
> + help
> + Say Y if you want to use the RISC-V IOMMU performance monitor
> + implementation. The performance monitor is an optional hardware
> + feature, and whether it is actually enabled depends on IOMMU
> + hardware support. If the underlying hardware does not implement
> + the PMU, this option will have no effect.
> +
> config STARFIVE_STARLINK_PMU
> depends on ARCH_STARFIVE || COMPILE_TEST
> depends on 64BIT
> diff --git a/drivers/perf/Makefile b/drivers/perf/Makefile
> index eb8a022dad9a..90c75f3c0ac1 100644
> --- a/drivers/perf/Makefile
> +++ b/drivers/perf/Makefile
> @@ -20,6 +20,7 @@ obj-$(CONFIG_QCOM_L3_PMU) += qcom_l3_pmu.o
> obj-$(CONFIG_RISCV_PMU) += riscv_pmu.o
> obj-$(CONFIG_RISCV_PMU_LEGACY) += riscv_pmu_legacy.o
> obj-$(CONFIG_RISCV_PMU_SBI) += riscv_pmu_sbi.o
> +obj-$(CONFIG_RISCV_IOMMU_PMU) += riscv_iommu_pmu.o
> obj-$(CONFIG_STARFIVE_STARLINK_PMU) += starfive_starlink_pmu.o
> obj-$(CONFIG_THUNDERX2_PMU) += thunderx2_pmu.o
> obj-$(CONFIG_XGENE_PMU) += xgene_pmu.o
> diff --git a/drivers/perf/riscv_iommu_pmu.c b/drivers/perf/riscv_iommu_pmu.c
> new file mode 100644
> index 000000000000..eb331d4633e1
> --- /dev/null
> +++ b/drivers/perf/riscv_iommu_pmu.c
> @@ -0,0 +1,722 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Copyright (C) 2026 SiFive
> + *
> + * Authors
> + * Zong Li <zong.li@sifive.com>
> + */
> +
> +#include <linux/auxiliary_bus.h>
> +#include <linux/io-64-nonatomic-hi-lo.h>
> +#include <linux/perf_event.h>
> +
> +#include "../iommu/riscv/iommu.h"
> +
> +/* 5.19 Performance monitoring counter overflow status (32bits) */
> +#define RISCV_IOMMU_REG_IOCOUNTOVF 0x0058
> +#define RISCV_IOMMU_IOCOUNTOVF_CY BIT(0)
> +#define RISCV_IOMMU_IOCOUNTOVF_HPM GENMASK_ULL(31, 1)
> +
> +/* 5.20 Performance monitoring counter inhibits (32bits) */
> +#define RISCV_IOMMU_REG_IOCOUNTINH 0x005C
> +#define RISCV_IOMMU_IOCOUNTINH_CY BIT(0)
> +#define RISCV_IOMMU_IOCOUNTINH_HPM GENMASK(31, 0)
> +
> +/* 5.21 Performance monitoring cycles counter (64bits) */
> +#define RISCV_IOMMU_REG_IOHPMCYCLES 0x0060
> +#define RISCV_IOMMU_IOHPMCYCLES_COUNTER GENMASK_ULL(62, 0)
> +#define RISCV_IOMMU_IOHPMCYCLES_OF BIT_ULL(63)
> +#define RISCV_IOMMU_REG_IOHPMCTR(_n) (RISCV_IOMMU_REG_IOHPMCYCLES + ((_n) * 0x8))
> +
> +/* 5.22 Performance monitoring event counters (31 * 64bits) */
> +#define RISCV_IOMMU_REG_IOHPMCTR_BASE 0x0068
> +#define RISCV_IOMMU_IOHPMCTR_COUNTER GENMASK_ULL(63, 0)
> +
> +/* 5.23 Performance monitoring event selectors (31 * 64bits) */
> +#define RISCV_IOMMU_REG_IOHPMEVT_BASE 0x0160
> +#define RISCV_IOMMU_REG_IOHPMEVT(_n) (RISCV_IOMMU_REG_IOHPMEVT_BASE + ((_n) * 0x8))
> +#define RISCV_IOMMU_IOHPMEVT_EVENTID GENMASK_ULL(14, 0)
> +#define RISCV_IOMMU_IOHPMEVT_DMASK BIT_ULL(15)
> +#define RISCV_IOMMU_IOHPMEVT_PID_PSCID GENMASK_ULL(35, 16)
> +#define RISCV_IOMMU_IOHPMEVT_DID_GSCID GENMASK_ULL(59, 36)
> +#define RISCV_IOMMU_IOHPMEVT_PV_PSCV BIT_ULL(60)
> +#define RISCV_IOMMU_IOHPMEVT_DV_GSCV BIT_ULL(61)
> +#define RISCV_IOMMU_IOHPMEVT_IDT BIT_ULL(62)
> +#define RISCV_IOMMU_IOHPMEVT_OF BIT_ULL(63)
> +#define RISCV_IOMMU_IOHPMEVT_EVENT GENMASK_ULL(62, 0)
> +
> +/* The total number of counters is 31 event counters plus 1 cycle counter */
> +#define RISCV_IOMMU_HPM_COUNTER_NUM 32
> +
> +static int cpuhp_state;
> +
> +/**
> + * enum riscv_iommu_hpmevent_id - Performance-monitoring event identifier
> + *
> + * @RISCV_IOMMU_HPMEVENT_CYCLE: Clock cycle counter
> + * @RISCV_IOMMU_HPMEVENT_URQ: Untranslated requests
> + * @RISCV_IOMMU_HPMEVENT_TRQ: Translated requests
> + * @RISCV_IOMMU_HPMEVENT_ATS_RQ: ATS translation requests
> + * @RISCV_IOMMU_HPMEVENT_TLB_MISS: TLB misses
> + * @RISCV_IOMMU_HPMEVENT_DD_WALK: Device directory walks
> + * @RISCV_IOMMU_HPMEVENT_PD_WALK: Process directory walks
> + * @RISCV_IOMMU_HPMEVENT_S_VS_WALKS: First-stage page table walks
> + * @RISCV_IOMMU_HPMEVENT_G_WALKS: Second-stage page table walks
> + * @RISCV_IOMMU_HPMEVENT_MAX: Value to denote maximum Event IDs
> + *
> + * The specification does not define an event ID for counting the
> + * number of clock cycles, meaning there is no associated 'iohpmevt0'.
> + * Event ID 0 is an invalid event and does not overlap with any valid
> + * event ID. Let's repurpose ID 0 as the cycle for perf, the cycle
> + * event is not actually written into any register, it serves solely
> + * as an identifier.
> + */
> +enum riscv_iommu_hpmevent_id {
> + RISCV_IOMMU_HPMEVENT_CYCLE = 0,
> + RISCV_IOMMU_HPMEVENT_URQ = 1,
> + RISCV_IOMMU_HPMEVENT_TRQ = 2,
> + RISCV_IOMMU_HPMEVENT_ATS_RQ = 3,
> + RISCV_IOMMU_HPMEVENT_TLB_MISS = 4,
> + RISCV_IOMMU_HPMEVENT_DD_WALK = 5,
> + RISCV_IOMMU_HPMEVENT_PD_WALK = 6,
> + RISCV_IOMMU_HPMEVENT_S_VS_WALKS = 7,
> + RISCV_IOMMU_HPMEVENT_G_WALKS = 8,
> + RISCV_IOMMU_HPMEVENT_MAX = 9
> +};
> +
> +struct riscv_iommu_pmu {
> + struct pmu pmu;
> + struct hlist_node node;
> + void __iomem *reg;
> + unsigned int on_cpu;
> + unsigned int irq;
> + int num_counters;
> + u64 cycle_cntr_mask;
> + u64 event_cntr_mask;
> + struct perf_event *events[RISCV_IOMMU_HPM_COUNTER_NUM];
> + DECLARE_BITMAP(used_counters, RISCV_IOMMU_HPM_COUNTER_NUM);
> + u32 hi_prev[RISCV_IOMMU_HPM_COUNTER_NUM];
> + u32 lo_prev[RISCV_IOMMU_HPM_COUNTER_NUM];
> +};
> +
> +#define to_riscv_iommu_pmu(p) (container_of(p, struct riscv_iommu_pmu, pmu))
> +
> +#define RISCV_IOMMU_PMU_ATTR_EXTRACTOR(_name, _mask) \
> + static inline u32 get_##_name(struct perf_event *event) \
> + { \
> + return FIELD_GET(_mask, event->attr.config); \
> + } \
> +
> +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(event, RISCV_IOMMU_IOHPMEVT_EVENTID);
> +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(partial_matching, RISCV_IOMMU_IOHPMEVT_DMASK);
> +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(pid_pscid, RISCV_IOMMU_IOHPMEVT_PID_PSCID);
> +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(did_gscid, RISCV_IOMMU_IOHPMEVT_DID_GSCID);
> +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_pid_pscid, RISCV_IOMMU_IOHPMEVT_PV_PSCV);
> +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_did_gscid, RISCV_IOMMU_IOHPMEVT_DV_GSCV);
> +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_id_type, RISCV_IOMMU_IOHPMEVT_IDT);
> +
> +/* Formats */
> +PMU_FORMAT_ATTR(event, "config:0-14");
> +PMU_FORMAT_ATTR(partial_matching, "config:15");
> +PMU_FORMAT_ATTR(pid_pscid, "config:16-35");
> +PMU_FORMAT_ATTR(did_gscid, "config:36-59");
> +PMU_FORMAT_ATTR(filter_pid_pscid, "config:60");
> +PMU_FORMAT_ATTR(filter_did_gscid, "config:61");
> +PMU_FORMAT_ATTR(filter_id_type, "config:62");
> +
> +static struct attribute *riscv_iommu_pmu_formats[] = {
> + &format_attr_event.attr,
> + &format_attr_partial_matching.attr,
> + &format_attr_pid_pscid.attr,
> + &format_attr_did_gscid.attr,
> + &format_attr_filter_pid_pscid.attr,
> + &format_attr_filter_did_gscid.attr,
> + &format_attr_filter_id_type.attr,
> + NULL,
> +};
> +
> +static const struct attribute_group riscv_iommu_pmu_format_group = {
> + .name = "format",
> + .attrs = riscv_iommu_pmu_formats,
> +};
> +
> +/* Events */
> +static ssize_t riscv_iommu_pmu_event_show(struct device *dev,
> + struct device_attribute *attr,
> + char *page)
> +{
> + struct perf_pmu_events_attr *pmu_attr;
> +
> + pmu_attr = container_of(attr, struct perf_pmu_events_attr, attr);
> +
> + return sysfs_emit(page, "event=0x%02llx\n", pmu_attr->id);
> +}
> +
> +#define RISCV_IOMMU_PMU_EVENT_ATTR(name, id) \
> + PMU_EVENT_ATTR_ID(name, riscv_iommu_pmu_event_show, id)
> +
> +static struct attribute *riscv_iommu_pmu_events[] = {
> + RISCV_IOMMU_PMU_EVENT_ATTR(cycle, RISCV_IOMMU_HPMEVENT_CYCLE),
> + RISCV_IOMMU_PMU_EVENT_ATTR(untranslated_req, RISCV_IOMMU_HPMEVENT_URQ),
> + RISCV_IOMMU_PMU_EVENT_ATTR(translated_req, RISCV_IOMMU_HPMEVENT_TRQ),
> + RISCV_IOMMU_PMU_EVENT_ATTR(ats_trans_req, RISCV_IOMMU_HPMEVENT_ATS_RQ),
> + RISCV_IOMMU_PMU_EVENT_ATTR(tlb_miss, RISCV_IOMMU_HPMEVENT_TLB_MISS),
> + RISCV_IOMMU_PMU_EVENT_ATTR(ddt_walks, RISCV_IOMMU_HPMEVENT_DD_WALK),
> + RISCV_IOMMU_PMU_EVENT_ATTR(pdt_walks, RISCV_IOMMU_HPMEVENT_PD_WALK),
> + RISCV_IOMMU_PMU_EVENT_ATTR(s_vs_pt_walks, RISCV_IOMMU_HPMEVENT_S_VS_WALKS),
> + RISCV_IOMMU_PMU_EVENT_ATTR(g_pt_walks, RISCV_IOMMU_HPMEVENT_G_WALKS),
> + NULL,
> +};
> +
> +static const struct attribute_group riscv_iommu_pmu_events_group = {
> + .name = "events",
> + .attrs = riscv_iommu_pmu_events,
> +};
> +
> +/* cpumask */
> +static ssize_t riscv_iommu_cpumask_show(struct device *dev,
> + struct device_attribute *attr,
> + char *buf)
> +{
> + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(dev_get_drvdata(dev));
> +
> + return cpumap_print_to_pagebuf(true, buf, cpumask_of(pmu->on_cpu));
> +}
> +
> +static struct device_attribute riscv_iommu_cpumask_attr =
> + __ATTR(cpumask, 0444, riscv_iommu_cpumask_show, NULL);
> +
> +static struct attribute *riscv_iommu_cpumask_attrs[] = {
> + &riscv_iommu_cpumask_attr.attr,
> + NULL
> +};
> +
> +static const struct attribute_group riscv_iommu_pmu_cpumask_group = {
> + .attrs = riscv_iommu_cpumask_attrs,
> +};
> +
> +static const struct attribute_group *riscv_iommu_pmu_attr_grps[] = {
> + &riscv_iommu_pmu_cpumask_group,
> + &riscv_iommu_pmu_format_group,
> + &riscv_iommu_pmu_events_group,
> + NULL,
> +};
> +
> +/*
> + * Register access wrapper
> + *
> + * According to RISC-V IOMMU specification Chapter 6:
> + * A 4 byte access to an IOMMU register must be single-copy atomic.
> + * Whether an 8 byte access to an IOMMU register is single-copy atomic is UNSPECIFIED
> + *
> + * Use two separate 4 byte accesses for hardware compatibility
> + */
> +static u64 riscv_iommu_pmu_readq(void __iomem *addr)
> +{
> + return hi_lo_readq(addr);
> +}
> +
> +static void riscv_iommu_pmu_writeq(u64 value, void __iomem *addr)
> +{
> + hi_lo_writeq(value, addr);
> +}
> +
> +/* PMU Operations */
> +static void riscv_iommu_pmu_set_counter(struct riscv_iommu_pmu *pmu, u32 idx,
> + u64 value)
> +{
> + u64 counter_mask = idx ? pmu->event_cntr_mask : pmu->cycle_cntr_mask;
> +
> + riscv_iommu_pmu_writeq(value & counter_mask, pmu->reg + RISCV_IOMMU_REG_IOHPMCTR(idx));
> +}
> +
> +/*
> + * As stated in the RISC-V IOMMU Specification, Chapter 6:
> + * Whether an 8 byte access to an IOMMU register is single-copy atomic
> + * is UNSPECIFIED, and such an access may appear, internally to the
> + * IOMMU, as if two separate 4 byte accesses - first to the high half
> + * and second to the low half - were performed
> + *
> + * To make sure the driver works correctly on different hardware,
> + * the software will always use two 4-byte access for the counter.
> + *
> + * This function implements the hi-lo-hi pattern to detect and handle
> + * wraparound during the read operation:
> + * 1. Read high half (hi)
> + * 2. Read low half (lo)
> + * 3. Check if low half wrapped or high half changed:
> + * - If lo <= lo_prev: possible wraparound occurred
> + * - If hi != hi_prev: high half changed during read
> + * 4. If wraparound detected, re-read high half and assume low half is 0
> + * 5. Update previous values for next read
> + */
> +static u64 riscv_iommu_pmu_get_counter(struct riscv_iommu_pmu *pmu, u32 idx)
> +{
> + void __iomem *addr = pmu->reg + RISCV_IOMMU_REG_IOHPMCTR(idx);
> + u64 value, counter_mask = idx ? pmu->event_cntr_mask : pmu->cycle_cntr_mask;
> + u32 hi, lo;
> +
> + hi = readl(addr + 4);
> + lo = readl(addr);
> +
> + if (lo <= pmu->lo_prev[idx] || hi != pmu->hi_prev[idx]) {
> + u32 hi_tmp = readl(addr + 4);
> +
> + /*
> + * If hi changes, then hi+1:0 must have happened while
> + * the code was running so it is a safe return value
> + */
> + if (hi_tmp != hi) {
> + hi = hi_tmp;
> + lo = 0;
> + }
> + pmu->lo_prev[idx] = ~0u;
> + pmu->hi_prev[idx] = hi;
> + }
> + pmu->lo_prev[idx] = lo;
> +
> + value = (((u64)hi << 32) | lo) & counter_mask;
> +
> + /* The bit 63 of cycle counter (i.e., idx == 0) is OF bit */
> + return idx ? value : (value & ~RISCV_IOMMU_IOHPMCYCLES_OF);
> +}
> +
> +static bool is_cycle_event(u64 event)
> +{
> + return FIELD_GET(RISCV_IOMMU_IOHPMEVT_EVENTID, event) ==
> + RISCV_IOMMU_HPMEVENT_CYCLE;
> +}
> +
> +static void riscv_iommu_pmu_set_event(struct riscv_iommu_pmu *pmu, u32 idx,
> + u64 value)
> +{
> + /* There is no associtated IOHPMEVT0 for IOHPMCYCLES */
> + if (is_cycle_event(value))
> + return;
> +
> + /* Event counter start from idx 1 */
> + riscv_iommu_pmu_writeq(FIELD_GET(RISCV_IOMMU_IOHPMEVT_EVENT, value),
> + pmu->reg + RISCV_IOMMU_REG_IOHPMEVT(idx - 1));
> +}
> +
> +static void riscv_iommu_pmu_enable_counter(struct riscv_iommu_pmu *pmu, u32 idx)
> +{
> + void __iomem *addr = pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH;
> + u32 value = readl(addr);
> +
> + writel(value & ~BIT(idx), addr);
> +}
> +
> +static void riscv_iommu_pmu_disable_counter(struct riscv_iommu_pmu *pmu, u32 idx)
> +{
> + void __iomem *addr = pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH;
> + u32 value = readl(addr);
> +
> + writel(value | BIT(idx), addr);
> +}
> +
> +static void riscv_iommu_pmu_clear_ovf(struct perf_event *event, u32 idx)
> +{
> + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
> + u64 value;
> +
> + /* Counter is disabled here, making it safe to read and write registers */
> + if (is_cycle_event(get_event(event))) {
> + value = riscv_iommu_pmu_readq(pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES) &
> + ~RISCV_IOMMU_IOHPMCYCLES_OF;
> + riscv_iommu_pmu_writeq(value, pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES);
> + } else {
> + /* Event counter start from idx 1 */
> + value = riscv_iommu_pmu_readq(pmu->reg + RISCV_IOMMU_REG_IOHPMEVT(idx - 1)) &
> + ~RISCV_IOMMU_IOHPMEVT_OF;
> + riscv_iommu_pmu_writeq(value, pmu->reg + RISCV_IOMMU_REG_IOHPMEVT(idx - 1));
> + }
> +}
> +static void riscv_iommu_pmu_start_all(struct riscv_iommu_pmu *pmu)
> +{
> + void __iomem *addr = pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH;
> + u32 used_cntr = 0;
> +
> + /* The performance-monitoring counter inhibits is a 32-bit WARL register */
> + bitmap_to_arr32(&used_cntr, pmu->used_counters, pmu->num_counters);
> +
> + writel(~used_cntr, addr);
> +}
> +
> +static void riscv_iommu_pmu_stop_all(struct riscv_iommu_pmu *pmu)
> +{
> + writel(GENMASK_ULL(pmu->num_counters - 1, 0),
> + pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH);
> +}
> +
> +/* PMU APIs */
> +static void riscv_iommu_pmu_set_period(struct perf_event *event)
> +{
> + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
> + struct hw_perf_event *hwc = &event->hw;
> + u64 counter_mask = hwc->idx ? pmu->event_cntr_mask : pmu->cycle_cntr_mask;
> + u64 period;
> +
> + /*
> + * Limit the maximum period to prevent the counter value
> + * from overtaking the one we are about to program.
> + * In effect we are reducing max_period to account for
> + * interrupt latency (and we are being very conservative).
> + */
> + period = counter_mask >> 1;
> + riscv_iommu_pmu_set_counter(pmu, hwc->idx, period);
> + local64_set(&hwc->prev_count, period);
> +}
> +
> +static int riscv_iommu_pmu_event_init(struct perf_event *event)
> +{
> + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
> + struct hw_perf_event *hwc = &event->hw;
> + struct perf_event *sibling;
> + int total_event_counters = pmu->num_counters - 1;
> + int counters = 0;
> +
> + if (event->attr.type != event->pmu->type)
> + return -ENOENT;
> +
> + if (hwc->sample_period)
> + return -EOPNOTSUPP;
> +
> + if (event->cpu < 0)
> + return -EOPNOTSUPP;
> +
> + event->cpu = pmu->on_cpu;
> +
> + hwc->idx = -1;
> + hwc->config = event->attr.config;
> +
> + if (event->group_leader == event)
> + return 0;
> +
> + if (!is_cycle_event(get_event(event->group_leader)))
> + if (++counters > total_event_counters)
> + return -EINVAL;
> +
> + for_each_sibling_event(sibling, event->group_leader) {
> + if (is_cycle_event(get_event(sibling)) || is_software_event(sibling))
> + continue;
> +
> + if (sibling->pmu != event->pmu)
> + return -EINVAL;
> +
> + if (++counters > total_event_counters)
> + return -EINVAL;
> + }
> +
> + return 0;
> +}
> +
> +static void riscv_iommu_pmu_update(struct perf_event *event)
> +{
> + struct hw_perf_event *hwc = &event->hw;
> + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
> + u64 delta, prev, now;
> + u32 idx = hwc->idx;
> + u64 counter_mask = idx ? pmu->event_cntr_mask : pmu->cycle_cntr_mask;
> +
> + do {
> + prev = local64_read(&hwc->prev_count);
> + now = riscv_iommu_pmu_get_counter(pmu, idx);
> + } while (local64_cmpxchg(&hwc->prev_count, prev, now) != prev);
> +
> + delta = (now - prev) & counter_mask;
> + local64_add(delta, &event->count);
> +}
> +
> +static void riscv_iommu_pmu_start(struct perf_event *event, int flags)
> +{
> + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
> + struct hw_perf_event *hwc = &event->hw;
> +
> + if (WARN_ON_ONCE(!(event->hw.state & PERF_HES_STOPPED)))
> + return;
> +
> + if (flags & PERF_EF_RELOAD)
> + WARN_ON_ONCE(!(event->hw.state & PERF_HES_UPTODATE));
> +
> + hwc->state = 0;
> + riscv_iommu_pmu_set_period(event);
> + riscv_iommu_pmu_set_event(pmu, hwc->idx, hwc->config);
> + riscv_iommu_pmu_enable_counter(pmu, hwc->idx);
> +
> + perf_event_update_userpage(event);
> +}
> +
> +static void riscv_iommu_pmu_stop(struct perf_event *event, int flags)
> +{
> + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
> + struct hw_perf_event *hwc = &event->hw;
> + int idx = hwc->idx;
> +
> + if (hwc->state & PERF_HES_STOPPED)
> + return;
> +
> + riscv_iommu_pmu_disable_counter(pmu, idx);
> +
> + if ((flags & PERF_EF_UPDATE) && !(hwc->state & PERF_HES_UPTODATE))
> + riscv_iommu_pmu_update(event);
> +
> + hwc->state |= PERF_HES_STOPPED | PERF_HES_UPTODATE;
> +}
> +
> +static int riscv_iommu_pmu_add(struct perf_event *event, int flags)
> +{
> + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
> + struct hw_perf_event *hwc = &event->hw;
> + unsigned int num_counters = pmu->num_counters;
> + int idx;
> +
> + /* Reserve index zero for iohpmcycles */
> + if (is_cycle_event(get_event(event)))
> + idx = 0;
> + else
> + idx = find_next_zero_bit(pmu->used_counters, num_counters, 1);
> +
> + /* All event counters or cycle counter are in use */
> + if (idx == num_counters || pmu->events[idx])
> + return -EAGAIN;
> +
> + set_bit(idx, pmu->used_counters);
> +
> + pmu->events[idx] = event;
> + hwc->idx = idx;
> + hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
> + local64_set(&hwc->prev_count, 0);
> +
> + if (flags & PERF_EF_START)
> + riscv_iommu_pmu_start(event, flags);
> +
> + /* Propagate changes to the userspace mapping. */
> + perf_event_update_userpage(event);
> +
> + return 0;
> +}
> +
> +static void riscv_iommu_pmu_read(struct perf_event *event)
> +{
> + riscv_iommu_pmu_update(event);
> +}
> +
> +static void riscv_iommu_pmu_del(struct perf_event *event, int flags)
> +{
> + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
> + struct hw_perf_event *hwc = &event->hw;
> + int idx = hwc->idx;
> +
> + riscv_iommu_pmu_stop(event, PERF_EF_UPDATE);
> + pmu->events[idx] = NULL;
> + clear_bit(idx, pmu->used_counters);
> +
> + perf_event_update_userpage(event);
> +}
> +
> +static int riscv_iommu_pmu_online_cpu(unsigned int cpu, struct hlist_node *node)
> +{
> + struct riscv_iommu_pmu *iommu_pmu;
> +
> + iommu_pmu = hlist_entry_safe(node, struct riscv_iommu_pmu, node);
> +
> + if (iommu_pmu->on_cpu == -1)
> + iommu_pmu->on_cpu = cpu;
> +
> + return 0;
> +}
> +
> +static int riscv_iommu_pmu_offline_cpu(unsigned int cpu, struct hlist_node *node)
> +{
> + struct riscv_iommu_pmu *iommu_pmu;
> + unsigned int target_cpu;
> +
> + iommu_pmu = hlist_entry_safe(node, struct riscv_iommu_pmu, node);
> +
> + if (cpu != iommu_pmu->on_cpu)
> + return 0;
> +
> + iommu_pmu->on_cpu = -1;
> +
> + target_cpu = cpumask_any_but(cpu_online_mask, cpu);
> + if (target_cpu >= nr_cpu_ids)
> + return 0;
> +
> + perf_pmu_migrate_context(&iommu_pmu->pmu, cpu, target_cpu);
> + iommu_pmu->on_cpu = target_cpu;
> + WARN_ON(irq_set_affinity(iommu_pmu->irq, cpumask_of(target_cpu)));
> +
> + return 0;
> +}
> +
> +static irqreturn_t riscv_iommu_pmu_irq_handler(int irq, void *dev_id)
> +{
> + struct riscv_iommu_pmu *pmu = (struct riscv_iommu_pmu *)dev_id;
> + DECLARE_BITMAP(ovf_bitmap, RISCV_IOMMU_HPM_COUNTER_NUM);
> + u32 ovf, idx;
> +
> + /* Check whether this interrupt is for PMU */
> + if (!(readl_relaxed(pmu->reg + RISCV_IOMMU_REG_IPSR) & RISCV_IOMMU_IPSR_PMIP))
> + return IRQ_NONE;
> +
> + /* Process PMU IRQ */
> + riscv_iommu_pmu_stop_all(pmu);
> +
> + ovf = readl(pmu->reg + RISCV_IOMMU_REG_IOCOUNTOVF);
> + if (!ovf)
> + return IRQ_NONE;
> +
> + bitmap_from_u64(ovf_bitmap, ovf);
> + for_each_set_bit(idx, ovf_bitmap, pmu->num_counters) {
> + struct perf_event *event = pmu->events[idx];
> +
> + if (WARN_ON_ONCE(!event))
> + continue;
> +
> + riscv_iommu_pmu_update(event);
> + riscv_iommu_pmu_set_period(event);
> + riscv_iommu_pmu_clear_ovf(event, idx);
> + }
> +
> + /* Clear performance monitoring interrupt pending bit */
> + writel_relaxed(RISCV_IOMMU_IPSR_PMIP, pmu->reg + RISCV_IOMMU_REG_IPSR);
> +
> + riscv_iommu_pmu_start_all(pmu);
> +
> + return IRQ_HANDLED;
> +}
> +
> +static unsigned int riscv_iommu_pmu_get_irq_num(struct riscv_iommu_device *iommu)
> +{
> + /* Reuse ICVEC.CIV mask for all interrupt vectors mapping */
> + int vec = (iommu->icvec >> (RISCV_IOMMU_INTR_PM * 4)) & RISCV_IOMMU_ICVEC_CIV;
> +
> + return iommu->irqs[vec];
> +}
> +
> +static int riscv_iommu_pmu_request_irq(struct riscv_iommu_device *iommu,
> + struct riscv_iommu_pmu *pmu)
> +{
> + return devm_request_irq(iommu->dev, pmu->irq, riscv_iommu_pmu_irq_handler,
> + IRQF_NOBALANCING | IRQF_SHARED, dev_name(iommu->dev), pmu);
> +}
> +
> +static int riscv_iommu_pmu_probe(struct auxiliary_device *auxdev,
> + const struct auxiliary_device_id *id)
> +{
> + struct riscv_iommu_device *iommu_dev = dev_get_platdata(&auxdev->dev);
> + struct riscv_iommu_pmu *iommu_pmu;
> + void __iomem *addr;
> + char *name;
> + int ret;
> +
> + iommu_pmu = devm_kzalloc(&auxdev->dev, sizeof(*iommu_pmu), GFP_KERNEL);
> + if (!iommu_pmu)
> + return -ENOMEM;
> +
> + iommu_pmu->reg = iommu_dev->reg;
> +
> + /* Counter number and width are hardware-implemented. Detect them by write 1s */
> + addr = iommu_pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH;
> + writel(RISCV_IOMMU_IOCOUNTINH_HPM, addr);
> + iommu_pmu->num_counters = hweight32(readl(addr));
> +
> + addr = iommu_pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES;
> + riscv_iommu_pmu_writeq(RISCV_IOMMU_IOHPMCYCLES_COUNTER, addr);
> + iommu_pmu->cycle_cntr_mask = riscv_iommu_pmu_readq(addr);
> +
> + /* Assume the width of all event counters are the same */
> + addr = iommu_pmu->reg + RISCV_IOMMU_REG_IOHPMCTR_BASE;
> + riscv_iommu_pmu_writeq(RISCV_IOMMU_IOHPMCTR_COUNTER, addr);
> + iommu_pmu->event_cntr_mask = riscv_iommu_pmu_readq(addr);
> +
> + iommu_pmu->pmu = (struct pmu) {
> + .module = THIS_MODULE,
> + .parent = &auxdev->dev,
> + .task_ctx_nr = perf_invalid_context,
> + .event_init = riscv_iommu_pmu_event_init,
> + .add = riscv_iommu_pmu_add,
> + .del = riscv_iommu_pmu_del,
> + .start = riscv_iommu_pmu_start,
> + .stop = riscv_iommu_pmu_stop,
> + .read = riscv_iommu_pmu_read,
> + .attr_groups = riscv_iommu_pmu_attr_grps,
> + .capabilities = PERF_PMU_CAP_NO_EXCLUDE,
> + };
> +
> + auxiliary_set_drvdata(auxdev, iommu_pmu);
> +
> + name = devm_kasprintf(&auxdev->dev, GFP_KERNEL,
> + "riscv_iommu_pmu_%s", dev_name(iommu_dev->dev));
> + if (!name) {
> + dev_err(&auxdev->dev, "Failed to create name riscv_iommu_pmu_%s\n",
> + dev_name(iommu_dev->dev));
> + return -ENOMEM;
> + }
> +
> + /* Bind all events to the same cpu context to avoid race enabling */
> + iommu_pmu->on_cpu = raw_smp_processor_id();
> + iommu_pmu->irq = riscv_iommu_pmu_get_irq_num(iommu_dev);
> + WARN_ON(irq_set_affinity(iommu_pmu->irq, cpumask_of(iommu_pmu->on_cpu)));
> +
> + ret = riscv_iommu_pmu_request_irq(iommu_dev, iommu_pmu);
> + if (ret) {
> + dev_err(&auxdev->dev, "Failed to request irq %s: %d\n", name, ret);
> + return ret;
> + }
> +
> + ret = cpuhp_state_add_instance_nocalls(cpuhp_state, &iommu_pmu->node);
> + if (ret) {
> + dev_err(&auxdev->dev, "Failed to register hotplug %s: %d\n", name, ret);
> + return ret;
> + }
> +
> + ret = perf_pmu_register(&iommu_pmu->pmu, name, -1);
> + if (ret) {
> + dev_err(&auxdev->dev, "Failed to registe %s: %d\n", name, ret);
> + goto err_unregister;
> + }
> +
> + dev_info(&auxdev->dev, "%s: Registered with %d counters\n",
> + name, iommu_pmu->num_counters);
> +
> + return 0;
> +
> +err_unregister:
> + cpuhp_state_remove_instance_nocalls(cpuhp_state, &iommu_pmu->node);
> + return ret;
> +}
> +
> +static const struct auxiliary_device_id riscv_iommu_pmu_id_table[] = {
> + { .name = "iommu.pmu" },
> + {}
> +};
> +MODULE_DEVICE_TABLE(auxiliary, riscv_iommu_pmu_id_table);
> +
> +static struct auxiliary_driver iommu_pmu_driver = {
> + .probe = riscv_iommu_pmu_probe,
> + .id_table = riscv_iommu_pmu_id_table,
> +};
> +
> +static int __init riscv_iommu_pmu_init(void)
> +{
> + int ret;
> +
> + cpuhp_state = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN,
> + "perf/riscv/iommu:online",
> + riscv_iommu_pmu_online_cpu,
> + riscv_iommu_pmu_offline_cpu);
> + if (cpuhp_state < 0)
> + return cpuhp_state;
> +
> + ret = auxiliary_driver_register(&iommu_pmu_driver);
> + if (ret)
> + cpuhp_remove_multi_state(cpuhp_state);
> +
> + return ret;
> +}
> +module_init(riscv_iommu_pmu_init);
> +
> +MODULE_DESCRIPTION("RISC-V IOMMU PMU");
> +MODULE_LICENSE("GPL");
> --
> 2.43.7
>
LGTM!
Reviewed-by: Guo Ren (Alibaba DAMO Academy) <guoren@kernel.org>
--
Best Regards
Guo Ren
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v4 1/2] drivers/perf: riscv-iommu: add risc-v iommu pmu driver
@ 2026-07-14 11:22 ` Guo Ren
0 siblings, 0 replies; 14+ messages in thread
From: Guo Ren @ 2026-07-14 11:22 UTC (permalink / raw)
To: Zong Li
Cc: tjeznach, joro, will, robin.murphy, pjw, palmer, aou, alex,
mark.rutland, andrew.jones, david.laight.linux,
zhangzhanpeng.jasper, iommu, linux-riscv, linux-kernel,
linux-perf-users
On Tue, Jul 14, 2026 at 4:36 PM Zong Li <zong.li@sifive.com> wrote:
>
> Add a new driver to support the RISC-V IOMMU PMU. This is an auxiliary
> device driver created by the parent RISC-V IOMMU driver.
>
> The RISC-V IOMMU PMU separates the cycle counter from the event counters.
> The cycle counter is not associated with iohpmevt0, so a software-defined
> cycle event is required for the perf subsystem.
>
> The number and width of the counters are hardware-implemented and must
> be detected at runtime.
>
> The performance monitor provides counters with filtering support to
> collect events for specific device ID/process ID, or GSCID/PSCID.
>
> According to RISC-V IOMMU specification Chapter 6:
> Whether an 8 byte access to an IOMMU register is single-copy atomic is
> UNSPECIFIED. Use two separate 4 byte accesses for hardware
> compatibility.
>
> PMU-related definitions are moved into the perf driver, where they are
> used exclusively.
>
> Suggested-by: David Laight <david.laight.linux@gmail.com>
> Suggested-by: Guo Ren <guoren@kernel.org>
> Link: https://lore.kernel.org/linux-riscv/20260618143634.7f3dd6c5@pumpkin/
> Signed-off-by: Zong Li <zong.li@sifive.com>
> ---
> drivers/iommu/riscv/iommu-bits.h | 61 ---
> drivers/perf/Kconfig | 12 +
> drivers/perf/Makefile | 1 +
> drivers/perf/riscv_iommu_pmu.c | 722 +++++++++++++++++++++++++++++++
> 4 files changed, 735 insertions(+), 61 deletions(-)
> create mode 100644 drivers/perf/riscv_iommu_pmu.c
>
> diff --git a/drivers/iommu/riscv/iommu-bits.h b/drivers/iommu/riscv/iommu-bits.h
> index f2ef9bd3cde9..6b5de913a032 100644
> --- a/drivers/iommu/riscv/iommu-bits.h
> +++ b/drivers/iommu/riscv/iommu-bits.h
> @@ -192,67 +192,6 @@ enum riscv_iommu_ddtp_modes {
> #define RISCV_IOMMU_IPSR_PMIP BIT(RISCV_IOMMU_INTR_PM)
> #define RISCV_IOMMU_IPSR_PIP BIT(RISCV_IOMMU_INTR_PQ)
>
> -/* 5.19 Performance monitoring counter overflow status (32bits) */
> -#define RISCV_IOMMU_REG_IOCOUNTOVF 0x0058
> -#define RISCV_IOMMU_IOCOUNTOVF_CY BIT(0)
> -#define RISCV_IOMMU_IOCOUNTOVF_HPM GENMASK_ULL(31, 1)
> -
> -/* 5.20 Performance monitoring counter inhibits (32bits) */
> -#define RISCV_IOMMU_REG_IOCOUNTINH 0x005C
> -#define RISCV_IOMMU_IOCOUNTINH_CY BIT(0)
> -#define RISCV_IOMMU_IOCOUNTINH_HPM GENMASK(31, 1)
> -
> -/* 5.21 Performance monitoring cycles counter (64bits) */
> -#define RISCV_IOMMU_REG_IOHPMCYCLES 0x0060
> -#define RISCV_IOMMU_IOHPMCYCLES_COUNTER GENMASK_ULL(62, 0)
> -#define RISCV_IOMMU_IOHPMCYCLES_OF BIT_ULL(63)
> -
> -/* 5.22 Performance monitoring event counters (31 * 64bits) */
> -#define RISCV_IOMMU_REG_IOHPMCTR_BASE 0x0068
> -#define RISCV_IOMMU_REG_IOHPMCTR(_n) (RISCV_IOMMU_REG_IOHPMCTR_BASE + ((_n) * 0x8))
> -
> -/* 5.23 Performance monitoring event selectors (31 * 64bits) */
> -#define RISCV_IOMMU_REG_IOHPMEVT_BASE 0x0160
> -#define RISCV_IOMMU_REG_IOHPMEVT(_n) (RISCV_IOMMU_REG_IOHPMEVT_BASE + ((_n) * 0x8))
> -#define RISCV_IOMMU_IOHPMEVT_EVENTID GENMASK_ULL(14, 0)
> -#define RISCV_IOMMU_IOHPMEVT_DMASK BIT_ULL(15)
> -#define RISCV_IOMMU_IOHPMEVT_PID_PSCID GENMASK_ULL(35, 16)
> -#define RISCV_IOMMU_IOHPMEVT_DID_GSCID GENMASK_ULL(59, 36)
> -#define RISCV_IOMMU_IOHPMEVT_PV_PSCV BIT_ULL(60)
> -#define RISCV_IOMMU_IOHPMEVT_DV_GSCV BIT_ULL(61)
> -#define RISCV_IOMMU_IOHPMEVT_IDT BIT_ULL(62)
> -#define RISCV_IOMMU_IOHPMEVT_OF BIT_ULL(63)
> -
> -/* Number of defined performance-monitoring event selectors */
> -#define RISCV_IOMMU_IOHPMEVT_CNT 31
> -
> -/**
> - * enum riscv_iommu_hpmevent_id - Performance-monitoring event identifier
> - *
> - * @RISCV_IOMMU_HPMEVENT_INVALID: Invalid event, do not count
> - * @RISCV_IOMMU_HPMEVENT_URQ: Untranslated requests
> - * @RISCV_IOMMU_HPMEVENT_TRQ: Translated requests
> - * @RISCV_IOMMU_HPMEVENT_ATS_RQ: ATS translation requests
> - * @RISCV_IOMMU_HPMEVENT_TLB_MISS: TLB misses
> - * @RISCV_IOMMU_HPMEVENT_DD_WALK: Device directory walks
> - * @RISCV_IOMMU_HPMEVENT_PD_WALK: Process directory walks
> - * @RISCV_IOMMU_HPMEVENT_S_VS_WALKS: First-stage page table walks
> - * @RISCV_IOMMU_HPMEVENT_G_WALKS: Second-stage page table walks
> - * @RISCV_IOMMU_HPMEVENT_MAX: Value to denote maximum Event IDs
> - */
> -enum riscv_iommu_hpmevent_id {
> - RISCV_IOMMU_HPMEVENT_INVALID = 0,
> - RISCV_IOMMU_HPMEVENT_URQ = 1,
> - RISCV_IOMMU_HPMEVENT_TRQ = 2,
> - RISCV_IOMMU_HPMEVENT_ATS_RQ = 3,
> - RISCV_IOMMU_HPMEVENT_TLB_MISS = 4,
> - RISCV_IOMMU_HPMEVENT_DD_WALK = 5,
> - RISCV_IOMMU_HPMEVENT_PD_WALK = 6,
> - RISCV_IOMMU_HPMEVENT_S_VS_WALKS = 7,
> - RISCV_IOMMU_HPMEVENT_G_WALKS = 8,
> - RISCV_IOMMU_HPMEVENT_MAX = 9
> -};
> -
> /* 5.24 Translation request IOVA (64bits) */
> #define RISCV_IOMMU_REG_TR_REQ_IOVA 0x0258
> #define RISCV_IOMMU_TR_REQ_IOVA_VPN GENMASK_ULL(63, 12)
> diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig
> index 245e7bb763b9..8cce6c2ea626 100644
> --- a/drivers/perf/Kconfig
> +++ b/drivers/perf/Kconfig
> @@ -105,6 +105,18 @@ config RISCV_PMU_SBI
> full perf feature support i.e. counter overflow, privilege mode
> filtering, counter configuration.
>
> +config RISCV_IOMMU_PMU
> + depends on RISCV || COMPILE_TEST
> + depends on RISCV_IOMMU
> + bool "RISC-V IOMMU Hardware Performance Monitor"
> + default y
> + help
> + Say Y if you want to use the RISC-V IOMMU performance monitor
> + implementation. The performance monitor is an optional hardware
> + feature, and whether it is actually enabled depends on IOMMU
> + hardware support. If the underlying hardware does not implement
> + the PMU, this option will have no effect.
> +
> config STARFIVE_STARLINK_PMU
> depends on ARCH_STARFIVE || COMPILE_TEST
> depends on 64BIT
> diff --git a/drivers/perf/Makefile b/drivers/perf/Makefile
> index eb8a022dad9a..90c75f3c0ac1 100644
> --- a/drivers/perf/Makefile
> +++ b/drivers/perf/Makefile
> @@ -20,6 +20,7 @@ obj-$(CONFIG_QCOM_L3_PMU) += qcom_l3_pmu.o
> obj-$(CONFIG_RISCV_PMU) += riscv_pmu.o
> obj-$(CONFIG_RISCV_PMU_LEGACY) += riscv_pmu_legacy.o
> obj-$(CONFIG_RISCV_PMU_SBI) += riscv_pmu_sbi.o
> +obj-$(CONFIG_RISCV_IOMMU_PMU) += riscv_iommu_pmu.o
> obj-$(CONFIG_STARFIVE_STARLINK_PMU) += starfive_starlink_pmu.o
> obj-$(CONFIG_THUNDERX2_PMU) += thunderx2_pmu.o
> obj-$(CONFIG_XGENE_PMU) += xgene_pmu.o
> diff --git a/drivers/perf/riscv_iommu_pmu.c b/drivers/perf/riscv_iommu_pmu.c
> new file mode 100644
> index 000000000000..eb331d4633e1
> --- /dev/null
> +++ b/drivers/perf/riscv_iommu_pmu.c
> @@ -0,0 +1,722 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Copyright (C) 2026 SiFive
> + *
> + * Authors
> + * Zong Li <zong.li@sifive.com>
> + */
> +
> +#include <linux/auxiliary_bus.h>
> +#include <linux/io-64-nonatomic-hi-lo.h>
> +#include <linux/perf_event.h>
> +
> +#include "../iommu/riscv/iommu.h"
> +
> +/* 5.19 Performance monitoring counter overflow status (32bits) */
> +#define RISCV_IOMMU_REG_IOCOUNTOVF 0x0058
> +#define RISCV_IOMMU_IOCOUNTOVF_CY BIT(0)
> +#define RISCV_IOMMU_IOCOUNTOVF_HPM GENMASK_ULL(31, 1)
> +
> +/* 5.20 Performance monitoring counter inhibits (32bits) */
> +#define RISCV_IOMMU_REG_IOCOUNTINH 0x005C
> +#define RISCV_IOMMU_IOCOUNTINH_CY BIT(0)
> +#define RISCV_IOMMU_IOCOUNTINH_HPM GENMASK(31, 0)
> +
> +/* 5.21 Performance monitoring cycles counter (64bits) */
> +#define RISCV_IOMMU_REG_IOHPMCYCLES 0x0060
> +#define RISCV_IOMMU_IOHPMCYCLES_COUNTER GENMASK_ULL(62, 0)
> +#define RISCV_IOMMU_IOHPMCYCLES_OF BIT_ULL(63)
> +#define RISCV_IOMMU_REG_IOHPMCTR(_n) (RISCV_IOMMU_REG_IOHPMCYCLES + ((_n) * 0x8))
> +
> +/* 5.22 Performance monitoring event counters (31 * 64bits) */
> +#define RISCV_IOMMU_REG_IOHPMCTR_BASE 0x0068
> +#define RISCV_IOMMU_IOHPMCTR_COUNTER GENMASK_ULL(63, 0)
> +
> +/* 5.23 Performance monitoring event selectors (31 * 64bits) */
> +#define RISCV_IOMMU_REG_IOHPMEVT_BASE 0x0160
> +#define RISCV_IOMMU_REG_IOHPMEVT(_n) (RISCV_IOMMU_REG_IOHPMEVT_BASE + ((_n) * 0x8))
> +#define RISCV_IOMMU_IOHPMEVT_EVENTID GENMASK_ULL(14, 0)
> +#define RISCV_IOMMU_IOHPMEVT_DMASK BIT_ULL(15)
> +#define RISCV_IOMMU_IOHPMEVT_PID_PSCID GENMASK_ULL(35, 16)
> +#define RISCV_IOMMU_IOHPMEVT_DID_GSCID GENMASK_ULL(59, 36)
> +#define RISCV_IOMMU_IOHPMEVT_PV_PSCV BIT_ULL(60)
> +#define RISCV_IOMMU_IOHPMEVT_DV_GSCV BIT_ULL(61)
> +#define RISCV_IOMMU_IOHPMEVT_IDT BIT_ULL(62)
> +#define RISCV_IOMMU_IOHPMEVT_OF BIT_ULL(63)
> +#define RISCV_IOMMU_IOHPMEVT_EVENT GENMASK_ULL(62, 0)
> +
> +/* The total number of counters is 31 event counters plus 1 cycle counter */
> +#define RISCV_IOMMU_HPM_COUNTER_NUM 32
> +
> +static int cpuhp_state;
> +
> +/**
> + * enum riscv_iommu_hpmevent_id - Performance-monitoring event identifier
> + *
> + * @RISCV_IOMMU_HPMEVENT_CYCLE: Clock cycle counter
> + * @RISCV_IOMMU_HPMEVENT_URQ: Untranslated requests
> + * @RISCV_IOMMU_HPMEVENT_TRQ: Translated requests
> + * @RISCV_IOMMU_HPMEVENT_ATS_RQ: ATS translation requests
> + * @RISCV_IOMMU_HPMEVENT_TLB_MISS: TLB misses
> + * @RISCV_IOMMU_HPMEVENT_DD_WALK: Device directory walks
> + * @RISCV_IOMMU_HPMEVENT_PD_WALK: Process directory walks
> + * @RISCV_IOMMU_HPMEVENT_S_VS_WALKS: First-stage page table walks
> + * @RISCV_IOMMU_HPMEVENT_G_WALKS: Second-stage page table walks
> + * @RISCV_IOMMU_HPMEVENT_MAX: Value to denote maximum Event IDs
> + *
> + * The specification does not define an event ID for counting the
> + * number of clock cycles, meaning there is no associated 'iohpmevt0'.
> + * Event ID 0 is an invalid event and does not overlap with any valid
> + * event ID. Let's repurpose ID 0 as the cycle for perf, the cycle
> + * event is not actually written into any register, it serves solely
> + * as an identifier.
> + */
> +enum riscv_iommu_hpmevent_id {
> + RISCV_IOMMU_HPMEVENT_CYCLE = 0,
> + RISCV_IOMMU_HPMEVENT_URQ = 1,
> + RISCV_IOMMU_HPMEVENT_TRQ = 2,
> + RISCV_IOMMU_HPMEVENT_ATS_RQ = 3,
> + RISCV_IOMMU_HPMEVENT_TLB_MISS = 4,
> + RISCV_IOMMU_HPMEVENT_DD_WALK = 5,
> + RISCV_IOMMU_HPMEVENT_PD_WALK = 6,
> + RISCV_IOMMU_HPMEVENT_S_VS_WALKS = 7,
> + RISCV_IOMMU_HPMEVENT_G_WALKS = 8,
> + RISCV_IOMMU_HPMEVENT_MAX = 9
> +};
> +
> +struct riscv_iommu_pmu {
> + struct pmu pmu;
> + struct hlist_node node;
> + void __iomem *reg;
> + unsigned int on_cpu;
> + unsigned int irq;
> + int num_counters;
> + u64 cycle_cntr_mask;
> + u64 event_cntr_mask;
> + struct perf_event *events[RISCV_IOMMU_HPM_COUNTER_NUM];
> + DECLARE_BITMAP(used_counters, RISCV_IOMMU_HPM_COUNTER_NUM);
> + u32 hi_prev[RISCV_IOMMU_HPM_COUNTER_NUM];
> + u32 lo_prev[RISCV_IOMMU_HPM_COUNTER_NUM];
> +};
> +
> +#define to_riscv_iommu_pmu(p) (container_of(p, struct riscv_iommu_pmu, pmu))
> +
> +#define RISCV_IOMMU_PMU_ATTR_EXTRACTOR(_name, _mask) \
> + static inline u32 get_##_name(struct perf_event *event) \
> + { \
> + return FIELD_GET(_mask, event->attr.config); \
> + } \
> +
> +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(event, RISCV_IOMMU_IOHPMEVT_EVENTID);
> +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(partial_matching, RISCV_IOMMU_IOHPMEVT_DMASK);
> +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(pid_pscid, RISCV_IOMMU_IOHPMEVT_PID_PSCID);
> +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(did_gscid, RISCV_IOMMU_IOHPMEVT_DID_GSCID);
> +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_pid_pscid, RISCV_IOMMU_IOHPMEVT_PV_PSCV);
> +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_did_gscid, RISCV_IOMMU_IOHPMEVT_DV_GSCV);
> +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_id_type, RISCV_IOMMU_IOHPMEVT_IDT);
> +
> +/* Formats */
> +PMU_FORMAT_ATTR(event, "config:0-14");
> +PMU_FORMAT_ATTR(partial_matching, "config:15");
> +PMU_FORMAT_ATTR(pid_pscid, "config:16-35");
> +PMU_FORMAT_ATTR(did_gscid, "config:36-59");
> +PMU_FORMAT_ATTR(filter_pid_pscid, "config:60");
> +PMU_FORMAT_ATTR(filter_did_gscid, "config:61");
> +PMU_FORMAT_ATTR(filter_id_type, "config:62");
> +
> +static struct attribute *riscv_iommu_pmu_formats[] = {
> + &format_attr_event.attr,
> + &format_attr_partial_matching.attr,
> + &format_attr_pid_pscid.attr,
> + &format_attr_did_gscid.attr,
> + &format_attr_filter_pid_pscid.attr,
> + &format_attr_filter_did_gscid.attr,
> + &format_attr_filter_id_type.attr,
> + NULL,
> +};
> +
> +static const struct attribute_group riscv_iommu_pmu_format_group = {
> + .name = "format",
> + .attrs = riscv_iommu_pmu_formats,
> +};
> +
> +/* Events */
> +static ssize_t riscv_iommu_pmu_event_show(struct device *dev,
> + struct device_attribute *attr,
> + char *page)
> +{
> + struct perf_pmu_events_attr *pmu_attr;
> +
> + pmu_attr = container_of(attr, struct perf_pmu_events_attr, attr);
> +
> + return sysfs_emit(page, "event=0x%02llx\n", pmu_attr->id);
> +}
> +
> +#define RISCV_IOMMU_PMU_EVENT_ATTR(name, id) \
> + PMU_EVENT_ATTR_ID(name, riscv_iommu_pmu_event_show, id)
> +
> +static struct attribute *riscv_iommu_pmu_events[] = {
> + RISCV_IOMMU_PMU_EVENT_ATTR(cycle, RISCV_IOMMU_HPMEVENT_CYCLE),
> + RISCV_IOMMU_PMU_EVENT_ATTR(untranslated_req, RISCV_IOMMU_HPMEVENT_URQ),
> + RISCV_IOMMU_PMU_EVENT_ATTR(translated_req, RISCV_IOMMU_HPMEVENT_TRQ),
> + RISCV_IOMMU_PMU_EVENT_ATTR(ats_trans_req, RISCV_IOMMU_HPMEVENT_ATS_RQ),
> + RISCV_IOMMU_PMU_EVENT_ATTR(tlb_miss, RISCV_IOMMU_HPMEVENT_TLB_MISS),
> + RISCV_IOMMU_PMU_EVENT_ATTR(ddt_walks, RISCV_IOMMU_HPMEVENT_DD_WALK),
> + RISCV_IOMMU_PMU_EVENT_ATTR(pdt_walks, RISCV_IOMMU_HPMEVENT_PD_WALK),
> + RISCV_IOMMU_PMU_EVENT_ATTR(s_vs_pt_walks, RISCV_IOMMU_HPMEVENT_S_VS_WALKS),
> + RISCV_IOMMU_PMU_EVENT_ATTR(g_pt_walks, RISCV_IOMMU_HPMEVENT_G_WALKS),
> + NULL,
> +};
> +
> +static const struct attribute_group riscv_iommu_pmu_events_group = {
> + .name = "events",
> + .attrs = riscv_iommu_pmu_events,
> +};
> +
> +/* cpumask */
> +static ssize_t riscv_iommu_cpumask_show(struct device *dev,
> + struct device_attribute *attr,
> + char *buf)
> +{
> + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(dev_get_drvdata(dev));
> +
> + return cpumap_print_to_pagebuf(true, buf, cpumask_of(pmu->on_cpu));
> +}
> +
> +static struct device_attribute riscv_iommu_cpumask_attr =
> + __ATTR(cpumask, 0444, riscv_iommu_cpumask_show, NULL);
> +
> +static struct attribute *riscv_iommu_cpumask_attrs[] = {
> + &riscv_iommu_cpumask_attr.attr,
> + NULL
> +};
> +
> +static const struct attribute_group riscv_iommu_pmu_cpumask_group = {
> + .attrs = riscv_iommu_cpumask_attrs,
> +};
> +
> +static const struct attribute_group *riscv_iommu_pmu_attr_grps[] = {
> + &riscv_iommu_pmu_cpumask_group,
> + &riscv_iommu_pmu_format_group,
> + &riscv_iommu_pmu_events_group,
> + NULL,
> +};
> +
> +/*
> + * Register access wrapper
> + *
> + * According to RISC-V IOMMU specification Chapter 6:
> + * A 4 byte access to an IOMMU register must be single-copy atomic.
> + * Whether an 8 byte access to an IOMMU register is single-copy atomic is UNSPECIFIED
> + *
> + * Use two separate 4 byte accesses for hardware compatibility
> + */
> +static u64 riscv_iommu_pmu_readq(void __iomem *addr)
> +{
> + return hi_lo_readq(addr);
> +}
> +
> +static void riscv_iommu_pmu_writeq(u64 value, void __iomem *addr)
> +{
> + hi_lo_writeq(value, addr);
> +}
> +
> +/* PMU Operations */
> +static void riscv_iommu_pmu_set_counter(struct riscv_iommu_pmu *pmu, u32 idx,
> + u64 value)
> +{
> + u64 counter_mask = idx ? pmu->event_cntr_mask : pmu->cycle_cntr_mask;
> +
> + riscv_iommu_pmu_writeq(value & counter_mask, pmu->reg + RISCV_IOMMU_REG_IOHPMCTR(idx));
> +}
> +
> +/*
> + * As stated in the RISC-V IOMMU Specification, Chapter 6:
> + * Whether an 8 byte access to an IOMMU register is single-copy atomic
> + * is UNSPECIFIED, and such an access may appear, internally to the
> + * IOMMU, as if two separate 4 byte accesses - first to the high half
> + * and second to the low half - were performed
> + *
> + * To make sure the driver works correctly on different hardware,
> + * the software will always use two 4-byte access for the counter.
> + *
> + * This function implements the hi-lo-hi pattern to detect and handle
> + * wraparound during the read operation:
> + * 1. Read high half (hi)
> + * 2. Read low half (lo)
> + * 3. Check if low half wrapped or high half changed:
> + * - If lo <= lo_prev: possible wraparound occurred
> + * - If hi != hi_prev: high half changed during read
> + * 4. If wraparound detected, re-read high half and assume low half is 0
> + * 5. Update previous values for next read
> + */
> +static u64 riscv_iommu_pmu_get_counter(struct riscv_iommu_pmu *pmu, u32 idx)
> +{
> + void __iomem *addr = pmu->reg + RISCV_IOMMU_REG_IOHPMCTR(idx);
> + u64 value, counter_mask = idx ? pmu->event_cntr_mask : pmu->cycle_cntr_mask;
> + u32 hi, lo;
> +
> + hi = readl(addr + 4);
> + lo = readl(addr);
> +
> + if (lo <= pmu->lo_prev[idx] || hi != pmu->hi_prev[idx]) {
> + u32 hi_tmp = readl(addr + 4);
> +
> + /*
> + * If hi changes, then hi+1:0 must have happened while
> + * the code was running so it is a safe return value
> + */
> + if (hi_tmp != hi) {
> + hi = hi_tmp;
> + lo = 0;
> + }
> + pmu->lo_prev[idx] = ~0u;
> + pmu->hi_prev[idx] = hi;
> + }
> + pmu->lo_prev[idx] = lo;
> +
> + value = (((u64)hi << 32) | lo) & counter_mask;
> +
> + /* The bit 63 of cycle counter (i.e., idx == 0) is OF bit */
> + return idx ? value : (value & ~RISCV_IOMMU_IOHPMCYCLES_OF);
> +}
> +
> +static bool is_cycle_event(u64 event)
> +{
> + return FIELD_GET(RISCV_IOMMU_IOHPMEVT_EVENTID, event) ==
> + RISCV_IOMMU_HPMEVENT_CYCLE;
> +}
> +
> +static void riscv_iommu_pmu_set_event(struct riscv_iommu_pmu *pmu, u32 idx,
> + u64 value)
> +{
> + /* There is no associtated IOHPMEVT0 for IOHPMCYCLES */
> + if (is_cycle_event(value))
> + return;
> +
> + /* Event counter start from idx 1 */
> + riscv_iommu_pmu_writeq(FIELD_GET(RISCV_IOMMU_IOHPMEVT_EVENT, value),
> + pmu->reg + RISCV_IOMMU_REG_IOHPMEVT(idx - 1));
> +}
> +
> +static void riscv_iommu_pmu_enable_counter(struct riscv_iommu_pmu *pmu, u32 idx)
> +{
> + void __iomem *addr = pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH;
> + u32 value = readl(addr);
> +
> + writel(value & ~BIT(idx), addr);
> +}
> +
> +static void riscv_iommu_pmu_disable_counter(struct riscv_iommu_pmu *pmu, u32 idx)
> +{
> + void __iomem *addr = pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH;
> + u32 value = readl(addr);
> +
> + writel(value | BIT(idx), addr);
> +}
> +
> +static void riscv_iommu_pmu_clear_ovf(struct perf_event *event, u32 idx)
> +{
> + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
> + u64 value;
> +
> + /* Counter is disabled here, making it safe to read and write registers */
> + if (is_cycle_event(get_event(event))) {
> + value = riscv_iommu_pmu_readq(pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES) &
> + ~RISCV_IOMMU_IOHPMCYCLES_OF;
> + riscv_iommu_pmu_writeq(value, pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES);
> + } else {
> + /* Event counter start from idx 1 */
> + value = riscv_iommu_pmu_readq(pmu->reg + RISCV_IOMMU_REG_IOHPMEVT(idx - 1)) &
> + ~RISCV_IOMMU_IOHPMEVT_OF;
> + riscv_iommu_pmu_writeq(value, pmu->reg + RISCV_IOMMU_REG_IOHPMEVT(idx - 1));
> + }
> +}
> +static void riscv_iommu_pmu_start_all(struct riscv_iommu_pmu *pmu)
> +{
> + void __iomem *addr = pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH;
> + u32 used_cntr = 0;
> +
> + /* The performance-monitoring counter inhibits is a 32-bit WARL register */
> + bitmap_to_arr32(&used_cntr, pmu->used_counters, pmu->num_counters);
> +
> + writel(~used_cntr, addr);
> +}
> +
> +static void riscv_iommu_pmu_stop_all(struct riscv_iommu_pmu *pmu)
> +{
> + writel(GENMASK_ULL(pmu->num_counters - 1, 0),
> + pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH);
> +}
> +
> +/* PMU APIs */
> +static void riscv_iommu_pmu_set_period(struct perf_event *event)
> +{
> + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
> + struct hw_perf_event *hwc = &event->hw;
> + u64 counter_mask = hwc->idx ? pmu->event_cntr_mask : pmu->cycle_cntr_mask;
> + u64 period;
> +
> + /*
> + * Limit the maximum period to prevent the counter value
> + * from overtaking the one we are about to program.
> + * In effect we are reducing max_period to account for
> + * interrupt latency (and we are being very conservative).
> + */
> + period = counter_mask >> 1;
> + riscv_iommu_pmu_set_counter(pmu, hwc->idx, period);
> + local64_set(&hwc->prev_count, period);
> +}
> +
> +static int riscv_iommu_pmu_event_init(struct perf_event *event)
> +{
> + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
> + struct hw_perf_event *hwc = &event->hw;
> + struct perf_event *sibling;
> + int total_event_counters = pmu->num_counters - 1;
> + int counters = 0;
> +
> + if (event->attr.type != event->pmu->type)
> + return -ENOENT;
> +
> + if (hwc->sample_period)
> + return -EOPNOTSUPP;
> +
> + if (event->cpu < 0)
> + return -EOPNOTSUPP;
> +
> + event->cpu = pmu->on_cpu;
> +
> + hwc->idx = -1;
> + hwc->config = event->attr.config;
> +
> + if (event->group_leader == event)
> + return 0;
> +
> + if (!is_cycle_event(get_event(event->group_leader)))
> + if (++counters > total_event_counters)
> + return -EINVAL;
> +
> + for_each_sibling_event(sibling, event->group_leader) {
> + if (is_cycle_event(get_event(sibling)) || is_software_event(sibling))
> + continue;
> +
> + if (sibling->pmu != event->pmu)
> + return -EINVAL;
> +
> + if (++counters > total_event_counters)
> + return -EINVAL;
> + }
> +
> + return 0;
> +}
> +
> +static void riscv_iommu_pmu_update(struct perf_event *event)
> +{
> + struct hw_perf_event *hwc = &event->hw;
> + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
> + u64 delta, prev, now;
> + u32 idx = hwc->idx;
> + u64 counter_mask = idx ? pmu->event_cntr_mask : pmu->cycle_cntr_mask;
> +
> + do {
> + prev = local64_read(&hwc->prev_count);
> + now = riscv_iommu_pmu_get_counter(pmu, idx);
> + } while (local64_cmpxchg(&hwc->prev_count, prev, now) != prev);
> +
> + delta = (now - prev) & counter_mask;
> + local64_add(delta, &event->count);
> +}
> +
> +static void riscv_iommu_pmu_start(struct perf_event *event, int flags)
> +{
> + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
> + struct hw_perf_event *hwc = &event->hw;
> +
> + if (WARN_ON_ONCE(!(event->hw.state & PERF_HES_STOPPED)))
> + return;
> +
> + if (flags & PERF_EF_RELOAD)
> + WARN_ON_ONCE(!(event->hw.state & PERF_HES_UPTODATE));
> +
> + hwc->state = 0;
> + riscv_iommu_pmu_set_period(event);
> + riscv_iommu_pmu_set_event(pmu, hwc->idx, hwc->config);
> + riscv_iommu_pmu_enable_counter(pmu, hwc->idx);
> +
> + perf_event_update_userpage(event);
> +}
> +
> +static void riscv_iommu_pmu_stop(struct perf_event *event, int flags)
> +{
> + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
> + struct hw_perf_event *hwc = &event->hw;
> + int idx = hwc->idx;
> +
> + if (hwc->state & PERF_HES_STOPPED)
> + return;
> +
> + riscv_iommu_pmu_disable_counter(pmu, idx);
> +
> + if ((flags & PERF_EF_UPDATE) && !(hwc->state & PERF_HES_UPTODATE))
> + riscv_iommu_pmu_update(event);
> +
> + hwc->state |= PERF_HES_STOPPED | PERF_HES_UPTODATE;
> +}
> +
> +static int riscv_iommu_pmu_add(struct perf_event *event, int flags)
> +{
> + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
> + struct hw_perf_event *hwc = &event->hw;
> + unsigned int num_counters = pmu->num_counters;
> + int idx;
> +
> + /* Reserve index zero for iohpmcycles */
> + if (is_cycle_event(get_event(event)))
> + idx = 0;
> + else
> + idx = find_next_zero_bit(pmu->used_counters, num_counters, 1);
> +
> + /* All event counters or cycle counter are in use */
> + if (idx == num_counters || pmu->events[idx])
> + return -EAGAIN;
> +
> + set_bit(idx, pmu->used_counters);
> +
> + pmu->events[idx] = event;
> + hwc->idx = idx;
> + hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
> + local64_set(&hwc->prev_count, 0);
> +
> + if (flags & PERF_EF_START)
> + riscv_iommu_pmu_start(event, flags);
> +
> + /* Propagate changes to the userspace mapping. */
> + perf_event_update_userpage(event);
> +
> + return 0;
> +}
> +
> +static void riscv_iommu_pmu_read(struct perf_event *event)
> +{
> + riscv_iommu_pmu_update(event);
> +}
> +
> +static void riscv_iommu_pmu_del(struct perf_event *event, int flags)
> +{
> + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu);
> + struct hw_perf_event *hwc = &event->hw;
> + int idx = hwc->idx;
> +
> + riscv_iommu_pmu_stop(event, PERF_EF_UPDATE);
> + pmu->events[idx] = NULL;
> + clear_bit(idx, pmu->used_counters);
> +
> + perf_event_update_userpage(event);
> +}
> +
> +static int riscv_iommu_pmu_online_cpu(unsigned int cpu, struct hlist_node *node)
> +{
> + struct riscv_iommu_pmu *iommu_pmu;
> +
> + iommu_pmu = hlist_entry_safe(node, struct riscv_iommu_pmu, node);
> +
> + if (iommu_pmu->on_cpu == -1)
> + iommu_pmu->on_cpu = cpu;
> +
> + return 0;
> +}
> +
> +static int riscv_iommu_pmu_offline_cpu(unsigned int cpu, struct hlist_node *node)
> +{
> + struct riscv_iommu_pmu *iommu_pmu;
> + unsigned int target_cpu;
> +
> + iommu_pmu = hlist_entry_safe(node, struct riscv_iommu_pmu, node);
> +
> + if (cpu != iommu_pmu->on_cpu)
> + return 0;
> +
> + iommu_pmu->on_cpu = -1;
> +
> + target_cpu = cpumask_any_but(cpu_online_mask, cpu);
> + if (target_cpu >= nr_cpu_ids)
> + return 0;
> +
> + perf_pmu_migrate_context(&iommu_pmu->pmu, cpu, target_cpu);
> + iommu_pmu->on_cpu = target_cpu;
> + WARN_ON(irq_set_affinity(iommu_pmu->irq, cpumask_of(target_cpu)));
> +
> + return 0;
> +}
> +
> +static irqreturn_t riscv_iommu_pmu_irq_handler(int irq, void *dev_id)
> +{
> + struct riscv_iommu_pmu *pmu = (struct riscv_iommu_pmu *)dev_id;
> + DECLARE_BITMAP(ovf_bitmap, RISCV_IOMMU_HPM_COUNTER_NUM);
> + u32 ovf, idx;
> +
> + /* Check whether this interrupt is for PMU */
> + if (!(readl_relaxed(pmu->reg + RISCV_IOMMU_REG_IPSR) & RISCV_IOMMU_IPSR_PMIP))
> + return IRQ_NONE;
> +
> + /* Process PMU IRQ */
> + riscv_iommu_pmu_stop_all(pmu);
> +
> + ovf = readl(pmu->reg + RISCV_IOMMU_REG_IOCOUNTOVF);
> + if (!ovf)
> + return IRQ_NONE;
> +
> + bitmap_from_u64(ovf_bitmap, ovf);
> + for_each_set_bit(idx, ovf_bitmap, pmu->num_counters) {
> + struct perf_event *event = pmu->events[idx];
> +
> + if (WARN_ON_ONCE(!event))
> + continue;
> +
> + riscv_iommu_pmu_update(event);
> + riscv_iommu_pmu_set_period(event);
> + riscv_iommu_pmu_clear_ovf(event, idx);
> + }
> +
> + /* Clear performance monitoring interrupt pending bit */
> + writel_relaxed(RISCV_IOMMU_IPSR_PMIP, pmu->reg + RISCV_IOMMU_REG_IPSR);
> +
> + riscv_iommu_pmu_start_all(pmu);
> +
> + return IRQ_HANDLED;
> +}
> +
> +static unsigned int riscv_iommu_pmu_get_irq_num(struct riscv_iommu_device *iommu)
> +{
> + /* Reuse ICVEC.CIV mask for all interrupt vectors mapping */
> + int vec = (iommu->icvec >> (RISCV_IOMMU_INTR_PM * 4)) & RISCV_IOMMU_ICVEC_CIV;
> +
> + return iommu->irqs[vec];
> +}
> +
> +static int riscv_iommu_pmu_request_irq(struct riscv_iommu_device *iommu,
> + struct riscv_iommu_pmu *pmu)
> +{
> + return devm_request_irq(iommu->dev, pmu->irq, riscv_iommu_pmu_irq_handler,
> + IRQF_NOBALANCING | IRQF_SHARED, dev_name(iommu->dev), pmu);
> +}
> +
> +static int riscv_iommu_pmu_probe(struct auxiliary_device *auxdev,
> + const struct auxiliary_device_id *id)
> +{
> + struct riscv_iommu_device *iommu_dev = dev_get_platdata(&auxdev->dev);
> + struct riscv_iommu_pmu *iommu_pmu;
> + void __iomem *addr;
> + char *name;
> + int ret;
> +
> + iommu_pmu = devm_kzalloc(&auxdev->dev, sizeof(*iommu_pmu), GFP_KERNEL);
> + if (!iommu_pmu)
> + return -ENOMEM;
> +
> + iommu_pmu->reg = iommu_dev->reg;
> +
> + /* Counter number and width are hardware-implemented. Detect them by write 1s */
> + addr = iommu_pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH;
> + writel(RISCV_IOMMU_IOCOUNTINH_HPM, addr);
> + iommu_pmu->num_counters = hweight32(readl(addr));
> +
> + addr = iommu_pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES;
> + riscv_iommu_pmu_writeq(RISCV_IOMMU_IOHPMCYCLES_COUNTER, addr);
> + iommu_pmu->cycle_cntr_mask = riscv_iommu_pmu_readq(addr);
> +
> + /* Assume the width of all event counters are the same */
> + addr = iommu_pmu->reg + RISCV_IOMMU_REG_IOHPMCTR_BASE;
> + riscv_iommu_pmu_writeq(RISCV_IOMMU_IOHPMCTR_COUNTER, addr);
> + iommu_pmu->event_cntr_mask = riscv_iommu_pmu_readq(addr);
> +
> + iommu_pmu->pmu = (struct pmu) {
> + .module = THIS_MODULE,
> + .parent = &auxdev->dev,
> + .task_ctx_nr = perf_invalid_context,
> + .event_init = riscv_iommu_pmu_event_init,
> + .add = riscv_iommu_pmu_add,
> + .del = riscv_iommu_pmu_del,
> + .start = riscv_iommu_pmu_start,
> + .stop = riscv_iommu_pmu_stop,
> + .read = riscv_iommu_pmu_read,
> + .attr_groups = riscv_iommu_pmu_attr_grps,
> + .capabilities = PERF_PMU_CAP_NO_EXCLUDE,
> + };
> +
> + auxiliary_set_drvdata(auxdev, iommu_pmu);
> +
> + name = devm_kasprintf(&auxdev->dev, GFP_KERNEL,
> + "riscv_iommu_pmu_%s", dev_name(iommu_dev->dev));
> + if (!name) {
> + dev_err(&auxdev->dev, "Failed to create name riscv_iommu_pmu_%s\n",
> + dev_name(iommu_dev->dev));
> + return -ENOMEM;
> + }
> +
> + /* Bind all events to the same cpu context to avoid race enabling */
> + iommu_pmu->on_cpu = raw_smp_processor_id();
> + iommu_pmu->irq = riscv_iommu_pmu_get_irq_num(iommu_dev);
> + WARN_ON(irq_set_affinity(iommu_pmu->irq, cpumask_of(iommu_pmu->on_cpu)));
> +
> + ret = riscv_iommu_pmu_request_irq(iommu_dev, iommu_pmu);
> + if (ret) {
> + dev_err(&auxdev->dev, "Failed to request irq %s: %d\n", name, ret);
> + return ret;
> + }
> +
> + ret = cpuhp_state_add_instance_nocalls(cpuhp_state, &iommu_pmu->node);
> + if (ret) {
> + dev_err(&auxdev->dev, "Failed to register hotplug %s: %d\n", name, ret);
> + return ret;
> + }
> +
> + ret = perf_pmu_register(&iommu_pmu->pmu, name, -1);
> + if (ret) {
> + dev_err(&auxdev->dev, "Failed to registe %s: %d\n", name, ret);
> + goto err_unregister;
> + }
> +
> + dev_info(&auxdev->dev, "%s: Registered with %d counters\n",
> + name, iommu_pmu->num_counters);
> +
> + return 0;
> +
> +err_unregister:
> + cpuhp_state_remove_instance_nocalls(cpuhp_state, &iommu_pmu->node);
> + return ret;
> +}
> +
> +static const struct auxiliary_device_id riscv_iommu_pmu_id_table[] = {
> + { .name = "iommu.pmu" },
> + {}
> +};
> +MODULE_DEVICE_TABLE(auxiliary, riscv_iommu_pmu_id_table);
> +
> +static struct auxiliary_driver iommu_pmu_driver = {
> + .probe = riscv_iommu_pmu_probe,
> + .id_table = riscv_iommu_pmu_id_table,
> +};
> +
> +static int __init riscv_iommu_pmu_init(void)
> +{
> + int ret;
> +
> + cpuhp_state = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN,
> + "perf/riscv/iommu:online",
> + riscv_iommu_pmu_online_cpu,
> + riscv_iommu_pmu_offline_cpu);
> + if (cpuhp_state < 0)
> + return cpuhp_state;
> +
> + ret = auxiliary_driver_register(&iommu_pmu_driver);
> + if (ret)
> + cpuhp_remove_multi_state(cpuhp_state);
> +
> + return ret;
> +}
> +module_init(riscv_iommu_pmu_init);
> +
> +MODULE_DESCRIPTION("RISC-V IOMMU PMU");
> +MODULE_LICENSE("GPL");
> --
> 2.43.7
>
LGTM!
Reviewed-by: Guo Ren (Alibaba DAMO Academy) <guoren@kernel.org>
--
Best Regards
Guo Ren
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v4 0/2] RISC-V IOMMU HPM support
2026-07-14 8:36 ` Zong Li
@ 2026-07-14 12:50 ` Chen Pei
-1 siblings, 0 replies; 14+ messages in thread
From: Chen Pei @ 2026-07-14 12:50 UTC (permalink / raw)
To: Zong Li
Cc: tjeznach, joro, will, robin.murphy, pjw, palmer, aou, alex,
mark.rutland, andrew.jones, guoren, david.laight.linux,
zhangzhanpeng.jasper, iommu, linux-riscv, linux-kernel,
linux-perf-users
On Tue, Jul 14, 2026 at 01:36:20AM -0700, Zong Li wrote:
> This series implements support for the RISC-V IOMMU hardware performance
> monitor.
>
> The RISC-V IOMMU PMU driver is implemented as an auxiliary device driver
> created by the parent RISC-V IOMMU driver. Therefore, the child driver
> can obtain resources and information from the parent device, such as
> the MMIO base address and IRQ number.
I tested this series on QEMU and it works as expected.
Test setup:
- Base: v7.2-rc2, both patches applied cleanly.
- Kernel: rv64 defconfig with CONFIG_RISCV_IOMMU=y, CONFIG_RISCV_IOMMU_PCI=y,
CONFIG_RISCV_IOMMU_PMU=y and CONFIG_PERF_EVENTS=y.
- QEMU 10.2.0: -M virt,aia=aplic-imsic -device riscv-iommu-pci
(hpm-counters=31).
Results:
- The RISC-V IOMMU probes, the auxiliary device is created and the PMU
driver registers successfully:
riscv_iommu_pmu ...: Registered with 32 counters
- The PMU shows up under
/sys/bus/event_source/devices/riscv_iommu_pmu_<BDF> with the expected
9 events (cycle, untranslated_req, translated_req, ats_trans_req,
tlb_miss, dd_walk, pd_walk, s_vs_pt_walks, g_pt_walks) and 7 format
attributes; "perf list" enumerates all of them.
- No oops/WARN/KASAN splat during boot or driver bring-up.
Tested-by: Chen Pei <cp0613@linux.alibaba.com>
Thanks,
Pei
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v4 0/2] RISC-V IOMMU HPM support
@ 2026-07-14 12:50 ` Chen Pei
0 siblings, 0 replies; 14+ messages in thread
From: Chen Pei @ 2026-07-14 12:50 UTC (permalink / raw)
To: Zong Li
Cc: tjeznach, joro, will, robin.murphy, pjw, palmer, aou, alex,
mark.rutland, andrew.jones, guoren, david.laight.linux,
zhangzhanpeng.jasper, iommu, linux-riscv, linux-kernel,
linux-perf-users
On Tue, Jul 14, 2026 at 01:36:20AM -0700, Zong Li wrote:
> This series implements support for the RISC-V IOMMU hardware performance
> monitor.
>
> The RISC-V IOMMU PMU driver is implemented as an auxiliary device driver
> created by the parent RISC-V IOMMU driver. Therefore, the child driver
> can obtain resources and information from the parent device, such as
> the MMIO base address and IRQ number.
I tested this series on QEMU and it works as expected.
Test setup:
- Base: v7.2-rc2, both patches applied cleanly.
- Kernel: rv64 defconfig with CONFIG_RISCV_IOMMU=y, CONFIG_RISCV_IOMMU_PCI=y,
CONFIG_RISCV_IOMMU_PMU=y and CONFIG_PERF_EVENTS=y.
- QEMU 10.2.0: -M virt,aia=aplic-imsic -device riscv-iommu-pci
(hpm-counters=31).
Results:
- The RISC-V IOMMU probes, the auxiliary device is created and the PMU
driver registers successfully:
riscv_iommu_pmu ...: Registered with 32 counters
- The PMU shows up under
/sys/bus/event_source/devices/riscv_iommu_pmu_<BDF> with the expected
9 events (cycle, untranslated_req, translated_req, ats_trans_req,
tlb_miss, dd_walk, pd_walk, s_vs_pt_walks, g_pt_walks) and 7 format
attributes; "perf list" enumerates all of them.
- No oops/WARN/KASAN splat during boot or driver bring-up.
Tested-by: Chen Pei <cp0613@linux.alibaba.com>
Thanks,
Pei
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v4 0/2] RISC-V IOMMU HPM support
2026-07-14 8:36 ` Zong Li
@ 2026-07-14 13:13 ` fangyu.yu
-1 siblings, 0 replies; 14+ messages in thread
From: fangyu.yu @ 2026-07-14 13:13 UTC (permalink / raw)
To: zong.li
Cc: alex, andrew.jones, aou, david.laight.linux, guoren, iommu, joro,
linux-kernel, linux-perf-users, linux-riscv, mark.rutland, palmer,
pjw, robin.murphy, tjeznach, will, zhangzhanpeng.jasper,
Fangyu Yu
>This series implements support for the RISC-V IOMMU hardware performance
>monitor.
>
>The RISC-V IOMMU PMU driver is implemented as an auxiliary device driver
>created by the parent RISC-V IOMMU driver. Therefore, the child driver
>can obtain resources and information from the parent device, such as
>the MMIO base address and IRQ number.
>
Hi Zong,
I tested this series on a riscv64 QEMU with a QEMU NVMe device.
The NVMe device was enumerated correctly, and the RISC-V IOMMU PMU
events could be counted successfully while running I/O to the NVMe disk.
The following command completed successfully:
perf stat -a \
-e riscv_iommu_pmu_3010000.iommu/tlb_miss/ \
-e riscv_iommu_pmu_3010000.iommu/cycle/ \
-e riscv_iommu_pmu_3010000.iommu/untranslated_req/ \
-e riscv_iommu_pmu_3010000.iommu/s_vs_pt_walks/ \
dd if=/dev/nvme0n1 of=/dev/null bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 1.43612 s, 73.0 MB/s
Performance counter stats for 'system wide':
26449 riscv_iommu_pmu_3010000.iommu/tlb_miss/
256659 riscv_iommu_pmu_3010000.iommu/cycle/
32063 riscv_iommu_pmu_3010000.iommu/untranslated_req/
132245 riscv_iommu_pmu_3010000.iommu/s_vs_pt_walks/
1.464510700 seconds time elapsed
Tested-by: Fangyu Yu <fangyu.yu@linux.alibaba.com>
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v4 0/2] RISC-V IOMMU HPM support
@ 2026-07-14 13:13 ` fangyu.yu
0 siblings, 0 replies; 14+ messages in thread
From: fangyu.yu @ 2026-07-14 13:13 UTC (permalink / raw)
To: zong.li
Cc: alex, andrew.jones, aou, david.laight.linux, guoren, iommu, joro,
linux-kernel, linux-perf-users, linux-riscv, mark.rutland, palmer,
pjw, robin.murphy, tjeznach, will, zhangzhanpeng.jasper,
Fangyu Yu
>This series implements support for the RISC-V IOMMU hardware performance
>monitor.
>
>The RISC-V IOMMU PMU driver is implemented as an auxiliary device driver
>created by the parent RISC-V IOMMU driver. Therefore, the child driver
>can obtain resources and information from the parent device, such as
>the MMIO base address and IRQ number.
>
Hi Zong,
I tested this series on a riscv64 QEMU with a QEMU NVMe device.
The NVMe device was enumerated correctly, and the RISC-V IOMMU PMU
events could be counted successfully while running I/O to the NVMe disk.
The following command completed successfully:
perf stat -a \
-e riscv_iommu_pmu_3010000.iommu/tlb_miss/ \
-e riscv_iommu_pmu_3010000.iommu/cycle/ \
-e riscv_iommu_pmu_3010000.iommu/untranslated_req/ \
-e riscv_iommu_pmu_3010000.iommu/s_vs_pt_walks/ \
dd if=/dev/nvme0n1 of=/dev/null bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 1.43612 s, 73.0 MB/s
Performance counter stats for 'system wide':
26449 riscv_iommu_pmu_3010000.iommu/tlb_miss/
256659 riscv_iommu_pmu_3010000.iommu/cycle/
32063 riscv_iommu_pmu_3010000.iommu/untranslated_req/
132245 riscv_iommu_pmu_3010000.iommu/s_vs_pt_walks/
1.464510700 seconds time elapsed
Tested-by: Fangyu Yu <fangyu.yu@linux.alibaba.com>
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2026-07-14 13:13 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-07-14 8:36 [PATCH v4 0/2] RISC-V IOMMU HPM support Zong Li
2026-07-14 8:36 ` Zong Li
2026-07-14 8:36 ` [PATCH v4 1/2] drivers/perf: riscv-iommu: add risc-v iommu pmu driver Zong Li
2026-07-14 8:36 ` Zong Li
2026-07-14 8:53 ` sashiko-bot
2026-07-14 11:22 ` Guo Ren
2026-07-14 11:22 ` Guo Ren
2026-07-14 8:36 ` [PATCH v4 2/2] iommu/riscv: create a auxiliary device for HPM Zong Li
2026-07-14 8:36 ` Zong Li
2026-07-14 9:03 ` sashiko-bot
2026-07-14 12:50 ` [PATCH v4 0/2] RISC-V IOMMU HPM support Chen Pei
2026-07-14 12:50 ` Chen Pei
2026-07-14 13:13 ` fangyu.yu
2026-07-14 13:13 ` fangyu.yu
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.