[PATCH v6 0/8] Introduce debugfs support in IOMMU

[PATCH v6 0/8] Introduce debugfs support in IOMMU

[Dheeraj Kumar Srivastava]

Introducing debugfs support in AMD/IOMMU driver that will allow
userspace to dump below IOMMU information
1) MMIO and Capability register per IOMMU
2) Command buffer
3) Device table entry
4) Interrupt remapping table entry

Analyzing contents of IOMMU data structures helps in understanding IOMMU
capabilities and behavior and debug issues faster.

1. MMIO and Capability registers - Add support to dump MMIO and Capability
   registers per IOMMU.

   Example:
   a. Write MMIO register offset to dump it
      $ echo 0x18 > /sys/kernel/debug/iommu/amd/iommu00/mmio
      $ cat /sys/kernel/debug/iommu/amd/iommu00/mmio

      Output:
      $ Offset:0x18 Value:0x000c22000003f48d

    b. Write capability register offset to dump it
       $ echo 0x10 > /sys/kernel/debug/iommu/amd/iommu00/capability
       $ cat /sys/kernel/debug/iommu/amd/iommu00/capability

       Output:
       $ Offset:0x10 Value:0x00203040

2. Command buffer - Add support to dump per IOMMU command buffer.

   Example:
   a. cat /sys/kernel/debug/iommu/amd/iommu00/cmdbuf

   Output:
   CMD Buffer Head Offset:339 Tail Offset:339
     0: 00835001 10000001 00009900 00000000
     1: 00000000 30000005 fffff003 7fffffff
     2: 00835001 10000001 00009901 00000000
     3: 00000000 30000005 fffff003 7fffffff
     4: 00835001 10000001 00009902 00000000
   ........................................
   ........................................
   ........................................

3. Device table - Add support to dump device table per IOMMU.

   Example:
   a. Write device id to dump device table entry for that device
      $ echo 0000:01:00.0 > /sys/kernel/debug/iommu/amd/devid
      $ cat /sys/kernel/debug/iommu/amd/devid

      Output:
      0000:01:00.0

      Dump the device table entry for the input given
      $ cat /sys/kernel/debug/iommu/amd/devtbl

      Output:
      DeviceId             QWORD[3]         QWORD[2]         QWORD[1]         QWORD[0] iommu
      0000:01:00.0 0000000000000000 20000001373b8013 0000000000000038 6000000114d7b603 iommu3

    b. Write device id to dump device table entry for that device
      $ echo 01:00.0 > /sys/kernel/debug/iommu/amd/devid
      $ cat /sys/kernel/debug/iommu/amd/devid

      Output:
      0000:01:00.0

      Dump the device table entry for the input given
      $ cat /sys/kernel/debug/iommu/amd/devtbl

      Output:
      DeviceId             QWORD[3]         QWORD[2]         QWORD[1]         QWORD[0] iommu
      0000:01:00.0 0000000000000000 20000001373b8013 0000000000000038 6000000114d7b603 iommu3

4. Interrupt remapping table - Add support to dump IRT table valid entries in
   "iommu_irqtbl" file. This supports user input to dump IRT entry for a
   specific pci device.

   Example:
   a. Write device id to dump device table entry for that device
      $ echo 0000:01:00.0 > /sys/kernel/debug/iommu/amd/devid
      $ cat /sys/kernel/debug/iommu/amd/devid

      Output:
      0000:01:00.0

      Dump the device table entry for the input given
      $ cat /sys/kernel/debug/iommu/amd/irtbl

      Output:
      DeviceId 0000:01:00.0
      IRT[0000] 0000000000000020 0000000000000241
      IRT[0001] 0000000000000020 0000000000000841
      IRT[0002] 0000000000000020 0000000000002041
      IRT[0003] 0000000000000020 0000000000008041
      IRT[0004] 0000000000000020 0000000000020041
      ..........................................
      ..........................................
      ..........................................

   b. Write device id to dump device table entry for that device
      $ echo 01:00.0 > /sys/kernel/debug/iommu/amd/devid
      $ cat /sys/kernel/debug/iommu/amd/devid

      Output:
      0000:01:00.0

      Dump the device table entry for the input given
      $ cat /sys/kernel/debug/iommu/amd/irttbl

      Output:
      Device 0000:01:00.0
      IRT[0000] 0000000000000020 0000000000000241
      IRT[0001] 0000000000000020 0000000000000841
      IRT[0002] 0000000000000020 0000000000002041
      IRT[0003] 0000000000000020 0000000000008041
      IRT[0004] 0000000000000020 0000000000020041
      ..........................................
      ..........................................
      ..........................................

Changes since v5:
-> Patch 7/8: Use macros to fetch IRT table length for DTE.

Changes since v4:
-> Maintain per IOMMU variable for mmio_offset and capability_offset.
-> Get intrrupt table length from DTE instead of using MAX_IRQS_PER_TABLE.

Changes since v3:
-> Patch 2/8: Print 64 bits instead of 32 bits for MMIO registers dump. Since most of
   the MMIO registers are 64 bits.
-> Patch 5/8: Update commit message with the need to add support for device_id input.
-> Patch 3/8: Throw error message mentioning valid capability offset range if user capability
   offset's input is not in valid range and input fails.

Changes since v2:
-> Patch 2/8 and 3/8: Use single file mmio and capability for input and output registers
   offset and register's dump respctively.
-> Patch 8/8: Correct path for files "irqtbl" and "devtbl" in Documentation

Changes since v1:
-> Patch 2/8 and 3/8: Use kstrtou32_from_user() instead of memdup_user_nul() --> kstrtou32()
-> Patch 4/8: Dump command buffer head and tail offset instead of head and tail pointer registers.
-> Patch 8/8: Fix bot reported warning on v1 patch series.

Dheeraj Kumar Srivastava (8):
  iommu/amd: Refactor AMD IOMMU debugfs initial setup
  iommu/amd: Add debugfs support to dump IOMMU MMIO registers
  iommu/amd: Add debugfs support to dump IOMMU Capability registers
  iommu/amd: Add debugfs support to dump IOMMU command buffer
  iommu/amd: Add support for device id user input
  iommu/amd: Add debugfs support to dump device table
  iommu/amd: Add debugfs support to dump IRT Table
  iommu/amd: Add documentation for AMD IOMMU debugfs support

 Documentation/ABI/testing/debugfs-amd-iommu | 114 ++++++
 drivers/iommu/amd/amd_iommu.h               |   4 +-
 drivers/iommu/amd/amd_iommu_types.h         |   9 +
 drivers/iommu/amd/debugfs.c                 | 377 +++++++++++++++++++-
 drivers/iommu/amd/init.c                    |   5 +-
 drivers/iommu/amd/iommu.c                   |   7 -
 6 files changed, 495 insertions(+), 21 deletions(-)
 create mode 100644 Documentation/ABI/testing/debugfs-amd-iommu

-- 
2.25.1

[PATCH v6 1/8] iommu/amd: Refactor AMD IOMMU debugfs initial setup

[Dheeraj Kumar Srivastava]

Rearrange initial setup of AMD IOMMU debugfs to segregate per IOMMU
setup and setup which is common for all IOMMUs. This ensures that common
debugfs paths (introduced in subsequent patches) are created only once
instead of being created for each IOMMU.

With the change, there is no need to use lock as amd_iommu_debugfs_setup()
will be called only once during AMD IOMMU initialization. So remove lock
acquisition in amd_iommu_debugfs_setup().

Signed-off-by: Dheeraj Kumar Srivastava <dheerajkumar.srivastava@amd.com>
Reviewed-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Reviewed-by: Vasant Hegde <vasant.hegde@amd.com>
---
 drivers/iommu/amd/amd_iommu.h |  4 ++--
 drivers/iommu/amd/debugfs.c   | 16 +++++++---------
 drivers/iommu/amd/init.c      |  5 ++---
 3 files changed, 11 insertions(+), 14 deletions(-)

diff --git a/drivers/iommu/amd/amd_iommu.h b/drivers/iommu/amd/amd_iommu.h
index 220c598b7e14..70873bb78b3c 100644
--- a/drivers/iommu/amd/amd_iommu.h
+++ b/drivers/iommu/amd/amd_iommu.h
@@ -28,9 +28,9 @@ void *__init iommu_alloc_4k_pages(struct amd_iommu *iommu,
 				  gfp_t gfp, size_t size);
 
 #ifdef CONFIG_AMD_IOMMU_DEBUGFS
-void amd_iommu_debugfs_setup(struct amd_iommu *iommu);
+void amd_iommu_debugfs_setup(void);
 #else
-static inline void amd_iommu_debugfs_setup(struct amd_iommu *iommu) {}
+static inline void amd_iommu_debugfs_setup(void) {}
 #endif
 
 /* Needed for interrupt remapping */
diff --git a/drivers/iommu/amd/debugfs.c b/drivers/iommu/amd/debugfs.c
index 545372fcc72f..ff9520e002be 100644
--- a/drivers/iommu/amd/debugfs.c
+++ b/drivers/iommu/amd/debugfs.c
@@ -13,20 +13,18 @@
 #include "amd_iommu.h"
 
 static struct dentry *amd_iommu_debugfs;
-static DEFINE_MUTEX(amd_iommu_debugfs_lock);
 
 #define	MAX_NAME_LEN	20
 
-void amd_iommu_debugfs_setup(struct amd_iommu *iommu)
+void amd_iommu_debugfs_setup(void)
 {
+	struct amd_iommu *iommu;
 	char name[MAX_NAME_LEN + 1];
 
-	mutex_lock(&amd_iommu_debugfs_lock);
-	if (!amd_iommu_debugfs)
-		amd_iommu_debugfs = debugfs_create_dir("amd",
-						       iommu_debugfs_dir);
-	mutex_unlock(&amd_iommu_debugfs_lock);
+	amd_iommu_debugfs = debugfs_create_dir("amd", iommu_debugfs_dir);
 
-	snprintf(name, MAX_NAME_LEN, "iommu%02d", iommu->index);
-	iommu->debugfs = debugfs_create_dir(name, amd_iommu_debugfs);
+	for_each_iommu(iommu) {
+		snprintf(name, MAX_NAME_LEN, "iommu%02d", iommu->index);
+		iommu->debugfs = debugfs_create_dir(name, amd_iommu_debugfs);
+	}
 }
diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c
index 14aa0d77df26..ecc6c246f87e 100644
--- a/drivers/iommu/amd/init.c
+++ b/drivers/iommu/amd/init.c
@@ -3387,7 +3387,6 @@ int amd_iommu_enable_faulting(unsigned int cpu)
  */
 static int __init amd_iommu_init(void)
 {
-	struct amd_iommu *iommu;
 	int ret;
 
 	ret = iommu_go_to_state(IOMMU_INITIALIZED);
@@ -3401,8 +3400,8 @@ static int __init amd_iommu_init(void)
 	}
 #endif
 
-	for_each_iommu(iommu)
-		amd_iommu_debugfs_setup(iommu);
+	if (!ret)
+		amd_iommu_debugfs_setup();
 
 	return ret;
 }
-- 
2.25.1

[PATCH v6 2/8] iommu/amd: Add debugfs support to dump IOMMU MMIO registers

[Dheeraj Kumar Srivastava]

Analyzing IOMMU MMIO registers gives a view of what IOMMU is
configured with on the system and is helpful to debug issues
with IOMMU.

eg.
-> To get mmio registers value at offset 0x18 for iommu<x> (say, iommu00)
   # echo "0x18" > /sys/kernel/debug/iommu/amd/iommu00/mmio
   # cat /sys/kernel/debug/iommu/amd/iommu00/mmio

Signed-off-by: Dheeraj Kumar Srivastava <dheerajkumar.srivastava@amd.com>
---
 drivers/iommu/amd/amd_iommu_types.h |  1 +
 drivers/iommu/amd/debugfs.c         | 47 +++++++++++++++++++++++++++++
 2 files changed, 48 insertions(+)

diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index 5089b58e528a..38eca75f6ef1 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -796,6 +796,7 @@ struct amd_iommu {
 #ifdef CONFIG_AMD_IOMMU_DEBUGFS
 	/* DebugFS Info */
 	struct dentry *debugfs;
+	int mmio_offset;
 #endif
 
 	/* IOPF support */
diff --git a/drivers/iommu/amd/debugfs.c b/drivers/iommu/amd/debugfs.c
index ff9520e002be..520377ab6731 100644
--- a/drivers/iommu/amd/debugfs.c
+++ b/drivers/iommu/amd/debugfs.c
@@ -15,6 +15,48 @@
 static struct dentry *amd_iommu_debugfs;
 
 #define	MAX_NAME_LEN	20
+#define	OFS_IN_SZ	8
+
+static ssize_t iommu_mmio_write(struct file *filp, const char __user *ubuf,
+				size_t cnt, loff_t *ppos)
+{
+	struct seq_file *m = filp->private_data;
+	struct amd_iommu *iommu = m->private;
+	int ret;
+
+	iommu->mmio_offset = -1;
+
+	if (cnt > OFS_IN_SZ)
+		return -EINVAL;
+
+	ret = kstrtou32_from_user(ubuf, cnt, 0, &iommu->mmio_offset);
+	if (ret)
+		return ret;
+
+	if (iommu->mmio_offset > iommu->mmio_phys_end - 4) {
+		iommu->mmio_offset = -1;
+		return  -EINVAL;
+	}
+
+	return cnt;
+}
+
+static int iommu_mmio_show(struct seq_file *m, void *unused)
+{
+	struct amd_iommu *iommu = m->private;
+	u64 value;
+
+	if (iommu->mmio_offset < 0) {
+		seq_puts(m, "Please provide mmio register's offset\n");
+		return 0;
+	}
+
+	value = readq(iommu->mmio_base + iommu->mmio_offset);
+	seq_printf(m, "Offset:0x%x Value:0x%016llx\n", iommu->mmio_offset, value);
+
+	return 0;
+}
+DEFINE_SHOW_STORE_ATTRIBUTE(iommu_mmio);
 
 void amd_iommu_debugfs_setup(void)
 {
@@ -24,7 +66,12 @@ void amd_iommu_debugfs_setup(void)
 	amd_iommu_debugfs = debugfs_create_dir("amd", iommu_debugfs_dir);
 
 	for_each_iommu(iommu) {
+		iommu->mmio_offset = -1;
+
 		snprintf(name, MAX_NAME_LEN, "iommu%02d", iommu->index);
 		iommu->debugfs = debugfs_create_dir(name, amd_iommu_debugfs);
+
+		debugfs_create_file("mmio", 0644, iommu->debugfs, iommu,
+				    &iommu_mmio_fops);
 	}
 }
-- 
2.25.1

[PATCH v6 3/8] iommu/amd: Add debugfs support to dump IOMMU Capability registers

[Dheeraj Kumar Srivastava]

IOMMU Capability registers defines capabilities of IOMMU and information
needed for initialising MMIO registers and device table. This is useful
to dump these registers for debugging IOMMU related issues.

e.g.
-> To get capability registers value at offset 0x10 for iommu<x> (say,
   iommu00)
   # echo "0x10" > /sys/kernel/debug/iommu/amd/iommu00/capability
   # cat /sys/kernel/debug/iommu/amd/iommu00/capability

Signed-off-by: Dheeraj Kumar Srivastava <dheerajkumar.srivastava@amd.com>
---
 drivers/iommu/amd/amd_iommu_types.h |  1 +
 drivers/iommu/amd/debugfs.c         | 51 +++++++++++++++++++++++++++++
 2 files changed, 52 insertions(+)

diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index 38eca75f6ef1..22679ca0bc26 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -797,6 +797,7 @@ struct amd_iommu {
 	/* DebugFS Info */
 	struct dentry *debugfs;
 	int mmio_offset;
+	int cap_offset;
 #endif
 
 	/* IOPF support */
diff --git a/drivers/iommu/amd/debugfs.c b/drivers/iommu/amd/debugfs.c
index 520377ab6731..5ac12c3fe069 100644
--- a/drivers/iommu/amd/debugfs.c
+++ b/drivers/iommu/amd/debugfs.c
@@ -58,6 +58,54 @@ static int iommu_mmio_show(struct seq_file *m, void *unused)
 }
 DEFINE_SHOW_STORE_ATTRIBUTE(iommu_mmio);
 
+static ssize_t iommu_capability_write(struct file *filp, const char __user *ubuf,
+				      size_t cnt, loff_t *ppos)
+{
+	struct seq_file *m = filp->private_data;
+	struct amd_iommu *iommu = m->private;
+	int ret;
+
+	iommu->cap_offset = -1;
+
+	if (cnt > OFS_IN_SZ)
+		return -EINVAL;
+
+	ret = kstrtou32_from_user(ubuf, cnt, 0, &iommu->cap_offset);
+	if (ret)
+		return ret;
+
+	/* Capability register at offset 0x14 is the last IOMMU capability register. */
+	if (iommu->cap_offset > 0x14) {
+		iommu->cap_offset = -1;
+		return -EINVAL;
+	}
+
+	return cnt;
+}
+
+static int iommu_capability_show(struct seq_file *m, void *unused)
+{
+	struct amd_iommu *iommu = m->private;
+	u32 value;
+	int err;
+
+	if (iommu->cap_offset < 0) {
+		seq_puts(m, "Please provide capability register's offset in the range [0x00 - 0x14]\n");
+		return 0;
+	}
+
+	err = pci_read_config_dword(iommu->dev, iommu->cap_ptr + iommu->cap_offset, &value);
+	if (err) {
+		seq_printf(m, "Not able to read capability register at 0x%x\n", iommu->cap_offset);
+		return 0;
+	}
+
+	seq_printf(m, "Offset:0x%x Value:0x%08x\n", iommu->cap_offset, value);
+
+	return 0;
+}
+DEFINE_SHOW_STORE_ATTRIBUTE(iommu_capability);
+
 void amd_iommu_debugfs_setup(void)
 {
 	struct amd_iommu *iommu;
@@ -67,11 +115,14 @@ void amd_iommu_debugfs_setup(void)
 
 	for_each_iommu(iommu) {
 		iommu->mmio_offset = -1;
+		iommu->cap_offset = -1;
 
 		snprintf(name, MAX_NAME_LEN, "iommu%02d", iommu->index);
 		iommu->debugfs = debugfs_create_dir(name, amd_iommu_debugfs);
 
 		debugfs_create_file("mmio", 0644, iommu->debugfs, iommu,
 				    &iommu_mmio_fops);
+		debugfs_create_file("capability", 0644, iommu->debugfs, iommu,
+				    &iommu_capability_fops);
 	}
 }
-- 
2.25.1

[PATCH v6 4/8] iommu/amd: Add debugfs support to dump IOMMU command buffer

[Dheeraj Kumar Srivastava]

IOMMU driver sends command to IOMMU hardware via command buffer. In cases
where IOMMU hardware fails to process commands in command buffer, dumping
it is a valuable input to debug the issue.

IOMMU hardware processes command buffer entry at offset equals to the head
pointer. Dumping just the entry at the head pointer may not always be
useful. The current head may not be pointing to the entry of the command
buffer which is causing the issue. IOMMU Hardware may have processed the
entry and updated the head pointer. So dumping the entire command buffer
gives a broad understanding of what hardware was/is doing. The command
buffer dump will have all entries from start to end of the command buffer.
Along with that, it will have a head and tail command buffer pointer
register dump to facilitate where the IOMMU driver and hardware are in
the command buffer for injecting and processing the entries respectively.

Command buffer is a per IOMMU data structure. So dumping on per IOMMU
basis.
eg.
-> To get command buffer dump for iommu<x> (say, iommu00)
   #cat /sys/kernel/debug/iommu/amd/iommu00/cmdbuf

Signed-off-by: Dheeraj Kumar Srivastava <dheerajkumar.srivastava@amd.com>
---
 drivers/iommu/amd/amd_iommu_types.h |  7 +++++++
 drivers/iommu/amd/debugfs.c         | 26 ++++++++++++++++++++++++++
 drivers/iommu/amd/iommu.c           |  7 -------
 3 files changed, 33 insertions(+), 7 deletions(-)

diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index 22679ca0bc26..2cf6fd676979 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -898,6 +898,13 @@ struct dev_table_entry {
 	};
 };
 
+/*
+ * Structure defining one entry in the command buffer
+ */
+struct iommu_cmd {
+	u32 data[4];
+};
+
 /*
  * Structure to sture persistent DTE flags from IVHD
  */
diff --git a/drivers/iommu/amd/debugfs.c b/drivers/iommu/amd/debugfs.c
index 5ac12c3fe069..61fde9ebfcfa 100644
--- a/drivers/iommu/amd/debugfs.c
+++ b/drivers/iommu/amd/debugfs.c
@@ -106,6 +106,30 @@ static int iommu_capability_show(struct seq_file *m, void *unused)
 }
 DEFINE_SHOW_STORE_ATTRIBUTE(iommu_capability);
 
+static int iommu_cmdbuf_show(struct seq_file *m, void *unused)
+{
+	struct amd_iommu *iommu = m->private;
+	struct iommu_cmd *cmd;
+	unsigned long flag;
+	u32 head, tail;
+	int i;
+
+	raw_spin_lock_irqsave(&iommu->lock, flag);
+	head = readl(iommu->mmio_base + MMIO_CMD_HEAD_OFFSET);
+	tail = readl(iommu->mmio_base + MMIO_CMD_TAIL_OFFSET);
+	seq_printf(m, "CMD Buffer Head Offset:%d Tail Offset:%d\n",
+		   (head >> 4) & 0x7fff, (tail >> 4) & 0x7fff);
+	for (i = 0; i < CMD_BUFFER_ENTRIES; i++) {
+		cmd = (struct iommu_cmd *)(iommu->cmd_buf + i * sizeof(*cmd));
+		seq_printf(m, "%3d: %08x %08x %08x %08x\n", i, cmd->data[0],
+			   cmd->data[1], cmd->data[2], cmd->data[3]);
+	}
+	raw_spin_unlock_irqrestore(&iommu->lock, flag);
+
+	return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(iommu_cmdbuf);
+
 void amd_iommu_debugfs_setup(void)
 {
 	struct amd_iommu *iommu;
@@ -124,5 +148,7 @@ void amd_iommu_debugfs_setup(void)
 				    &iommu_mmio_fops);
 		debugfs_create_file("capability", 0644, iommu->debugfs, iommu,
 				    &iommu_capability_fops);
+		debugfs_create_file("cmdbuf", 0444, iommu->debugfs, iommu,
+				    &iommu_cmdbuf_fops);
 	}
 }
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index f34209b08b4c..522251f60e8e 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -62,13 +62,6 @@ static const struct iommu_dirty_ops amd_dirty_ops;
 
 int amd_iommu_max_glx_val = -1;
 
-/*
- * general struct to manage commands send to an IOMMU
- */
-struct iommu_cmd {
-	u32 data[4];
-};
-
 /*
  * AMD IOMMU allows up to 2^16 different protection domains. This is a bitmap
  * to know which ones are already in use.
-- 
2.25.1

[PATCH v6 5/8] iommu/amd: Add support for device id user input

[Dheeraj Kumar Srivastava]

Dumping IOMMU data structures like device table, IRT, etc., for all devices
on the system will be a lot of data dumped in a file. Also, user may want
to dump and analyze these data structures just for one or few devices. So
dumping IOMMU data structures like device table, IRT etc for all devices
is not a good approach.

Add "device id" user input to be used for dumping IOMMU data structures
like device table, IRT etc in AMD IOMMU debugfs.

eg.
1. # echo 0000:01:00.0 > /sys/kernel/debug/iommu/amd/devid
   # cat /sys/kernel/debug/iommu/amd/devid
   Output : 0000:01:00.0

2. # echo 01:00.0 > /sys/kernel/debug/iommu/amd/devid
   # cat /sys/kernel/debug/iommu/amd/devid
   Output : 0000:01:00.0

Signed-off-by: Dheeraj Kumar Srivastava <dheerajkumar.srivastava@amd.com>
---
 drivers/iommu/amd/debugfs.c | 80 +++++++++++++++++++++++++++++++++++++
 1 file changed, 80 insertions(+)

diff --git a/drivers/iommu/amd/debugfs.c b/drivers/iommu/amd/debugfs.c
index 61fde9ebfcfa..8b57ab4ab809 100644
--- a/drivers/iommu/amd/debugfs.c
+++ b/drivers/iommu/amd/debugfs.c
@@ -16,6 +16,9 @@ static struct dentry *amd_iommu_debugfs;
 
 #define	MAX_NAME_LEN	20
 #define	OFS_IN_SZ	8
+#define	DEVID_IN_SZ	16
+
+static int sbdf = -1;
 
 static ssize_t iommu_mmio_write(struct file *filp, const char __user *ubuf,
 				size_t cnt, loff_t *ppos)
@@ -130,6 +133,80 @@ static int iommu_cmdbuf_show(struct seq_file *m, void *unused)
 }
 DEFINE_SHOW_ATTRIBUTE(iommu_cmdbuf);
 
+static ssize_t devid_write(struct file *filp, const char __user *ubuf,
+			   size_t cnt, loff_t *ppos)
+{
+	struct amd_iommu_pci_seg *pci_seg;
+	int seg, bus, slot, func;
+	struct amd_iommu *iommu;
+	char *srcid_ptr;
+	u16 devid;
+	int i;
+
+	sbdf = -1;
+
+	if (cnt >= DEVID_IN_SZ)
+		return -EINVAL;
+
+	srcid_ptr = memdup_user_nul(ubuf, cnt);
+	if (IS_ERR(srcid_ptr))
+		return PTR_ERR(srcid_ptr);
+
+	i = sscanf(srcid_ptr, "%x:%x:%x.%x", &seg, &bus, &slot, &func);
+	if (i != 4) {
+		i = sscanf(srcid_ptr, "%x:%x.%x", &bus, &slot, &func);
+		if (i != 3) {
+			kfree(srcid_ptr);
+			return -EINVAL;
+		}
+		seg = 0;
+	}
+
+	devid = PCI_DEVID(bus, PCI_DEVFN(slot, func));
+
+	/* Check if user device id input is a valid input */
+	for_each_pci_segment(pci_seg) {
+		if (pci_seg->id != seg)
+			continue;
+		if (devid > pci_seg->last_bdf) {
+			kfree(srcid_ptr);
+			return -EINVAL;
+		}
+		iommu = pci_seg->rlookup_table[devid];
+		if (!iommu) {
+			kfree(srcid_ptr);
+			return -ENODEV;
+		}
+		break;
+	}
+
+	if (pci_seg->id != seg) {
+		kfree(srcid_ptr);
+		return -EINVAL;
+	}
+
+	sbdf = PCI_SEG_DEVID_TO_SBDF(seg, devid);
+
+	kfree(srcid_ptr);
+
+	return cnt;
+}
+
+static int devid_show(struct seq_file *m, void *unused)
+{
+	u16 devid;
+
+	if (sbdf >= 0) {
+		devid = PCI_SBDF_TO_DEVID(sbdf);
+		seq_printf(m, "%04x:%02x:%02x.%x\n", PCI_SBDF_TO_SEGID(sbdf),
+			   PCI_BUS_NUM(devid), PCI_SLOT(devid), PCI_FUNC(devid));
+	} else
+		seq_puts(m, "No or Invalid input provided\n");
+
+	return 0;
+}
+DEFINE_SHOW_STORE_ATTRIBUTE(devid);
+
 void amd_iommu_debugfs_setup(void)
 {
 	struct amd_iommu *iommu;
@@ -151,4 +228,7 @@ void amd_iommu_debugfs_setup(void)
 		debugfs_create_file("cmdbuf", 0444, iommu->debugfs, iommu,
 				    &iommu_cmdbuf_fops);
 	}
+
+	debugfs_create_file("devid", 0644, amd_iommu_debugfs, NULL,
+			    &devid_fops);
 }
-- 
2.25.1

[PATCH v6 6/8] iommu/amd: Add debugfs support to dump device table

[Dheeraj Kumar Srivastava]

IOMMU uses device table data structure to get per-device information for
DMA remapping, interrupt remapping, and other functionalities. It's a
valuable data structure to visualize for debugging issues related to
IOMMU.

eg.
-> To dump device table entry for a particular device
   #echo 0000:c4:00.0 > /sys/kernel/debug/iommu/amd/devid
   #cat /sys/kernel/debug/iommu/amd/devtbl

   or

   #echo c4:00.0 > /sys/kernel/debug/iommu/amd/devid
   #cat /sys/kernel/debug/iommu/amd/devtbl

Signed-off-by: Dheeraj Kumar Srivastava <dheerajkumar.srivastava@amd.com>
---
 drivers/iommu/amd/debugfs.c | 49 +++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)

diff --git a/drivers/iommu/amd/debugfs.c b/drivers/iommu/amd/debugfs.c
index 8b57ab4ab809..38d3cab5fd8d 100644
--- a/drivers/iommu/amd/debugfs.c
+++ b/drivers/iommu/amd/debugfs.c
@@ -207,6 +207,53 @@ static int devid_show(struct seq_file *m, void *unused)
 }
 DEFINE_SHOW_STORE_ATTRIBUTE(devid);
 
+static void dump_dte(struct seq_file *m, struct amd_iommu_pci_seg *pci_seg, u16 devid)
+{
+	struct dev_table_entry *dev_table;
+	struct amd_iommu *iommu;
+
+	iommu = pci_seg->rlookup_table[devid];
+	if (!iommu)
+		return;
+
+	dev_table = get_dev_table(iommu);
+	if (!dev_table) {
+		seq_puts(m, "Device table not found");
+		return;
+	}
+
+	seq_printf(m, "%-12s %16s %16s %16s %16s iommu\n", "DeviceId",
+		   "QWORD[3]", "QWORD[2]", "QWORD[1]", "QWORD[0]");
+	seq_printf(m, "%04x:%02x:%02x.%x ", pci_seg->id, PCI_BUS_NUM(devid),
+		   PCI_SLOT(devid), PCI_FUNC(devid));
+	for (int i = 3; i >= 0; --i)
+		seq_printf(m, "%016llx ", dev_table[devid].data[i]);
+	seq_printf(m, "iommu%d\n", iommu->index);
+}
+
+static int iommu_devtbl_show(struct seq_file *m, void *unused)
+{
+	struct amd_iommu_pci_seg *pci_seg;
+	u16 seg, devid;
+
+	if (sbdf < 0) {
+		seq_puts(m, "Please provide valid device id input\n");
+		return 0;
+	}
+	seg = PCI_SBDF_TO_SEGID(sbdf);
+	devid = PCI_SBDF_TO_DEVID(sbdf);
+
+	for_each_pci_segment(pci_seg) {
+		if (pci_seg->id != seg)
+			continue;
+		dump_dte(m, pci_seg, devid);
+		break;
+	}
+
+	return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(iommu_devtbl);
+
 void amd_iommu_debugfs_setup(void)
 {
 	struct amd_iommu *iommu;
@@ -231,4 +278,6 @@ void amd_iommu_debugfs_setup(void)
 
 	debugfs_create_file("devid", 0644, amd_iommu_debugfs, NULL,
 			    &devid_fops);
+	debugfs_create_file("devtbl", 0444, amd_iommu_debugfs, NULL,
+			    &iommu_devtbl_fops);
 }
-- 
2.25.1

[PATCH v6 7/8] iommu/amd: Add debugfs support to dump IRT Table

[Dheeraj Kumar Srivastava]

In cases where we have an issue in the device interrupt path with IOMMU
interrupt remapping enabled, dumping valid IRT table entries for the device
is very useful and good input for debugging the issue.

eg.
-> To dump irte entries for a particular device
   #echo "c4:00.0" > /sys/kernel/debug/iommu/amd/devid
   #cat /sys/kernel/debug/iommu/amd/irqtbl | less

   or

   #echo "0000:c4:00.0" > /sys/kernel/debug/iommu/amd/devid
   #cat /sys/kernel/debug/iommu/amd/irqtbl | less

Signed-off-by: Dheeraj Kumar Srivastava <dheerajkumar.srivastava@amd.com>
---
 drivers/iommu/amd/debugfs.c | 108 ++++++++++++++++++++++++++++++++++++
 1 file changed, 108 insertions(+)

diff --git a/drivers/iommu/amd/debugfs.c b/drivers/iommu/amd/debugfs.c
index 38d3cab5fd8d..7a056ebcc7c3 100644
--- a/drivers/iommu/amd/debugfs.c
+++ b/drivers/iommu/amd/debugfs.c
@@ -11,6 +11,7 @@
 #include <linux/pci.h>
 
 #include "amd_iommu.h"
+#include "../irq_remapping.h"
 
 static struct dentry *amd_iommu_debugfs;
 
@@ -254,6 +255,111 @@ static int iommu_devtbl_show(struct seq_file *m, void *unused)
 }
 DEFINE_SHOW_ATTRIBUTE(iommu_devtbl);
 
+static void dump_128_irte(struct seq_file *m, struct irq_remap_table *table, u16 int_tab_len)
+{
+	struct irte_ga *ptr, *irte;
+	int index;
+
+	for (index = 0; index < int_tab_len; index++) {
+		ptr = (struct irte_ga *)table->table;
+		irte = &ptr[index];
+
+		if (AMD_IOMMU_GUEST_IR_VAPIC(amd_iommu_guest_ir) &&
+		    !irte->lo.fields_vapic.valid)
+			continue;
+		else if (!irte->lo.fields_remap.valid)
+			continue;
+		seq_printf(m, "IRT[%04d] %016llx %016llx\n", index, irte->hi.val, irte->lo.val);
+	}
+}
+
+static void dump_32_irte(struct seq_file *m, struct irq_remap_table *table, u16 int_tab_len)
+{
+	union irte *ptr, *irte;
+	int index;
+
+	for (index = 0; index < int_tab_len; index++) {
+		ptr = (union irte *)table->table;
+		irte = &ptr[index];
+
+		if (!irte->fields.valid)
+			continue;
+		seq_printf(m, "IRT[%04d] %08x\n", index, irte->val);
+	}
+}
+
+static void dump_irte(struct seq_file *m, u16 devid, struct amd_iommu_pci_seg *pci_seg)
+{
+	struct dev_table_entry *dev_table;
+	struct irq_remap_table *table;
+	struct amd_iommu *iommu;
+	unsigned long flags;
+	u16 int_tab_len;
+
+	table = pci_seg->irq_lookup_table[devid];
+	if (!table) {
+		seq_printf(m, "IRQ lookup table not set for %04x:%02x:%02x:%x\n",
+			   pci_seg->id, PCI_BUS_NUM(devid), PCI_SLOT(devid), PCI_FUNC(devid));
+		return;
+	}
+
+	iommu = pci_seg->rlookup_table[devid];
+	if (!iommu)
+		return;
+
+	dev_table = get_dev_table(iommu);
+	if (!dev_table) {
+		seq_puts(m, "Device table not found");
+		return;
+	}
+
+	int_tab_len = dev_table[devid].data[2] & DTE_INTTABLEN_MASK;
+	if (int_tab_len != DTE_INTTABLEN_512 && int_tab_len != DTE_INTTABLEN_2K) {
+		seq_puts(m, "The device's DTE contains an invalid IRT length value.");
+		return;
+	}
+
+	seq_printf(m, "DeviceId %04x:%02x:%02x.%x\n", pci_seg->id, PCI_BUS_NUM(devid),
+		   PCI_SLOT(devid), PCI_FUNC(devid));
+
+	raw_spin_lock_irqsave(&table->lock, flags);
+	if (AMD_IOMMU_GUEST_IR_GA(amd_iommu_guest_ir))
+		dump_128_irte(m, table, BIT(int_tab_len >> 1));
+	else
+		dump_32_irte(m, table, BIT(int_tab_len >> 1));
+	seq_puts(m, "\n");
+	raw_spin_unlock_irqrestore(&table->lock, flags);
+}
+
+static int iommu_irqtbl_show(struct seq_file *m, void *unused)
+{
+	struct amd_iommu_pci_seg *pci_seg;
+	u16 devid, seg;
+
+	if (!irq_remapping_enabled) {
+		seq_puts(m, "Interrupt remapping is disabled\n");
+		return 0;
+	}
+
+	if (sbdf < 0) {
+		seq_puts(m, "Please provide valid device id input\n");
+		return 0;
+	}
+
+	seg = PCI_SBDF_TO_SEGID(sbdf);
+	devid = PCI_SBDF_TO_DEVID(sbdf);
+
+	for_each_pci_segment(pci_seg) {
+		if (pci_seg->id != seg)
+			continue;
+		dump_irte(m, devid, pci_seg);
+		break;
+	}
+
+	return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(iommu_irqtbl);
+
 void amd_iommu_debugfs_setup(void)
 {
 	struct amd_iommu *iommu;
@@ -280,4 +386,6 @@ void amd_iommu_debugfs_setup(void)
 			    &devid_fops);
 	debugfs_create_file("devtbl", 0444, amd_iommu_debugfs, NULL,
 			    &iommu_devtbl_fops);
+	debugfs_create_file("irqtbl", 0444, amd_iommu_debugfs, NULL,
+			    &iommu_irqtbl_fops);
 }
-- 
2.25.1

[PATCH v6 8/8] iommu/amd: Add documentation for AMD IOMMU debugfs support

[Dheeraj Kumar Srivastava]

Add documentation describing how to use AMD IOMMU debugfs support to
dump IOMMU data structures - IRT table, Device table, Registers (MMIO and
Capability) and command buffer.

Signed-off-by: Dheeraj Kumar Srivastava <dheerajkumar.srivastava@amd.com>
---
 Documentation/ABI/testing/debugfs-amd-iommu | 114 ++++++++++++++++++++
 1 file changed, 114 insertions(+)
 create mode 100644 Documentation/ABI/testing/debugfs-amd-iommu

diff --git a/Documentation/ABI/testing/debugfs-amd-iommu b/Documentation/ABI/testing/debugfs-amd-iommu
new file mode 100644
index 000000000000..7d6cf1f602ed
--- /dev/null
+++ b/Documentation/ABI/testing/debugfs-amd-iommu
@@ -0,0 +1,114 @@
+What:		/sys/kernel/debug/iommu/amd/iommu<x>/mmio
+Date:		January 2025
+Contact:	Dheeraj Kumar Srivastava <dheerajkumar.srivastava@amd.com>
+Description:
+		This is an input read/write access file. In this file, the user input
+		mmio register offset for iommu<x> to print corresponding mmio register
+		of iommu<x>.
+
+		Example:
+		$ echo "0x18" > /sys/kernel/debug/iommu/amd/iommu00/mmio
+		$ cat /sys/kernel/debug/iommu/amd/iommu00/mmio
+
+		Output:
+		Offset:0x18 Value:0x000c22000003f48d
+
+What:		/sys/kernel/debug/iommu/amd/iommu<x>/capability
+Date:		January 2025
+Contact:	Dheeraj Kumar Srivastava <dheerajkumar.srivastava@amd.com>
+Description:
+		This is an input read/write access file. In this file, the user input
+                capability register offset for iommu<x> to print corresponding capability
+		register of iommu<x>.
+
+		Example:
+		$ echo "0x10" > /sys/kernel/debug/iommu/amd/iommu00/capability
+		$ cat /sys/kernel/debug/iommu/amd/iommu00/capability
+
+		Output:
+		Offset:0x10 Value:0x00203040
+
+What:		/sys/kernel/debug/iommu/amd/iommu<x>/cmdbuf
+Date:		January 2025
+Contact:	Dheeraj Kumar Srivastava <dheerajkumar.srivastava@amd.com>
+Description:
+		This file is an output read only file that contains iommu<x> command
+		buffer entries.
+
+		Examples:
+		$ cat /sys/kernel/debug/iommu/amd/iommu<x>/cmdbuf
+
+		Output:
+		CMD Buffer Head Offset:339 Tail Offset:339
+		  0: 00835001 10000001 00003c00 00000000
+		  1: 00000000 30000005 fffff003 7fffffff
+		  2: 00835001 10000001 00003c01 00000000
+		  3: 00000000 30000005 fffff003 7fffffff
+		  4: 00835001 10000001 00003c02 00000000
+		  5: 00000000 30000005 fffff003 7fffffff
+		  6: 00835001 10000001 00003c03 00000000
+		  7: 00000000 30000005 fffff003 7fffffff
+		  8: 00835001 10000001 00003c04 00000000
+		  9: 00000000 30000005 fffff003 7fffffff
+		 10: 00835001 10000001 00003c05 00000000
+		 11: 00000000 30000005 fffff003 7fffffff
+		[...]
+
+What:		/sys/kernel/debug/iommu/amd/devid
+Date:		January 2025
+Contact:	Dheeraj Kumar Srivastava <dheerajkumar.srivastava@amd.com>
+Description:
+		This is an input read/write file that takes device id user input.
+		This input can be used for dumping iommu data structures like
+		interrupt remapping table, device table etc.
+
+		Example:
+		1.
+		$ echo 0000:01:00.0 > /sys/kernel/debug/iommu/amd/devid
+		$ cat /sys/kernel/debug/iommu/amd/devid
+
+		Output:
+		0000:01:00.0
+
+		2.
+		$ echo 01:00.0 > /sys/kernel/debug/iommu/amd/devid
+		$ cat /sys/kernel/debug/iommu/amd/devid
+
+		Output:
+		0000:01:00.0
+
+What:		/sys/kernel/debug/iommu/amd/devtbl
+Date:		January 2025
+Contact:	Dheeraj Kumar Srivastava <dheerajkumar.srivastava@amd.com>
+Description:
+		This is an output read only file that contains device table entry for
+		the device id input given in /sys/kernel/debug/iommu/amd/devid.
+
+		Example:
+		$ cat /sys/kernel/debug/iommu/amd/devtbl
+
+		Output:
+		DeviceId             QWORD[3]         QWORD[2]         QWORD[1]         QWORD[0] iommu
+		0000:01:00.0 0000000000000000 20000001373b8013 0000000000000038 6000000114d7b603 iommu3
+
+What:		/sys/kernel/debug/iommu/amd/irqtbl
+Date:		January 2025
+Contact:	Dheeraj Kumar Srivastava <dheerajkumar.srivastava@amd.com>
+Description:
+		This is an output read only file that contains IRT table valid entries
+		for the device id input given in /sys/kernel/debug/iommu/amd/devid.
+
+		Example:
+		$ cat /sys/kernel/debug/iommu/amd/irqtbl
+
+		Output:
+		DeviceId 0000:01:00.0
+		IRT[0000] 0000000000000020 0000000000000241
+		IRT[0001] 0000000000000020 0000000000000841
+		IRT[0002] 0000000000000020 0000000000002041
+		IRT[0003] 0000000000000020 0000000000008041
+		IRT[0004] 0000000000000020 0000000000020041
+		IRT[0005] 0000000000000020 0000000000080041
+		IRT[0006] 0000000000000020 0000000000200041
+		IRT[0007] 0000000000000020 0000000000800041
+		[...]
-- 
2.25.1

Re: [PATCH v6 0/8] Introduce debugfs support in IOMMU

[Dheeraj Kumar Srivastava]

Hi,

Just a friendly reminder to review the patch series. Your valuable 
feedback would be greatly appreciated to help us move forward.

Thanks
Dheeraj

On 6/4/2025 10:06 PM, Dheeraj Kumar Srivastava wrote:
> Introducing debugfs support in AMD/IOMMU driver that will allow
> userspace to dump below IOMMU information
> 1) MMIO and Capability register per IOMMU
> 2) Command buffer
> 3) Device table entry
> 4) Interrupt remapping table entry
> 
> Analyzing contents of IOMMU data structures helps in understanding IOMMU
> capabilities and behavior and debug issues faster.
> 
> 1. MMIO and Capability registers - Add support to dump MMIO and Capability
>     registers per IOMMU.
> 
>     Example:
>     a. Write MMIO register offset to dump it
>        $ echo 0x18 > /sys/kernel/debug/iommu/amd/iommu00/mmio
>        $ cat /sys/kernel/debug/iommu/amd/iommu00/mmio
> 
>        Output:
>        $ Offset:0x18 Value:0x000c22000003f48d
> 
>      b. Write capability register offset to dump it
>         $ echo 0x10 > /sys/kernel/debug/iommu/amd/iommu00/capability
>         $ cat /sys/kernel/debug/iommu/amd/iommu00/capability
> 
>         Output:
>         $ Offset:0x10 Value:0x00203040
> 
> 2. Command buffer - Add support to dump per IOMMU command buffer.
> 
>     Example:
>     a. cat /sys/kernel/debug/iommu/amd/iommu00/cmdbuf
> 
>     Output:
>     CMD Buffer Head Offset:339 Tail Offset:339
>       0: 00835001 10000001 00009900 00000000
>       1: 00000000 30000005 fffff003 7fffffff
>       2: 00835001 10000001 00009901 00000000
>       3: 00000000 30000005 fffff003 7fffffff
>       4: 00835001 10000001 00009902 00000000
>     ........................................
>     ........................................
>     ........................................
> 
> 3. Device table - Add support to dump device table per IOMMU.
> 
>     Example:
>     a. Write device id to dump device table entry for that device
>        $ echo 0000:01:00.0 > /sys/kernel/debug/iommu/amd/devid
>        $ cat /sys/kernel/debug/iommu/amd/devid
> 
>        Output:
>        0000:01:00.0
> 
>        Dump the device table entry for the input given
>        $ cat /sys/kernel/debug/iommu/amd/devtbl
> 
>        Output:
>        DeviceId             QWORD[3]         QWORD[2]         QWORD[1]         QWORD[0] iommu
>        0000:01:00.0 0000000000000000 20000001373b8013 0000000000000038 6000000114d7b603 iommu3
> 
>      b. Write device id to dump device table entry for that device
>        $ echo 01:00.0 > /sys/kernel/debug/iommu/amd/devid
>        $ cat /sys/kernel/debug/iommu/amd/devid
> 
>        Output:
>        0000:01:00.0
> 
>        Dump the device table entry for the input given
>        $ cat /sys/kernel/debug/iommu/amd/devtbl
> 
>        Output:
>        DeviceId             QWORD[3]         QWORD[2]         QWORD[1]         QWORD[0] iommu
>        0000:01:00.0 0000000000000000 20000001373b8013 0000000000000038 6000000114d7b603 iommu3
> 
> 4. Interrupt remapping table - Add support to dump IRT table valid entries in
>     "iommu_irqtbl" file. This supports user input to dump IRT entry for a
>     specific pci device.
> 
>     Example:
>     a. Write device id to dump device table entry for that device
>        $ echo 0000:01:00.0 > /sys/kernel/debug/iommu/amd/devid
>        $ cat /sys/kernel/debug/iommu/amd/devid
> 
>        Output:
>        0000:01:00.0
> 
>        Dump the device table entry for the input given
>        $ cat /sys/kernel/debug/iommu/amd/irtbl
> 
>        Output:
>        DeviceId 0000:01:00.0
>        IRT[0000] 0000000000000020 0000000000000241
>        IRT[0001] 0000000000000020 0000000000000841
>        IRT[0002] 0000000000000020 0000000000002041
>        IRT[0003] 0000000000000020 0000000000008041
>        IRT[0004] 0000000000000020 0000000000020041
>        ..........................................
>        ..........................................
>        ..........................................
> 
>     b. Write device id to dump device table entry for that device
>        $ echo 01:00.0 > /sys/kernel/debug/iommu/amd/devid
>        $ cat /sys/kernel/debug/iommu/amd/devid
> 
>        Output:
>        0000:01:00.0
> 
>        Dump the device table entry for the input given
>        $ cat /sys/kernel/debug/iommu/amd/irttbl
> 
>        Output:
>        Device 0000:01:00.0
>        IRT[0000] 0000000000000020 0000000000000241
>        IRT[0001] 0000000000000020 0000000000000841
>        IRT[0002] 0000000000000020 0000000000002041
>        IRT[0003] 0000000000000020 0000000000008041
>        IRT[0004] 0000000000000020 0000000000020041
>        ..........................................
>        ..........................................
>        ..........................................
> 
> Changes since v5:
> -> Patch 7/8: Use macros to fetch IRT table length for DTE.
> 
> Changes since v4:
> -> Maintain per IOMMU variable for mmio_offset and capability_offset.
> -> Get intrrupt table length from DTE instead of using MAX_IRQS_PER_TABLE.
> 
> Changes since v3:
> -> Patch 2/8: Print 64 bits instead of 32 bits for MMIO registers dump. Since most of
>     the MMIO registers are 64 bits.
> -> Patch 5/8: Update commit message with the need to add support for device_id input.
> -> Patch 3/8: Throw error message mentioning valid capability offset range if user capability
>     offset's input is not in valid range and input fails.
> 
> Changes since v2:
> -> Patch 2/8 and 3/8: Use single file mmio and capability for input and output registers
>     offset and register's dump respctively.
> -> Patch 8/8: Correct path for files "irqtbl" and "devtbl" in Documentation
> 
> Changes since v1:
> -> Patch 2/8 and 3/8: Use kstrtou32_from_user() instead of memdup_user_nul() --> kstrtou32()
> -> Patch 4/8: Dump command buffer head and tail offset instead of head and tail pointer registers.
> -> Patch 8/8: Fix bot reported warning on v1 patch series.
> 
> Dheeraj Kumar Srivastava (8):
>    iommu/amd: Refactor AMD IOMMU debugfs initial setup
>    iommu/amd: Add debugfs support to dump IOMMU MMIO registers
>    iommu/amd: Add debugfs support to dump IOMMU Capability registers
>    iommu/amd: Add debugfs support to dump IOMMU command buffer
>    iommu/amd: Add support for device id user input
>    iommu/amd: Add debugfs support to dump device table
>    iommu/amd: Add debugfs support to dump IRT Table
>    iommu/amd: Add documentation for AMD IOMMU debugfs support
> 
>   Documentation/ABI/testing/debugfs-amd-iommu | 114 ++++++
>   drivers/iommu/amd/amd_iommu.h               |   4 +-
>   drivers/iommu/amd/amd_iommu_types.h         |   9 +
>   drivers/iommu/amd/debugfs.c                 | 377 +++++++++++++++++++-
>   drivers/iommu/amd/init.c                    |   5 +-
>   drivers/iommu/amd/iommu.c                   |   7 -
>   6 files changed, 495 insertions(+), 21 deletions(-)
>   create mode 100644 Documentation/ABI/testing/debugfs-amd-iommu
> 

Re: [PATCH v6 0/8] Introduce debugfs support in IOMMU

[Joerg Roedel]

Hi Dheeraj,

On Thu, Jun 26, 2025 at 11:27:54AM +0530, Dheeraj Kumar Srivastava wrote:
> Just a friendly reminder to review the patch series. Your valuable feedback
> would be greatly appreciated to help us move forward.

Will merge once acked/reviewed by Suravee or Vasant.

Regards,

	Joerg

Re: [PATCH v6 6/8] iommu/amd: Add debugfs support to dump device table

[Vasant Hegde]

Dheeraj,


On 6/4/2025 10:06 PM, Dheeraj Kumar Srivastava wrote:
> IOMMU uses device table data structure to get per-device information for
> DMA remapping, interrupt remapping, and other functionalities. It's a
> valuable data structure to visualize for debugging issues related to
> IOMMU.
> 
> eg.
> -> To dump device table entry for a particular device
>    #echo 0000:c4:00.0 > /sys/kernel/debug/iommu/amd/devid
>    #cat /sys/kernel/debug/iommu/amd/devtbl
> 
>    or
> 
>    #echo c4:00.0 > /sys/kernel/debug/iommu/amd/devid
>    #cat /sys/kernel/debug/iommu/amd/devtbl
> 
> Signed-off-by: Dheeraj Kumar Srivastava <dheerajkumar.srivastava@amd.com>
> ---
>  drivers/iommu/amd/debugfs.c | 49 +++++++++++++++++++++++++++++++++++++
>  1 file changed, 49 insertions(+)
> 
> diff --git a/drivers/iommu/amd/debugfs.c b/drivers/iommu/amd/debugfs.c
> index 8b57ab4ab809..38d3cab5fd8d 100644
> --- a/drivers/iommu/amd/debugfs.c
> +++ b/drivers/iommu/amd/debugfs.c
> @@ -207,6 +207,53 @@ static int devid_show(struct seq_file *m, void *unused)
>  }
>  DEFINE_SHOW_STORE_ATTRIBUTE(devid);
>  
> +static void dump_dte(struct seq_file *m, struct amd_iommu_pci_seg *pci_seg, u16 devid)
> +{
> +	struct dev_table_entry *dev_table;
> +	struct amd_iommu *iommu;
> +
> +	iommu = pci_seg->rlookup_table[devid];
> +	if (!iommu)
> +		return;
> +
> +	dev_table = get_dev_table(iommu);
> +	if (!dev_table) {
> +		seq_puts(m, "Device table not found");
> +		return;
> +	}
> +
> +	seq_printf(m, "%-12s %16s %16s %16s %16s iommu\n", "DeviceId",
> +		   "QWORD[3]", "QWORD[2]", "QWORD[1]", "QWORD[0]");
> +	seq_printf(m, "%04x:%02x:%02x.%x ", pci_seg->id, PCI_BUS_NUM(devid),
> +		   PCI_SLOT(devid), PCI_FUNC(devid));
> +	for (int i = 3; i >= 0; --i)
> +		seq_printf(m, "%016llx ", dev_table[devid].data[i]);
> +	seq_printf(m, "iommu%d\n", iommu->index);
> +}
> +
> +static int iommu_devtbl_show(struct seq_file *m, void *unused)
> +{
> +	struct amd_iommu_pci_seg *pci_seg;
> +	u16 seg, devid;
> +
> +	if (sbdf < 0) {
> +		seq_puts(m, "Please provide valid device id input\n");

Where? This should clearly indicate where to provide device id.


-Vasant

Re: [PATCH v6 7/8] iommu/amd: Add debugfs support to dump IRT Table

[Vasant Hegde]

Dheeraj,


On 6/4/2025 10:06 PM, Dheeraj Kumar Srivastava wrote:
> In cases where we have an issue in the device interrupt path with IOMMU
> interrupt remapping enabled, dumping valid IRT table entries for the device
> is very useful and good input for debugging the issue.
> 
> eg.
> -> To dump irte entries for a particular device
>    #echo "c4:00.0" > /sys/kernel/debug/iommu/amd/devid
>    #cat /sys/kernel/debug/iommu/amd/irqtbl | less
> 
>    or
> 
>    #echo "0000:c4:00.0" > /sys/kernel/debug/iommu/amd/devid
>    #cat /sys/kernel/debug/iommu/amd/irqtbl | less
> 
> Signed-off-by: Dheeraj Kumar Srivastava <dheerajkumar.srivastava@amd.com>
> ---
>  drivers/iommu/amd/debugfs.c | 108 ++++++++++++++++++++++++++++++++++++
>  1 file changed, 108 insertions(+)
> 
> diff --git a/drivers/iommu/amd/debugfs.c b/drivers/iommu/amd/debugfs.c
> index 38d3cab5fd8d..7a056ebcc7c3 100644
> --- a/drivers/iommu/amd/debugfs.c
> +++ b/drivers/iommu/amd/debugfs.c
> @@ -11,6 +11,7 @@
>  #include <linux/pci.h>
>  
>  #include "amd_iommu.h"
> +#include "../irq_remapping.h"
>  
>  static struct dentry *amd_iommu_debugfs;
>  
> @@ -254,6 +255,111 @@ static int iommu_devtbl_show(struct seq_file *m, void *unused)
>  }
>  DEFINE_SHOW_ATTRIBUTE(iommu_devtbl);
>  
> +static void dump_128_irte(struct seq_file *m, struct irq_remap_table *table, u16 int_tab_len)
> +{
> +	struct irte_ga *ptr, *irte;
> +	int index;
> +
> +	for (index = 0; index < int_tab_len; index++) {
> +		ptr = (struct irte_ga *)table->table;
> +		irte = &ptr[index];
> +
> +		if (AMD_IOMMU_GUEST_IR_VAPIC(amd_iommu_guest_ir) &&
> +		    !irte->lo.fields_vapic.valid)
> +			continue;
> +		else if (!irte->lo.fields_remap.valid)
> +			continue;
> +		seq_printf(m, "IRT[%04d] %016llx %016llx\n", index, irte->hi.val, irte->lo.val);
> +	}
> +}
> +
> +static void dump_32_irte(struct seq_file *m, struct irq_remap_table *table, u16 int_tab_len)
> +{
> +	union irte *ptr, *irte;
> +	int index;
> +
> +	for (index = 0; index < int_tab_len; index++) {
> +		ptr = (union irte *)table->table;
> +		irte = &ptr[index];
> +
> +		if (!irte->fields.valid)
> +			continue;
> +		seq_printf(m, "IRT[%04d] %08x\n", index, irte->val);
> +	}
> +}
> +
> +static void dump_irte(struct seq_file *m, u16 devid, struct amd_iommu_pci_seg *pci_seg)
> +{
> +	struct dev_table_entry *dev_table;
> +	struct irq_remap_table *table;
> +	struct amd_iommu *iommu;
> +	unsigned long flags;
> +	u16 int_tab_len;
> +
> +	table = pci_seg->irq_lookup_table[devid];
> +	if (!table) {
> +		seq_printf(m, "IRQ lookup table not set for %04x:%02x:%02x:%x\n",
> +			   pci_seg->id, PCI_BUS_NUM(devid), PCI_SLOT(devid), PCI_FUNC(devid));
> +		return;
> +	}
> +
> +	iommu = pci_seg->rlookup_table[devid];
> +	if (!iommu)
> +		return;
> +
> +	dev_table = get_dev_table(iommu);
> +	if (!dev_table) {
> +		seq_puts(m, "Device table not found");
> +		return;
> +	}
> +
> +	int_tab_len = dev_table[devid].data[2] & DTE_INTTABLEN_MASK;
> +	if (int_tab_len != DTE_INTTABLEN_512 && int_tab_len != DTE_INTTABLEN_2K) {
> +		seq_puts(m, "The device's DTE contains an invalid IRT length value.");
> +		return;
> +	}
> +
> +	seq_printf(m, "DeviceId %04x:%02x:%02x.%x\n", pci_seg->id, PCI_BUS_NUM(devid),
> +		   PCI_SLOT(devid), PCI_FUNC(devid));
> +
> +	raw_spin_lock_irqsave(&table->lock, flags);
> +	if (AMD_IOMMU_GUEST_IR_GA(amd_iommu_guest_ir))
> +		dump_128_irte(m, table, BIT(int_tab_len >> 1));
> +	else
> +		dump_32_irte(m, table, BIT(int_tab_len >> 1));
> +	seq_puts(m, "\n");
> +	raw_spin_unlock_irqrestore(&table->lock, flags);
> +}
> +
> +static int iommu_irqtbl_show(struct seq_file *m, void *unused)
> +{
> +	struct amd_iommu_pci_seg *pci_seg;
> +	u16 devid, seg;
> +
> +	if (!irq_remapping_enabled) {
> +		seq_puts(m, "Interrupt remapping is disabled\n");
> +		return 0;
> +	}
> +
> +	if (sbdf < 0) {
> +		seq_puts(m, "Please provide valid device id input\n");

Again same as previous patch comment. Please add proper description so that user
know where to input devid.

-Vasant

Re: [PATCH v6 2/8] iommu/amd: Add debugfs support to dump IOMMU MMIO registers

[Vasant Hegde]

On 6/4/2025 10:06 PM, Dheeraj Kumar Srivastava wrote:
> Analyzing IOMMU MMIO registers gives a view of what IOMMU is
> configured with on the system and is helpful to debug issues
> with IOMMU.
> 
> eg.
> -> To get mmio registers value at offset 0x18 for iommu<x> (say, iommu00)
>    # echo "0x18" > /sys/kernel/debug/iommu/amd/iommu00/mmio
>    # cat /sys/kernel/debug/iommu/amd/iommu00/mmio
> 
> Signed-off-by: Dheeraj Kumar Srivastava <dheerajkumar.srivastava@amd.com>
> ---
>  drivers/iommu/amd/amd_iommu_types.h |  1 +
>  drivers/iommu/amd/debugfs.c         | 47 +++++++++++++++++++++++++++++
>  2 files changed, 48 insertions(+)
> 
> diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
> index 5089b58e528a..38eca75f6ef1 100644
> --- a/drivers/iommu/amd/amd_iommu_types.h
> +++ b/drivers/iommu/amd/amd_iommu_types.h
> @@ -796,6 +796,7 @@ struct amd_iommu {
>  #ifdef CONFIG_AMD_IOMMU_DEBUGFS
>  	/* DebugFS Info */
>  	struct dentry *debugfs;
> +	int mmio_offset;

This is covered under DEBUGFS config. So this is fine.. but its nice if you
rename the variable such that it indicates its usage.

-Vasant

Re: [PATCH v6 0/8] Introduce debugfs support in IOMMU

[Vasant Hegde]

Dheeraj,


On 6/4/2025 10:06 PM, Dheeraj Kumar Srivastava wrote:
> Introducing debugfs support in AMD/IOMMU driver that will allow
> userspace to dump below IOMMU information
> 1) MMIO and Capability register per IOMMU
> 2) Command buffer
> 3) Device table entry
> 4) Interrupt remapping table entry
> 

Overall this series looks good to me. I have done some level of testing and its
all fine. I still have few minor comments. Please address them.

-Vasant

Re: [PATCH v6 0/8] Introduce debugfs support in IOMMU

[Dheeraj Kumar Srivastava]

Hi Vasant,

On 6/29/2025 11:24 AM, Vasant Hegde wrote:
> Dheeraj,
> 
> 
> On 6/4/2025 10:06 PM, Dheeraj Kumar Srivastava wrote:
>> Introducing debugfs support in AMD/IOMMU driver that will allow
>> userspace to dump below IOMMU information
>> 1) MMIO and Capability register per IOMMU
>> 2) Command buffer
>> 3) Device table entry
>> 4) Interrupt remapping table entry
>>
> 
> Overall this series looks good to me. I have done some level of testing and its
> all fine. I still have few minor comments. Please address them.

Addressed all your review comments on v6 series and posted v7 series. 
Please review.

Thanks
Dheeraj

> 
> -Vasant
> 
> 
> 
> 
>