[PATCH 0/3] AMD IOMMUv2 Performance Counter patches

[PATCH 0/3] AMD IOMMUv2 Performance Counter patches

[Steven L. Kinney]

From: "Steven L. Kinney" <steven.kinney-5C7GfCeVMHo@public.gmane.org>

These patches implement the AMD IOMMUv2.5 Performance Counter functionality
via custom perf PMU and implement static counting for various IOMMU
translations.  The patches address three areas of functionality:

1) 	Add a PCI quirk for the enablement of IOMMUv2 EFR PC within a specific
	AMD family/model in which BIOS has not enabled.  Based on code 
	implemented by Andreas Herrmann at AMD Dresden.
2)	Extend the AMD IOMMU initialization to include IOMMUv2 PC enablement
	and access to IOMMUv2 counter registers.  For all AMD family/models
	that implement IOMMUv2.5 functionality.
3)	Code the perf IOMMUv2 PMU to manage IOMMUv2 perf events, which call
	the function(s) extending the AMD IOMMU core driver.  For all AMD
	family/models that implement IOMMUv2.5 functionality.

The command-line, to invoke the iommuv2 PMU, is:

perf stat -e iommuv2/config=[data],config1=[data]/{u,r} [command]

For information regarding AMD IOMMU v2.5 PC configuration, see the public
specification.

Steven L. Kinney (3):
  AMD x86 quirks: Quirk for enabling IOMMUv2 PC feature
  AMD IOMMUv2 PC resource management hooks
  AMD IOMMUv2 PC perf PMU implementation

 arch/x86/kernel/cpu/Makefile                 |    1 +
 arch/x86/kernel/cpu/perf_event_amd_iommuv2.c |  429 ++++++++++++++++++++++++++
 arch/x86/kernel/cpu/perf_event_amd_iommuv2.h |   42 +++
 arch/x86/kernel/quirks.c                     |   17 +
 drivers/iommu/Kconfig                        |   10 +
 drivers/iommu/amd_iommu_init.c               |   99 ++++++
 drivers/iommu/amd_iommu_types.h              |   12 +
 include/linux/pci_ids.h                      |    2 +
 8 files changed, 612 insertions(+)
 create mode 100644 arch/x86/kernel/cpu/perf_event_amd_iommuv2.c
 create mode 100644 arch/x86/kernel/cpu/perf_event_amd_iommuv2.h

-- 
1.7.9.5

[PATCH 1/3] AMD x86 quirks: Quirk for enabling IOMMUv2 PC feature

[Steven L. Kinney]

From: "Steven L. Kinney" <steven.kinney-5C7GfCeVMHo@public.gmane.org>

Original code base by Andreas Herrmann at AMD with minor changes.

Signed-off-by: Steven L. Kinney <steven.kinney-5C7GfCeVMHo@public.gmane.org>
---
 arch/x86/kernel/quirks.c |   17 +++++++++++++++++
 include/linux/pci_ids.h  |    2 ++
 2 files changed, 19 insertions(+)

diff --git a/arch/x86/kernel/quirks.c b/arch/x86/kernel/quirks.c
index 26ee48a..ab27b88 100644
--- a/arch/x86/kernel/quirks.c
+++ b/arch/x86/kernel/quirks.c
@@ -567,3 +567,20 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_15H_NB_F5,
 			quirk_amd_nb_node);
 
 #endif
+
+#if defined(CONFIG_PCI) && defined(CONFIG_AMD_IOMMU_V2_PC)
+
+static void amd_force_iommu_v2_pc(struct pci_dev *dev)
+{
+	u32 val;
+
+	pci_read_config_dword(dev, 0x70, &val);
+	val |= (1<<9);
+	pci_write_config_dword(dev, 0x70, val);
+	dev_printk(KERN_DEBUG, &dev->dev, "Enabled IOMMUv2 PC\n");
+}
+
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_15H_M10H_IOMMU,
+			amd_force_iommu_v2_pc);
+
+#endif
diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
index 0eb6579..c6ce47b 100644
--- a/include/linux/pci_ids.h
+++ b/include/linux/pci_ids.h
@@ -396,6 +396,8 @@
 /* AMD RD890 Chipset */
 #define PCI_DEVICE_ID_RD890_IOMMU	0x5a23
 
+#define PCI_DEVICE_ID_AMD_15H_M10H_IOMMU       0x1419
+
 #define PCI_VENDOR_ID_ADL		0x1005
 #define PCI_DEVICE_ID_ADL_2301		0x2301
 
-- 
1.7.9.5

[PATCH 2/3] AMD IOMMUv2 PC resource management hooks

[Steven L. Kinney]

From: "Steven L. Kinney" <steven.kinney-5C7GfCeVMHo@public.gmane.org>

Add functionality to check the availability of the AMD IOMMUv2 Performance
Counters and export this functionality to other core drivers, such as in this
case, a perf IOMMUv2 PMU.  This feature is not bound to any specific AMD
family/model other than the presence of the IOMMUv2 with PC enabled.

The IOMMUv2 PC support static counting only at this time.

Signed-off-by: Steven L. Kinney <steven.kinney-5C7GfCeVMHo@public.gmane.org>
---
 drivers/iommu/Kconfig           |   10 ++++
 drivers/iommu/amd_iommu_init.c  |   99 +++++++++++++++++++++++++++++++++++++++
 drivers/iommu/amd_iommu_types.h |   12 +++++
 3 files changed, 121 insertions(+)

diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index e39f9db..4a941dd 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -73,6 +73,16 @@ config AMD_IOMMU_V2
 	  hardware. Select this option if you want to use devices that support
 	  the PCI PRI and PASID interface.
 
+# AMD IOMMUv2 Performance Counter support
+config AMD_IOMMU_V2_PC
+	bool "AMD IOMMUv2 Performance Counter (EXPERIMENTAL)"
+	depends on AMD_IOMMU_V2
+	---help---
+	  This option enables support for AMD IOMMUv2 Performance Counters.
+	  Select this option if you want to enable IOMMUv2 Performance
+	  Counter support.
+	  If unsure, say N.
+
 # Intel IOMMU support
 config DMAR_TABLE
 	bool
diff --git a/drivers/iommu/amd_iommu_init.c b/drivers/iommu/amd_iommu_init.c
index 81837b0..f38db14 100644
--- a/drivers/iommu/amd_iommu_init.c
+++ b/drivers/iommu/amd_iommu_init.c
@@ -154,6 +154,7 @@ bool amd_iommu_iotlb_sup __read_mostly = true;
 u32 amd_iommu_max_pasids __read_mostly = ~0;
 
 bool amd_iommu_v2_present __read_mostly;
+bool amd_iommu_v2_pc_present __read_mostly;
 
 bool amd_iommu_force_isolation __read_mostly;
 
@@ -1145,6 +1146,21 @@ static int iommu_init_pci(struct amd_iommu *iommu)
 	if (iommu->cap & (1UL << IOMMU_CAP_NPCACHE))
 		amd_iommu_np_cache = true;
 
+#ifdef CONFIG_AMD_IOMMU_V2_PC
+	if (iommu_feature(iommu, FEATURE_PC)) {
+		u32 val;
+		amd_iommu_v2_pc_present = true;
+		dev_printk(KERN_DEBUG, &iommu->dev->dev,
+			   "AMD-Vi: IOMMUv2 perf counters supported\n");
+		val = readl(iommu->mmio_base + MMIO_CNTR_CONF_OFFSET);
+		iommu->max_banks = (u8) ((val >> 12) & 0x3f);
+		iommu->max_counters = (u8) ((val >> 7) & 0xf);
+		dev_printk(KERN_DEBUG, &iommu->dev->dev,
+			   "AMD-Vi: %d counter banks, %d counters each\n",
+			   iommu->max_banks, iommu->max_counters);
+	}
+#endif
+
 	if (is_rd890_iommu(iommu->dev)) {
 		int i, j;
 
@@ -2076,3 +2092,86 @@ bool amd_iommu_v2_supported(void)
 	return amd_iommu_v2_present;
 }
 EXPORT_SYMBOL(amd_iommu_v2_supported);
+
+#ifdef CONFIG_AMD_IOMMU_V2_PC
+/****************************************************************************
+ *
+ * IOMMUv2 EFR Performance Counter support functionality. This code allows
+ * access to the IOMMUv2 PC functionality.
+ *
+ ****************************************************************************/
+
+u8 amd_iommu_v2_get_max_pc_banks(u16 devid)
+{
+	struct amd_iommu *iommu;
+
+	/* locate the iommu governing the devid */
+	iommu = amd_iommu_rlookup_table[devid];
+
+	if (iommu)
+		return iommu->max_banks;
+
+	return -ENODEV;
+}
+EXPORT_SYMBOL(amd_iommu_v2_get_max_pc_banks);
+
+bool amd_iommu_v2_pc_supported(void)
+{
+	return amd_iommu_v2_pc_present;
+}
+EXPORT_SYMBOL(amd_iommu_v2_pc_supported);
+
+u8 amd_iommu_v2_get_max_pc_counters(u16 devid)
+{
+	struct amd_iommu *iommu;
+
+	/* locate the iommu governing the devid */
+	iommu = amd_iommu_rlookup_table[devid];
+
+	if (iommu)
+		return iommu->max_counters;
+
+	return -ENODEV;
+}
+EXPORT_SYMBOL(amd_iommu_v2_get_max_pc_counters);
+
+int amd_iommu_v2_get_set_pc_reg_val(u16 devid, u8 bank, u8 cntr, u8 fxn,
+				    long long *value, bool is_write)
+{
+	struct amd_iommu *iommu;
+	u32 offset;
+	u32 max_offset_lim;
+
+	/* Make sure the IOMMUv2 PC resource is available */
+	if (!amd_iommu_v2_pc_present) {
+		pr_info("AMD IOMMUv2 - PC Not supported in amd_iommu_v2_get_set_pc_reg_val\n");
+		return -ENODEV;
+	}
+
+	/* locate the iommu associated with the device ID */
+	iommu = amd_iommu_rlookup_table[devid];
+	if (iommu == NULL)
+		return -ENODEV;
+
+	/* check for valid iommu pc register indexing */
+	if (fxn < 0 || fxn > 0x28 || (fxn & 7))
+		return -ENODEV;
+
+	offset = (u32)(((0x40|bank) << 12) | (cntr << 8) | fxn);
+
+	/* limit the offset to the hw defined mmio region aperture */
+	max_offset_lim = (u32)(((0x40|iommu->max_banks) << 12) |
+				(iommu->max_counters << 8) | 0x28);
+	if ((offset < IOMMU_V2_PC_REG_OFFSET) ||
+	    (offset > max_offset_lim))
+		return -EINVAL;
+
+	if (is_write)
+		writel((u32)*value, iommu->mmio_base + offset);
+	else
+		*value = readl(iommu->mmio_base + offset);
+
+	return 0;
+}
+EXPORT_SYMBOL(amd_iommu_v2_get_set_pc_reg_val);
+#endif
diff --git a/drivers/iommu/amd_iommu_types.h b/drivers/iommu/amd_iommu_types.h
index e38ab43..fdf236b 100644
--- a/drivers/iommu/amd_iommu_types.h
+++ b/drivers/iommu/amd_iommu_types.h
@@ -38,7 +38,12 @@
 #define RLOOKUP_TABLE_ENTRY_SIZE	(sizeof(void *))
 
 /* Length of the MMIO region for the AMD IOMMU */
+#ifdef CONFIG_AMD_IOMMU_V2_PC
+#define MMIO_REGION_LENGTH       0x80000
+#define IOMMU_V2_PC_REG_OFFSET	 0x40000
+#else
 #define MMIO_REGION_LENGTH       0x4000
+#endif
 
 /* Capability offsets used by the driver */
 #define MMIO_CAP_HDR_OFFSET	0x00
@@ -77,6 +82,7 @@
 #define MMIO_STATUS_OFFSET	0x2020
 #define MMIO_PPR_HEAD_OFFSET	0x2030
 #define MMIO_PPR_TAIL_OFFSET	0x2038
+#define MMIO_CNTR_CONF_OFFSET	0x4000
 
 
 /* Extended Feature Bits */
@@ -585,6 +591,12 @@ struct amd_iommu {
 
 	/* The l2 indirect registers */
 	u32 stored_l2[0x83];
+
+#ifdef CONFIG_AMD_IOMMU_V2_PC
+	/* The maximum PC banks and counters/bank (PCSup=1) */
+	u8 max_banks;
+	u8 max_counters;
+#endif
 };
 
 struct devid_map {
-- 
1.7.9.5

[PATCH 3/3] AMD IOMMUv2 PC perf PMU implementation

[Steven L. Kinney]

From: "Steven L. Kinney" <steven.kinney-5C7GfCeVMHo@public.gmane.org>

Implement a perf PMU to handle IOMMUv2 PC perf events.  This PMU will handle
static counter perf events relative to the AMD IOMMUv2 Performance Counters.

To invoke the AMD IOMMUv2 PMU issue a perf tool command such as:

./perf stat -e iommuv2/config=<config-data>,config1=<config1-data>/u <command>

For example:

./perf stat -e iommuv2/config=08000000000000005,config1=0/u <command>

The resulting count will be how many IOMMUv2 totsal peripheral memory
operations were performed during the command execution window.

Signed-off-by: Steven L. Kinney <steven.kinney-5C7GfCeVMHo@public.gmane.org>
---
 arch/x86/kernel/cpu/Makefile                 |    1 +
 arch/x86/kernel/cpu/perf_event_amd_iommuv2.c |  429 ++++++++++++++++++++++++++
 arch/x86/kernel/cpu/perf_event_amd_iommuv2.h |   42 +++
 3 files changed, 472 insertions(+)
 create mode 100644 arch/x86/kernel/cpu/perf_event_amd_iommuv2.c
 create mode 100644 arch/x86/kernel/cpu/perf_event_amd_iommuv2.h

diff --git a/arch/x86/kernel/cpu/Makefile b/arch/x86/kernel/cpu/Makefile
index a0e067d..4872b99 100644
--- a/arch/x86/kernel/cpu/Makefile
+++ b/arch/x86/kernel/cpu/Makefile
@@ -32,6 +32,7 @@ obj-$(CONFIG_PERF_EVENTS)		+= perf_event.o
 
 ifdef CONFIG_PERF_EVENTS
 obj-$(CONFIG_CPU_SUP_AMD)		+= perf_event_amd.o
+obj-$(CONFIG_AMD_IOMMU_V2_PC)           += perf_event_amd_iommuv2.o
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_p6.o perf_event_knc.o perf_event_p4.o
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_lbr.o perf_event_intel_ds.o perf_event_intel.o
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_uncore.o
diff --git a/arch/x86/kernel/cpu/perf_event_amd_iommuv2.c b/arch/x86/kernel/cpu/perf_event_amd_iommuv2.c
new file mode 100644
index 0000000..7c7b3ff
--- /dev/null
+++ b/arch/x86/kernel/cpu/perf_event_amd_iommuv2.c
@@ -0,0 +1,429 @@
+ /*
+ * Performance events - AMD IOMMUv2
+ *
+ * Copyright (x) 2012 Advanced Micro Devices, Inc., Steven Kinney
+ *
+ * For licensing details see kernel-base/COPYING
+ */
+#include <linux/perf_event.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/ptrace.h>
+
+#include "perf_event.h"
+#include "perf_event_amd_iommuv2.h"
+
+static u64 cntr_assign_mask;
+
+#if defined(CONFIG_PERF_EVENTS) && defined(CONFIG_AMD_IOMMU_V2_PC)
+
+/* define iommuv2 states based on ibs, add states if needed */
+enum iommuv2_states {
+	IOMMU_V2_PC_ENABLED	= 0,
+	IOMMU_V2_PC_STARTED	= 1,
+	IOMMU_V2_PC_STOPPING	= 2,
+
+	IOMMU_V2_PC_MAX_STATES,
+};
+
+
+struct cpu_perf_iommuv2 {
+	struct perf_event	*event;
+	unsigned long		state[BITS_TO_LONGS(IOMMU_V2_PC_MAX_STATES)];
+};
+
+
+struct perf_iommuv2 {
+	struct pmu	pmu;
+	struct attribute **format_attrs;
+	struct attribute_group format_group;
+	const struct attribute_group *attr_groups[11];
+	u16		iommuv2_devid;
+	u8		max_banks;
+	u8		max_counters;
+
+	struct cpu_perf_iommuv2  __percpu *pcpu;
+};
+
+
+PMU_FORMAT_ATTR(iommuv2_raw, "config:63");
+PMU_FORMAT_ATTR(csource, "config:7-0");
+PMU_FORMAT_ATTR(deviceid, "config:23-8");
+PMU_FORMAT_ATTR(pasid, "config:39-24");
+PMU_FORMAT_ATTR(domain, "config:55-40");
+PMU_FORMAT_ATTR(en_deviceid_filter, "config:56");
+PMU_FORMAT_ATTR(en_pasid_filter, "config:57");
+PMU_FORMAT_ATTR(en_domain_filter, "config:58");
+PMU_FORMAT_ATTR(deviceid_mask, "config1:15-0");
+PMU_FORMAT_ATTR(pasid_mask, "config1:31-16");
+PMU_FORMAT_ATTR(domain_mask, "config1:47-32");
+
+
+static struct attribute *iommuv2_fetch_format_attrs[] = {
+	&format_attr_iommuv2_raw.attr,
+	&format_attr_csource.attr,
+	&format_attr_deviceid.attr,
+	&format_attr_pasid.attr,
+	&format_attr_domain.attr,
+	&format_attr_en_deviceid_filter.attr,
+	&format_attr_en_pasid_filter.attr,
+	&format_attr_en_domain_filter.attr,
+	&format_attr_deviceid_mask.attr,
+	&format_attr_pasid_mask.attr,
+	&format_attr_domain_mask.attr,
+	NULL,
+};
+
+
+static struct perf_iommuv2 perf_iommuv2_fetch;
+
+
+static u64 read_iommu_v2_pc_icounter(u16 devid, u8 bank, u8 counter)
+{
+	long long val = 0;
+
+	/* modify to calculate pc offsets within this level */
+	amd_iommu_v2_get_set_pc_reg_val(devid,
+					bank,
+					counter,
+					IOMMUV2_PC_COUNTER_REG,
+					&val,
+					false);
+
+	return val;
+}
+
+
+static u16 get_next_avail_iommuv2_bnk_cntr(struct perf_iommuv2 *perf_iommuv2)
+{
+	int bank_index, cntr_index;
+	int max_banks, max_cntrs;
+	int shift = 0;
+	int bank_offset = 0;
+	u16 retval;
+
+	max_banks = perf_iommuv2->max_banks;
+	max_cntrs = perf_iommuv2->max_counters;
+
+	for (bank_index = 0; bank_index < max_banks; bank_index++) {
+		for (cntr_index = 0; cntr_index < max_cntrs; cntr_index++) {
+			shift = bank_index + cntr_index + bank_offset;
+			if (cntr_assign_mask & (1ULL<<shift))
+				continue;
+			else {
+				cntr_assign_mask |= (1ULL<<shift);
+				retval = ((u16)((u16)bank_index<<8) |
+					  (u8)(cntr_index));
+				goto out;
+			}
+		}
+		bank_offset += 3;
+	}
+	retval = -EINVAL;
+out:
+	return retval;
+}
+
+
+static int clear_avail_iommuv2_bnk_cntr(struct perf_iommuv2 *perf_iommuv2,
+					u8 bank,
+					u8 cntr)
+{
+	int max_banks, max_cntrs;
+	int shift = 0;
+
+	max_banks = perf_iommuv2->max_banks;
+	max_cntrs = perf_iommuv2->max_counters;
+
+	if ((bank > max_banks) || (cntr > max_cntrs))
+		return -EINVAL;
+
+	shift = bank + cntr + (bank*3);
+	cntr_assign_mask &= ~(1ULL<<shift);
+
+	return 0;
+}
+
+
+static int perf_iommuv2_init(struct perf_event *event)
+{
+	struct hw_perf_event *hwc = &event->hw;
+	struct perf_iommuv2 *perf_iommuv2;
+	u64 config, config1;
+
+	/* initialize the cntr_assign_mask */
+	cntr_assign_mask = 0;
+
+	/* test the event attr type check for PMU enumeration */
+	if (event->attr.type != event->pmu->type)
+		return -ENOENT;
+
+	perf_iommuv2 = &perf_iommuv2_fetch;
+
+	if (perf_iommuv2) {
+		config = event->attr.config;
+		config1 = event->attr.config1;
+	} else
+		return -EINVAL;
+
+	if (event->pmu != &perf_iommuv2->pmu)
+		return -ENOENT;
+
+	/* make sure the event->attr.config is raw (vendor specific) */
+	if (!(config & IOMMU_V2_PC_RAW_CONFIG_MASK))
+		return -EINVAL;
+
+	/* integrate with iommu base devid (0000), assume one iommu */
+	perf_iommuv2->max_banks =
+		amd_iommu_v2_get_max_pc_banks(IOMMU_V2_BASE_DEVID);
+	perf_iommuv2->max_counters =
+		amd_iommu_v2_get_max_pc_counters(IOMMU_V2_BASE_DEVID);
+
+	/* update the hw_perf_event struct with the iommuv2 config data */
+	hwc->config = config;
+	hwc->extra_reg.config = config1;
+
+	return 0;
+}
+
+
+static void perf_iommuv2_event_update(struct perf_event *event, u64 *config,
+				      u16 *bank_cntr)
+{
+	u64 count;
+	u16 devid;
+	u8  bank, cntr;
+	u64 prev_raw_count;
+	u64 delta;
+	struct hw_perf_event *hwc = &event->hw;
+
+	devid = (u16)(*config & IOMMU_V2_PC_DEVICEID_MATCH)>>8;
+	bank  = (u8)*bank_cntr >> 8;
+	cntr  = (u8)*bank_cntr;
+
+	count = read_iommu_v2_pc_icounter(devid, bank, cntr);
+
+	prev_raw_count =  local64_read(&hwc->prev_count);
+	if (local64_cmpxchg(&hwc->prev_count, prev_raw_count,
+					count) != prev_raw_count)
+		return;
+
+	delta = count - prev_raw_count;
+	local64_add(delta, &event->count);
+
+}
+
+
+static void perf_iommuv2_enable_event(struct hw_perf_event *hwc)
+{
+	u16 devid;
+	u8  bank, cntr, csource;
+	long long val;
+
+	devid   = (u16)(hwc->config & IOMMU_V2_PC_DEVICEID_MATCH)>>8;
+	bank    = (u8)hwc->extra_reg.reg >> 8;
+	cntr    = (u8)hwc->extra_reg.reg;
+	csource = (u8)(hwc->config & IOMMU_V2_PC_CSOURCE);
+
+	val = csource;
+
+	amd_iommu_v2_get_set_pc_reg_val(devid,
+					 bank,
+					 cntr,
+					 IOMMUV2_PC_COUNTER_SRC_REG,
+					 &val,
+					 true);
+}
+
+
+static void perf_iommuv2_disable_event(struct hw_perf_event *hwc)
+{
+	u16 devid;
+	u8  bank, cntr;
+	long long val;
+
+	devid = (u16)(hwc->config & IOMMU_V2_PC_DEVICEID_MATCH)>>8;
+	bank  = (u8)hwc->extra_reg.reg>>8;
+	cntr  = (u8)hwc->extra_reg.reg;
+
+	val = PC_CSOURCE_DISABLE_CNT;
+
+	amd_iommu_v2_get_set_pc_reg_val(devid,
+					 bank,
+					 cntr,
+					 IOMMUV2_PC_COUNTER_SRC_REG,
+					 &val,
+					 true);
+}
+
+
+static void perf_iommuv2_start(struct perf_event *event, int flags)
+{
+	struct hw_perf_event *hwc = &event->hw;
+	struct perf_iommuv2 *perf_iommuv2 =
+			container_of(event->pmu, struct perf_iommuv2, pmu);
+	struct cpu_perf_iommuv2 *pcpu = this_cpu_ptr(perf_iommuv2->pcpu);
+
+	if (WARN_ON_ONCE(!(hwc->state & PERF_HES_STOPPED)))
+		return;
+
+	WARN_ON_ONCE(!(hwc->state & PERF_HES_UPTODATE));
+	hwc->state = 0;
+
+	set_bit(IOMMU_V2_PC_STARTED, pcpu->state);
+	perf_iommuv2_enable_event(hwc);
+	perf_event_update_userpage(event);
+}
+
+
+static void perf_iommuv2_stop(struct perf_event *event, int flags)
+{
+	struct hw_perf_event *hwc = &event->hw;
+	struct perf_iommuv2 *perf_iommuv2 =
+			container_of(event->pmu, struct perf_iommuv2, pmu);
+	struct cpu_perf_iommuv2 *pcpu = this_cpu_ptr(perf_iommuv2->pcpu);
+	int stopping;
+	u16 bank_cntr;
+	u64 config;
+
+	stopping = test_and_clear_bit(IOMMU_V2_PC_STARTED, pcpu->state);
+
+	if (!stopping && (hwc->state & PERF_HES_UPTODATE))
+		return;
+
+	if (stopping) {
+		set_bit(IOMMU_V2_PC_STOPPING, pcpu->state);
+		perf_iommuv2_disable_event(hwc);
+		WARN_ON_ONCE(hwc->state & PERF_HES_STOPPED);
+		hwc->state |= PERF_HES_STOPPED;
+	}
+
+	if (hwc->state & PERF_HES_UPTODATE)
+		return;
+
+	config = hwc->config;
+	bank_cntr = (u16)hwc->extra_reg.reg;
+	perf_iommuv2_event_update(event, &config, &bank_cntr);
+	hwc->state |= PERF_HES_UPTODATE;
+}
+
+
+static int perf_iommuv2_add(struct perf_event *event, int flags)
+{
+	struct perf_iommuv2 *perf_iommuv2 =
+			container_of(event->pmu, struct perf_iommuv2, pmu);
+	struct cpu_perf_iommuv2 *pcpu = this_cpu_ptr(perf_iommuv2->pcpu);
+	u16 bank_cntr;
+
+	if (test_and_set_bit(IOMMU_V2_PC_ENABLED, pcpu->state))
+		return -ENOSPC;
+
+	event->hw.state = PERF_HES_UPTODATE | PERF_HES_STOPPED;
+
+	pcpu->event = event;
+
+	/* request an iommuv2 bank/counter */
+	bank_cntr = get_next_avail_iommuv2_bnk_cntr(perf_iommuv2);
+	event->hw.extra_reg.reg = bank_cntr;
+
+	if (flags & PERF_EF_START)
+		perf_iommuv2_start(event, PERF_EF_RELOAD);
+
+	return 0;
+}
+
+
+static void perf_iommuv2_del(struct perf_event *event, int flags)
+{
+	struct perf_iommuv2 *perf_iommuv2 =
+			container_of(event->pmu, struct perf_iommuv2, pmu);
+	struct cpu_perf_iommuv2 *pcpu = this_cpu_ptr(perf_iommuv2->pcpu);
+	u8 bank, cntr;
+
+	if (!test_and_clear_bit(IOMMU_V2_PC_ENABLED, pcpu->state))
+		return;
+
+	perf_iommuv2_stop(event, PERF_EF_UPDATE);
+
+	/* clear the assigned iommuv2 bank/counter */
+	bank = (u8)event->hw.extra_reg.reg >> 8;
+	cntr = (u8)event->hw.extra_reg.reg;
+	clear_avail_iommuv2_bnk_cntr(perf_iommuv2,
+				     bank,
+				     cntr);
+	pcpu->event = NULL;
+
+	perf_event_update_userpage(event);
+}
+
+
+static void perf_iommuv2_read(struct perf_event *event)
+{
+	pr_debug("AMD IOMMUv2: perf_iommuv2_read called\n");
+}
+
+
+static struct perf_iommuv2 perf_iommuv2_fetch = {
+	.pmu = {
+		.event_init	= perf_iommuv2_init,
+		.add		= perf_iommuv2_add,
+		.del		= perf_iommuv2_del,
+		.start		= perf_iommuv2_start,
+		.stop		= perf_iommuv2_stop,
+		.read		= perf_iommuv2_read,
+	},
+	.format_attrs		= iommuv2_fetch_format_attrs,
+	.max_banks		= 0x00,
+	.max_counters		= 0x00,
+};
+
+
+static __init int perf_iommuv2_pmu_init(struct perf_iommuv2 *perf_iommuv2,
+					char *name)
+{
+	struct cpu_perf_iommuv2 __percpu *pcpu;
+	int ret;
+
+	pcpu = alloc_percpu(struct cpu_perf_iommuv2);
+	if (!pcpu)
+		return -ENOMEM;
+
+	perf_iommuv2->pcpu = pcpu;
+
+	/* allocate and register atttributes */
+	if (perf_iommuv2->format_attrs[0]) {
+		memset(&perf_iommuv2->format_group, 0,
+			sizeof(perf_iommuv2->format_group));
+		perf_iommuv2->format_group.name = "format";
+		perf_iommuv2->format_group.attrs = perf_iommuv2->format_attrs;
+		memset(&perf_iommuv2->attr_groups, 0,
+			sizeof(perf_iommuv2->attr_groups));
+		perf_iommuv2->attr_groups[0] = &perf_iommuv2->format_group;
+		perf_iommuv2->pmu.attr_groups = perf_iommuv2->attr_groups;
+	}
+
+	ret = perf_pmu_register(&perf_iommuv2->pmu, name, -1);
+	if (ret) {
+		pr_info("AMD IOMMUv2 PMU - ERROR Registering IOMMUv2 Perf PMU\n");
+		perf_iommuv2->pcpu = NULL;
+		free_percpu(pcpu);
+	}
+
+	return ret;
+}
+
+static __init int amd_iommuv2_init(void)
+{
+	/* Make sure the IOMMUv2 PC resource is available */
+	if (!amd_iommu_v2_pc_supported()) {
+		pr_info("Perf IOMMUv2 - PMU not installed no support!\n");
+		return -ENODEV;
+	}
+
+	perf_iommuv2_pmu_init(&perf_iommuv2_fetch, "iommuv2");
+
+	return 0;
+}
+
+device_initcall(amd_iommuv2_init);
+#endif
diff --git a/arch/x86/kernel/cpu/perf_event_amd_iommuv2.h b/arch/x86/kernel/cpu/perf_event_amd_iommuv2.h
new file mode 100644
index 0000000..10d341e
--- /dev/null
+++ b/arch/x86/kernel/cpu/perf_event_amd_iommuv2.h
@@ -0,0 +1,42 @@
+
+/* iommuv2 pc mmio region register indexes */
+#define IOMMUV2_PC_COUNTER_REG			0x00
+#define IOMMUV2_PC_COUNTER_SRC_REG		0x08
+#define IOMMUV2_PC_PASID_MATCH_REG		0x10
+#define IOMMUV2_PC_DOMAIN_MATCH_REG		0x18
+#define IOMMUV2_PC_DEVICEID_MATCH_REG		0x20
+#define IOMMUV2_PC_COUNTER_REPORT_REG		0x28
+
+/* iommuv2 pc csource register values */
+#define PC_CSOURCE_DISABLE_CNT			0x00
+
+/* maximun specified bank/counters */
+#define PC_MAX_SPEC_BNKS			64
+#define PC_MAX_SPEC_CNTRS			16
+
+/* iommuv2 pmu config masks */
+#define IOMMU_V2_PC_RAW_CONFIG_MASK		(1ULL << 63)
+#define IOMMU_V2_PC_CSOURCE			(0xFFULL << 0)
+#define IOMMU_V2_PC_DEVICEID_MATCH		(0xFFFFULL << 8)
+#define IOMMU_V2_PC_PASID_MATCH			(0xFFFFULL << 24)
+#define IOMMU_V2_PC_DOMAIN_MATCH		(0xFFFFULL << 40)
+#define IOMMU_V2_PC_EN_DEVICEID			(1ULL << 56)
+#define IOMMU_V2_PC_EN_PASID			(1ULL << 57)
+#define IOMMU_V2_PC_EN_DOMAIN			(1ULL << 58)
+#define IOMMU_V2_PC_DEVICEID_MASK		(0xFFFFULL << 0)
+#define IOMMU_V2_PC_PASID_MASK			(0xFFFFULL << 16)
+#define IOMMU_V2_PC_DOMAIN_MASK			(0xFFFFULL << 32)
+
+#define IOMMU_V2_BASE_DEVID			0x0000
+
+/* amd_iommu_init.c external support functions */
+extern bool amd_iommu_v2_pc_supported(void);
+extern u8 amd_iommu_v2_get_max_pc_banks(u16 devid);
+extern u8 amd_iommu_v2_get_max_pc_counters(u16 devid);
+extern int amd_iommu_v2_get_set_pc_reg_val(
+					u16 devid,
+					u8 bank,
+					u8 cntr,
+					u8 fxn,
+					long long *value,
+					bool is_write);
-- 
1.7.9.5

Re: [PATCH 2/3] AMD IOMMUv2 PC resource management hooks

[Cyrill Gorcunov]

On Mon, Jan 21, 2013 at 02:20:57PM -0600, Steven L. Kinney wrote:
> +u8 amd_iommu_v2_get_max_pc_banks(u16 devid)
> +{
> +	struct amd_iommu *iommu;
> +
> +	/* locate the iommu governing the devid */
> +	iommu = amd_iommu_rlookup_table[devid];
> +
> +	if (iommu)
> +		return iommu->max_banks;
> +
> +	return -ENODEV;
> +}
> +EXPORT_SYMBOL(amd_iommu_v2_get_max_pc_banks);

Hi Steven, may not it be better to return int here, instead of u8?

RE: [PATCH 2/3] AMD IOMMUv2 PC resource management hooks

[Kinney, Steven]

Hi Cyrill,

                 Agree, will clean this up.

Thanks

-----Original Message-----
From: Cyrill Gorcunov [mailto:gorcunov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org] On Behalf Of Cyrill Gorcunov
Sent: Monday, January 21, 2013 3:32 PM
To: Kinney, Steven
Cc: Thomas Gleixner; Ingo Molnar; H. Peter Anvin; x86-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org; Joerg Roedel; Bjorn Helgaas; Greg Kroah-Hartman; Sebastian Andrzej Siewior; Myron Stowe; Hiroshi DOYU; Stephen Warren; Jiri Kosina; Kukjin Kim; linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; Peter Zijlstra; Paul Mackerras; Arnaldo Carvalho de Melo; Thomas Renninger; Andi Kleen
Subject: Re: [PATCH 2/3] AMD IOMMUv2 PC resource management hooks

On Mon, Jan 21, 2013 at 02:20:57PM -0600, Steven L. Kinney wrote:
> +u8 amd_iommu_v2_get_max_pc_banks(u16 devid) {
> +	struct amd_iommu *iommu;
> +
> +	/* locate the iommu governing the devid */
> +	iommu = amd_iommu_rlookup_table[devid];
> +
> +	if (iommu)
> +		return iommu->max_banks;
> +
> +	return -ENODEV;
> +}
> +EXPORT_SYMBOL(amd_iommu_v2_get_max_pc_banks);

Hi Steven, may not it be better to return int here, instead of u8?

Re: [PATCH 2/3] AMD IOMMUv2 PC resource management hooks

[Cyrill Gorcunov]

On Mon, Jan 21, 2013 at 09:47:30PM +0000, Kinney, Steven wrote:
> Hi Cyrill,
> Agree, will clean this up.

Yeah, but no rush. Lets wait for other opinions as well.

Re: [PATCH 0/3] AMD IOMMUv2 Performance Counter patches

[Konrad Rzeszutek Wilk]

On Mon, Jan 21, 2013 at 02:20:55PM -0600, Steven L. Kinney wrote:
> From: "Steven L. Kinney" <steven.kinney-5C7GfCeVMHo@public.gmane.org>
> 
> These patches implement the AMD IOMMUv2.5 Performance Counter functionality
> via custom perf PMU and implement static counting for various IOMMU
> translations.  The patches address three areas of functionality:
> 
> 1) 	Add a PCI quirk for the enablement of IOMMUv2 EFR PC within a specific
> 	AMD family/model in which BIOS has not enabled.  Based on code 
> 	implemented by Andreas Herrmann at AMD Dresden.
> 2)	Extend the AMD IOMMU initialization to include IOMMUv2 PC enablement
> 	and access to IOMMUv2 counter registers.  For all AMD family/models
> 	that implement IOMMUv2.5 functionality.
> 3)	Code the perf IOMMUv2 PMU to manage IOMMUv2 perf events, which call
> 	the function(s) extending the AMD IOMMU core driver.  For all AMD
> 	family/models that implement IOMMUv2.5 functionality.
> 
> The command-line, to invoke the iommuv2 PMU, is:
> 
> perf stat -e iommuv2/config=[data],config1=[data]/{u,r} [command]
> 
> For information regarding AMD IOMMU v2.5 PC configuration, see the public
> specification.

Is there a name for it? I see
http://support.amd.com/us/Processor_TechDocs/48882.pdf
but that looks to be for v2.0?
> 
> Steven L. Kinney (3):
>   AMD x86 quirks: Quirk for enabling IOMMUv2 PC feature
>   AMD IOMMUv2 PC resource management hooks
>   AMD IOMMUv2 PC perf PMU implementation
> 
>  arch/x86/kernel/cpu/Makefile                 |    1 +
>  arch/x86/kernel/cpu/perf_event_amd_iommuv2.c |  429 ++++++++++++++++++++++++++
>  arch/x86/kernel/cpu/perf_event_amd_iommuv2.h |   42 +++
>  arch/x86/kernel/quirks.c                     |   17 +
>  drivers/iommu/Kconfig                        |   10 +
>  drivers/iommu/amd_iommu_init.c               |   99 ++++++
>  drivers/iommu/amd_iommu_types.h              |   12 +
>  include/linux/pci_ids.h                      |    2 +
>  8 files changed, 612 insertions(+)
>  create mode 100644 arch/x86/kernel/cpu/perf_event_amd_iommuv2.c
>  create mode 100644 arch/x86/kernel/cpu/perf_event_amd_iommuv2.h
> 
> -- 
> 1.7.9.5
> 
> 
> _______________________________________________
> iommu mailing list
> iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
> https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: [PATCH 0/3] AMD IOMMUv2 Performance Counter patches

[Kinney, Steven]

Hi Konrad,

                      Thanks for your response.  When writing the driver, at AMD, I am provided an internal IOMMUv2.x specification that may not be out in the public as of yet; I am following up on this and will comment on the availability concerning this version.   Also, I checked the v2.0 of the IOMMU Specification, specifically the following:

MMIO Offset [40-7F][0-F]08h IOMMU Counter Source Register - Table 64: Architectural Counter Input Group : This is the CSource and the format can be found in /sys/bus/event_sources/devices/iommuv2/forrmat/
MMIO Offset [40-7F][0-F]20h IOMMU DeviceID Match Register - For the following three registers, the config value defines the Match and config1 value defines the Mask; see the specification v2.0 for details; also see sys/...../format for details.
MMIO Offset [40-7F][0-F]10h IOMMU PASID Match Register - 
MMIO Offset [40-7F][0-F]18h IOMMU Domain Match Register - 

The one caveat between the v2.0 and v2.5 concerns a slight expansion of the CSource register Input Group, but any event defined within v2.0 will execute properly regarding the driver patch(es) sent to the kernel community.

Please let me know, in any way, how I can better this driver code and I will make it so.  Also, Will have an answer for you regarding AMD IOMMUv2.5 ASAP.

Thanks again for your comments, appreciated.

-----Original Message-----
From: Konrad Rzeszutek Wilk [mailto:konrad.wilk-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org] 
Sent: Tuesday, January 22, 2013 11:48 AM
To: Kinney, Steven
Cc: Thomas Gleixner; Ingo Molnar; H. Peter Anvin; x86-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org; Joerg Roedel; Kukjin Kim; Peter Zijlstra; Stephen Warren; Greg Kroah-Hartman; linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; Cyrill Gorcunov; Sebastian Andrzej Siewior; Andi Kleen; Paul Mackerras; Arnaldo Carvalho de Melo; Jiri Kosina; Bjorn Helgaas; Myron Stowe; Thomas Renninger
Subject: Re: [PATCH 0/3] AMD IOMMUv2 Performance Counter patches

On Mon, Jan 21, 2013 at 02:20:55PM -0600, Steven L. Kinney wrote:
> From: "Steven L. Kinney" <steven.kinney-5C7GfCeVMHo@public.gmane.org>
> 
> These patches implement the AMD IOMMUv2.5 Performance Counter 
> functionality via custom perf PMU and implement static counting for 
> various IOMMU translations.  The patches address three areas of functionality:
> 
> 1) 	Add a PCI quirk for the enablement of IOMMUv2 EFR PC within a specific
> 	AMD family/model in which BIOS has not enabled.  Based on code 
> 	implemented by Andreas Herrmann at AMD Dresden.
> 2)	Extend the AMD IOMMU initialization to include IOMMUv2 PC enablement
> 	and access to IOMMUv2 counter registers.  For all AMD family/models
> 	that implement IOMMUv2.5 functionality.
> 3)	Code the perf IOMMUv2 PMU to manage IOMMUv2 perf events, which call
> 	the function(s) extending the AMD IOMMU core driver.  For all AMD
> 	family/models that implement IOMMUv2.5 functionality.
> 
> The command-line, to invoke the iommuv2 PMU, is:
> 
> perf stat -e iommuv2/config=[data],config1=[data]/{u,r} [command]
> 
> For information regarding AMD IOMMU v2.5 PC configuration, see the 
> public specification.

Is there a name for it? I see
http://support.amd.com/us/Processor_TechDocs/48882.pdf
but that looks to be for v2.0?
> 
> Steven L. Kinney (3):
>   AMD x86 quirks: Quirk for enabling IOMMUv2 PC feature
>   AMD IOMMUv2 PC resource management hooks
>   AMD IOMMUv2 PC perf PMU implementation
> 
>  arch/x86/kernel/cpu/Makefile                 |    1 +
>  arch/x86/kernel/cpu/perf_event_amd_iommuv2.c |  429 ++++++++++++++++++++++++++
>  arch/x86/kernel/cpu/perf_event_amd_iommuv2.h |   42 +++
>  arch/x86/kernel/quirks.c                     |   17 +
>  drivers/iommu/Kconfig                        |   10 +
>  drivers/iommu/amd_iommu_init.c               |   99 ++++++
>  drivers/iommu/amd_iommu_types.h              |   12 +
>  include/linux/pci_ids.h                      |    2 +
>  8 files changed, 612 insertions(+)
>  create mode 100644 arch/x86/kernel/cpu/perf_event_amd_iommuv2.c
>  create mode 100644 arch/x86/kernel/cpu/perf_event_amd_iommuv2.h
> 
> --
> 1.7.9.5
> 
> 
> _______________________________________________
> iommu mailing list
> iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
> https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 1/3] AMD x86 quirks: Quirk for enabling IOMMUv2 PC feature

[Joerg Roedel]

On Mon, Jan 21, 2013 at 02:20:56PM -0600, Steven L. Kinney wrote:
> From: "Steven L. Kinney" <steven.kinney-5C7GfCeVMHo@public.gmane.org>
> 
> Original code base by Andreas Herrmann at AMD with minor changes.

Why is it safe to enable this feature when it was disabled by BIOS? Was
it disabled for no reason?


	Joerg

RE: [PATCH 1/3] AMD x86 quirks: Quirk for enabling IOMMUv2 PC feature

[Kinney, Steven]

Hi Joerg,

                       It was disabled for a reason that does not pertain to IOMMUv2 PC static counts (reading from the ICounter associated with the selected bank/counter).  Testing with perf shows expected results.

Thanks,

Steve

-----Original Message-----
From: Joerg Roedel [mailto:joro@8bytes.org] 
Sent: Monday, January 28, 2013 8:29 AM
To: Kinney, Steven
Cc: Thomas Gleixner; Ingo Molnar; H. Peter Anvin; x86@kernel.org; Bjorn Helgaas; Greg Kroah-Hartman; Sebastian Andrzej Siewior; Myron Stowe; Hiroshi DOYU; Stephen Warren; Jiri Kosina; Kukjin Kim; linux-kernel@vger.kernel.org; iommu@lists.linux-foundation.org; Peter Zijlstra; Paul Mackerras; Arnaldo Carvalho de Melo; Thomas Renninger; Andi Kleen; Cyrill Gorcunov
Subject: Re: [PATCH 1/3] AMD x86 quirks: Quirk for enabling IOMMUv2 PC feature

On Mon, Jan 21, 2013 at 02:20:56PM -0600, Steven L. Kinney wrote:
> From: "Steven L. Kinney" <steven.kinney@amd.com>
> 
> Original code base by Andreas Herrmann at AMD with minor changes.

Why is it safe to enable this feature when it was disabled by BIOS? Was it disabled for no reason?


	Joerg

Re: [PATCH 1/3] AMD x86 quirks: Quirk for enabling IOMMUv2 PC feature

[Joerg Roedel]

On Mon, Jan 28, 2013 at 02:59:25PM +0000, Kinney, Steven wrote:
> Testing with perf shows expected results.

Can you give me an impression on how the results look like when perf is
used? Since the hardware is widely available yet I can't try this
myself.


	Joerg

RE: [PATCH 1/3] AMD x86 quirks: Quirk for enabling IOMMUv2 PC feature

[Kinney, Steven]

Hi Joerg,

                    Sorry for the delay regarding the response.  I can describe the invocation and the results, pertaining to static counts.  Also, I would imagine that driver writers or individuals wanting to measure IOMMU translation performance would be the consumers regarding this perf capability.  Of course, this is my understanding and why I am very interested in the kernel communities comments and advice.  First, to invoke the use of the IOMMUv2 PMU the following command will suffice:

	./perf stat -e iommuv2/config=0x8000000000000005,config1=0x0/u <command>  /* I have the RAW bit explicitly set (MSb) */

	The <config> will set the following:

		CSource [7:0] - Identifies the IOMMUv2 performance metric that will be counted.  In this case 0x05 which is the total peripheral memory operations translated.
		DeviceID [23:8] - The PCI BDF identifying the specific device that will be considered.  In this case 0x0000 is the IOMMU itself.
		PASID [39:24] - Filter based on PASID, optional.  0x0000, no filtering
		Domain [55:40] - Filter based on Domain, optional, 0x0000 no filtering.
		en_deviceid_filter[56] - Explicit enabling of DeviceID filtering, implicitly set if DeviceID is not 0x0000.
		en_pasid_filter[57] - Must be set to enable optional PASID filtering.
		en_domain_filter [58] - Must be set to enable optional Domain filtering.

	The <config1> will set the following (more obscure settings)

		deviceid_mask [15:0] - Apply a bit mask, regarding the associated filter, or match register, for refining purposes.
		pasid_mask [31:16] - Same as device_mask pertaining to PASID.
		domain_mask [47:32] - Same as device_mask, pertaining to Domain.

When the IOMMUv2 PMU is invoked, the first task is to verify there is a PC resource available.  The IOMMUv2 PMU uses a soft register and bit mask, linearized from bank/counter information populated within the amd_iommu struct during initialization, to allocate a free bank/counter to assign to the perf IOMMU event.  The bank/counter information is used, among other values, to calculate an offset into the IOMMU MMIO region to access registers; for example ICounter, CSource, etc.  So from an IOMMUv2 driver perspective, pertaining to the additional functionality written into amd_iommu_init.c, once the IOMMUv2 PMU has assigned the counter resource it needs to configure the physical IOMMUv2 PC registers.  For example,:

		1) Allocate IOMMUv2 Bank/Counter index, first go-around the assignment is bank=0, counter=0.
		2) At the moment, the code is only populating the DevID (PCI BDF) into DeviceID; PASID and Domain will be added later.   The devid is held to 0x0000.
		3) The Fxn is the functional register within the counter set and is used to calculate the counter register offset within the MMIO Region.  For example CSource is +08h; see Table 70: Counter Bank Addressing (MMIO) in IOMMUv2 2.0 specification.
		4) The value to be written, in the case of the above example, is 0x05, pertaining to the CSource register.
		5) Since this is a write operation is_write is true.
		6) Now there is enough information to access the IOMMUv2 PC register(s) and the perf IOMMUv2 calls into the IOMMU core driver (exported function)

			Int amd_iommu_v2_get_set_pc_reg_val( u16 devid, u8 bank, u8 cntr, u8 fxn, long long *value, bool is_write);

		Most of the IOMMUv2 driver functionality is self-explanatory, and the function, above, will verify IOMMUv2 PC capability, calculate the counter set offset within the IOMMU MMIO region and verify that the offset is within the MMIO region aperture.  After this is completed, the function simply writes to the selected register.  Since the number of banks and counters are dynamic, dependent upon future design, the limits for MMIO region offset values are calculated based on reported maximum bank/counter.

After the CSource register has been written to, other than a zero(0), the ICounter will start counting the relative IOMMU events described by the CSource value.

To stop the counter (ICounter), the CSource register is set to zero(0); so a perf event accessing the IOMMUv2 PC will write a defined value to the CSource register, execute a command, write a zero(0) to the CSource register then read the ICounter value.  The count, for the specific IOMMU perf event, is the previous count minus the current ICounter value; the ICounter cannot be reset other than overflow.

So, when the perf command example is executed, for example with a ls or some other trivial executable, the result will be a count of all IOMMU peripheral memory operations translated (total).  I choose this simply to assure count increment.  

Sorry for the long winded explanation, but we can look at any detail you would like to explore regarding the above description.

BR,

Steve


-----Original Message-----
From: Joerg Roedel [mailto:joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org] 
Sent: Monday, January 28, 2013 9:37 AM
To: Kinney, Steven
Cc: Thomas Gleixner; Ingo Molnar; H. Peter Anvin; x86-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org; Bjorn Helgaas; Greg Kroah-Hartman; Sebastian Andrzej Siewior; Myron Stowe; Hiroshi DOYU; Stephen Warren; Jiri Kosina; Kukjin Kim; linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; Peter Zijlstra; Paul Mackerras; Arnaldo Carvalho de Melo; Thomas Renninger; Andi Kleen; Cyrill Gorcunov
Subject: Re: [PATCH 1/3] AMD x86 quirks: Quirk for enabling IOMMUv2 PC feature

On Mon, Jan 28, 2013 at 02:59:25PM +0000, Kinney, Steven wrote:
> Testing with perf shows expected results.

Can you give me an impression on how the results look like when perf is used? Since the hardware is widely available yet I can't try this myself.


	Joerg