[PATCH V10 0/6] Enable PMUs in ACPI Systems

[PATCH V10 0/6] Enable PMUs in ACPI Systems

[Jeremy Linton]

This patch expands and reworks the patches published by Mark Salter
in order to clean up a few of the previous review comments, as well as
add support for newer CPUs and big/little configurations.

v10:
- Rebase to 4.9
- Rework the arm_perf_start_cpu changes to support the 4.9 hotplug
  changes.
- Remove the call to acpi_register_gsi() from the cpu online code path.
  Instead the GSI's are registered during the initcall. This changes
  the error handling a bit because we now try to clean up the
  previously registered GSIs in a couple important places. This
  was also a result of the rebase.
- Dropped the MIDR partnumber usage, its no longer necessary to
  differentiate by only the partnum, so this helps to clarify the code
  a bit. 
- Shuffle some code around and rename a few variables.
- Added a few comments to hopefully clarify some questions people have
  previously had about unused MADT entries, skipping processing cores
  with MIDR=0, etc.

v9:
- Add/cleanup an additional hotplug patch I've had sitting around. This
  patch brings the ACPI PMU mostly on par with the DT functionality with
  respect to having CPUs offline during boot. This should help clarify
  some of the code structuring.
- Cleanup the list of PMU types early if we fail to allocate memory for an
  additional pmu type.

v8:
- Rebase to 4.8rc4
- Assorted minor comment/hunk placement/etc tweaks per Punit Agrawal

v7:
- Rebase to 4.8rc3
- Remove cpu affinity sysfs entry. While providing a CPU mask for
  ARMv8 PMU's is really helpful in big/little environments, reworking
  the PMU code to support the cpumask attribute for !arm64 PMUs is out
  of the scope of this patch set.
- Fix CPU miscount problem where an alloc failure followed by successfully
  allocating the structure can result in under counting the CPUs associated
  with the PMU. This bug was created in v6 with the conversion to a linked
  list.
-  Remove initial platform device creation code by Mark Salter, and re-squash
   multiple platform device creation code together with helper routines.
   Other minor tweakage.

v6:
- Added cpu affinity sysfs entry
- Converted pmu_types array, to linked list
- Restrict use of the armv8_pmu_probe_table to ACPI systems
- Rename MADT parsing routines in smp.c
- Convert sysfs PMU name to use index rather than partnum
- Remove pr_devel statements
- Other Minor cleanups
- Add Partial Ack-by Will Deacon

v5:
- Remove list of CPU types for ACPI systems. We now match a generic
  event list, and use the PMCIED[01] to select events which exist on
  the given PMU. This avoids the need to update the kernel every time
  a new CPU is released.
- Update the maintainers list to include the new file.

v4:
- Correct build issues with ARM (!ARM64) kernels.
- Add ThunderX to list of PMU types.

v3:
- Enable ARM performance monitoring units on ACPI/arm64 machines.

Jeremy Linton (5):
  arm64: Rename the common MADT parse routine
  arm: arm64: Add routine to determine cpuid of other cpus
  arm: arm64: pmu: Assign platform PMU CPU affinity
  arm64: pmu: Detect and enable multiple PMUs in an ACPI system
  arm: pmu: Add PMU definitions for cores not initially online

Mark Salter (1):
  arm64: pmu: Cache PMU interrupt numbers from MADT parse

 arch/arm/include/asm/cputype.h   |   2 +
 arch/arm64/include/asm/cputype.h |   3 +
 arch/arm64/kernel/smp.c          |  18 ++-
 drivers/perf/Kconfig             |   4 +
 drivers/perf/Makefile            |   1 +
 drivers/perf/arm_pmu.c           | 107 +++++++++++++--
 drivers/perf/arm_pmu_acpi.c      | 272 +++++++++++++++++++++++++++++++++++++++
 include/linux/perf/arm_pmu.h     |  11 ++
 8 files changed, 400 insertions(+), 18 deletions(-)
 create mode 100644 drivers/perf/arm_pmu_acpi.c

-- 
2.5.5

[PATCH V10 1/6] arm64: Rename the common MADT parse routine

[Jeremy Linton]

The MADT parser in smp.c is now being used to parse
out NUMA, PMU and ACPI parking protocol information as
well as the GIC information for which it was originally
created. Rename it to avoid a misleading name.

Signed-off-by: Jeremy Linton <jeremy.linton@arm.com>
---
 arch/arm64/kernel/smp.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 8507703..f3f1c90 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -511,13 +511,14 @@ static unsigned int cpu_count = 1;
 
 #ifdef CONFIG_ACPI
 /*
- * acpi_map_gic_cpu_interface - parse processor MADT entry
+ * acpi_verify_and_map_madt - parse processor MADT entry
  *
  * Carry out sanity checks on MADT processor entry and initialize
- * cpu_logical_map on success
+ * cpu_logical_map, the ACPI parking protocol, NUMA mapping
+ * and the PMU interrupts on success
  */
 static void __init
-acpi_map_gic_cpu_interface(struct acpi_madt_generic_interrupt *processor)
+acpi_verify_and_map_madt(struct acpi_madt_generic_interrupt *processor)
 {
 	u64 hwid = processor->arm_mpidr;
 
@@ -571,7 +572,7 @@ acpi_map_gic_cpu_interface(struct acpi_madt_generic_interrupt *processor)
 }
 
 static int __init
-acpi_parse_gic_cpu_interface(struct acpi_subtable_header *header,
+acpi_parse_madt_common(struct acpi_subtable_header *header,
 			     const unsigned long end)
 {
 	struct acpi_madt_generic_interrupt *processor;
@@ -582,7 +583,7 @@ acpi_parse_gic_cpu_interface(struct acpi_subtable_header *header,
 
 	acpi_table_print_madt_entry(header);
 
-	acpi_map_gic_cpu_interface(processor);
+	acpi_verify_and_map_madt(processor);
 
 	return 0;
 }
@@ -666,7 +667,7 @@ void __init smp_init_cpus(void)
 		 * we need for SMP init
 		 */
 		acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_INTERRUPT,
-				      acpi_parse_gic_cpu_interface, 0);
+				      acpi_parse_madt_common, 0);
 
 	if (cpu_count > nr_cpu_ids)
 		pr_warn("Number of cores (%d) exceeds configured maximum of %d - clipping\n",
-- 
2.5.5

[PATCH V10 2/6] arm: arm64: Add routine to determine cpuid of other cpus

[Jeremy Linton]

It is helpful if we can read the cpuid/midr of other CPUs
in the system independent of arm/arm64.

Signed-off-by: Jeremy Linton <jeremy.linton@arm.com>
---
 arch/arm/include/asm/cputype.h   | 2 ++
 arch/arm64/include/asm/cputype.h | 3 +++
 2 files changed, 5 insertions(+)

diff --git a/arch/arm/include/asm/cputype.h b/arch/arm/include/asm/cputype.h
index 522b5fe..31fb273 100644
--- a/arch/arm/include/asm/cputype.h
+++ b/arch/arm/include/asm/cputype.h
@@ -235,6 +235,8 @@ static inline unsigned int __attribute_const__ read_cpuid_mpidr(void)
 #define cpu_is_sa1100() (read_cpuid_part() == ARM_CPU_PART_SA1100)
 #define cpu_is_sa1110() (read_cpuid_part() == ARM_CPU_PART_SA1110)
 
+#define read_specific_cpuid(cpu_num) per_cpu_ptr(&cpu_data, cpu_num)->cpuid
+
 /*
  * Intel's XScale3 core supports some v6 features (supersections, L2)
  * but advertises itself as v5 as it does not support the v6 ISA.  For
diff --git a/arch/arm64/include/asm/cputype.h b/arch/arm64/include/asm/cputype.h
index 26a68dd..a6d26e1 100644
--- a/arch/arm64/include/asm/cputype.h
+++ b/arch/arm64/include/asm/cputype.h
@@ -124,6 +124,9 @@ static inline u32 __attribute_const__ read_cpuid_cachetype(void)
 {
 	return read_cpuid(CTR_EL0);
 }
+
+#define read_specific_cpuid(cpu_num) per_cpu_ptr(&cpu_data, cpu_num)->reg_midr
+
 #endif /* __ASSEMBLY__ */
 
 #endif
-- 
2.5.5

[PATCH V10 3/6] arm64: pmu: Cache PMU interrupt numbers from MADT parse

[Jeremy Linton]

From: Mark Salter <msalter@redhat.com>

In the case of ACPI, the PMU IRQ information is contained in the
MADT table. Also, since the PMU does not exist as a device in the
ACPI DSDT table, it is necessary to create a platform device so
that the appropriate driver probing is triggered. Since the platform
device creation needs to happen after the CPU's have been started, and
the MADT parsing needs to happen before, we save off the interrupt
numbers discovered during the parsing.

Signed-off-by: Mark Salter <msalter@redhat.com>
Signed-off-by: Jeremy Linton <jeremy.linton@arm.com>
---
 arch/arm64/kernel/smp.c      |  5 +++++
 drivers/perf/Kconfig         |  4 ++++
 drivers/perf/Makefile        |  1 +
 drivers/perf/arm_pmu_acpi.c  | 40 ++++++++++++++++++++++++++++++++++++++++
 include/linux/perf/arm_pmu.h |  7 +++++++
 5 files changed, 57 insertions(+)
 create mode 100644 drivers/perf/arm_pmu_acpi.c

diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index f3f1c90..e350ccc 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -37,6 +37,7 @@
 #include <linux/completion.h>
 #include <linux/of.h>
 #include <linux/irq_work.h>
+#include <linux/perf/arm_pmu.h>
 
 #include <asm/alternative.h>
 #include <asm/atomic.h>
@@ -546,6 +547,7 @@ acpi_verify_and_map_madt(struct acpi_madt_generic_interrupt *processor)
 		}
 		bootcpu_valid = true;
 		early_map_cpu_to_node(0, acpi_numa_get_nid(0, hwid));
+		arm_pmu_parse_acpi(0, processor);
 		return;
 	}
 
@@ -566,6 +568,9 @@ acpi_verify_and_map_madt(struct acpi_madt_generic_interrupt *processor)
 	 */
 	acpi_set_mailbox_entry(cpu_count, processor);
 
+	/* get PMU irq info */
+	arm_pmu_parse_acpi(cpu_count, processor);
+
 	early_map_cpu_to_node(cpu_count, acpi_numa_get_nid(cpu_count, hwid));
 
 	cpu_count++;
diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig
index 4d5c5f9..697df05 100644
--- a/drivers/perf/Kconfig
+++ b/drivers/perf/Kconfig
@@ -19,4 +19,8 @@ config XGENE_PMU
         help
           Say y if you want to use APM X-Gene SoC performance monitors.
 
+config ARM_PMU_ACPI
+	def_bool y
+	depends on ARM_PMU && ACPI
+
 endmenu
diff --git a/drivers/perf/Makefile b/drivers/perf/Makefile
index b116e98..d1d7762 100644
--- a/drivers/perf/Makefile
+++ b/drivers/perf/Makefile
@@ -1,2 +1,3 @@
 obj-$(CONFIG_ARM_PMU) += arm_pmu.o
 obj-$(CONFIG_XGENE_PMU) += xgene_pmu.o
+obj-$(CONFIG_ARM_PMU_ACPI) += arm_pmu_acpi.o
diff --git a/drivers/perf/arm_pmu_acpi.c b/drivers/perf/arm_pmu_acpi.c
new file mode 100644
index 0000000..135851c
--- /dev/null
+++ b/drivers/perf/arm_pmu_acpi.c
@@ -0,0 +1,40 @@
+/*
+ * ARM ACPI PMU support
+ *
+ * Copyright (C) 2015 Red Hat Inc.
+ * Author: Mark Salter <msalter@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ */
+
+#include <asm/cpu.h>
+#include <linux/acpi.h>
+#include <linux/irq.h>
+#include <linux/irqdesc.h>
+#include <linux/list.h>
+#include <linux/perf/arm_pmu.h>
+#include <linux/platform_device.h>
+
+struct pmu_irq {
+	int  gsi;
+	int  trigger;
+	bool registered;
+};
+
+static struct pmu_irq pmu_irqs[NR_CPUS];
+
+/*
+ * Called from acpi_verify_and_map_madt()'s MADT parsing during boot.
+ * This routine saves off the GSI's and their trigger state for use when we are
+ * ready to build the PMU platform device.
+ */
+void __init arm_pmu_parse_acpi(int cpu, struct acpi_madt_generic_interrupt *gic)
+{
+	pmu_irqs[cpu].gsi = gic->performance_interrupt;
+	if (gic->flags & ACPI_MADT_PERFORMANCE_IRQ_MODE)
+		pmu_irqs[cpu].trigger = ACPI_EDGE_SENSITIVE;
+	else
+		pmu_irqs[cpu].trigger = ACPI_LEVEL_SENSITIVE;
+}
diff --git a/include/linux/perf/arm_pmu.h b/include/linux/perf/arm_pmu.h
index 8462da2..df1ba55 100644
--- a/include/linux/perf/arm_pmu.h
+++ b/include/linux/perf/arm_pmu.h
@@ -164,4 +164,11 @@ int arm_pmu_device_probe(struct platform_device *pdev,
 
 #endif /* CONFIG_ARM_PMU */
 
+#ifdef CONFIG_ARM_PMU_ACPI
+struct acpi_madt_generic_interrupt;
+void arm_pmu_parse_acpi(int cpu, struct acpi_madt_generic_interrupt *gic);
+#else
+#define arm_pmu_parse_acpi(a, b) do { } while (0)
+#endif /* CONFIG_ARM_PMU_ACPI */
+
 #endif /* __ARM_PMU_H__ */
-- 
2.5.5

[PATCH V10 4/6] arm: arm64: pmu: Assign platform PMU CPU affinity

[Jeremy Linton]

On systems with multiple PMU types the PMU to CPU affinity
needs to be detected and set. The CPU to interrupt affinity
should also be set.

Signed-off-by: Jeremy Linton <jeremy.linton@arm.com>
---
 drivers/perf/arm_pmu.c | 63 ++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 53 insertions(+), 10 deletions(-)

diff --git a/drivers/perf/arm_pmu.c b/drivers/perf/arm_pmu.c
index b37b572..6008be9 100644
--- a/drivers/perf/arm_pmu.c
+++ b/drivers/perf/arm_pmu.c
@@ -11,6 +11,7 @@
  */
 #define pr_fmt(fmt) "hw perfevents: " fmt
 
+#include <linux/acpi.h>
 #include <linux/bitmap.h>
 #include <linux/cpumask.h>
 #include <linux/cpu_pm.h>
@@ -24,6 +25,7 @@
 #include <linux/irq.h>
 #include <linux/irqdesc.h>
 
+#include <asm/cpu.h>
 #include <asm/cputype.h>
 #include <asm/irq_regs.h>
 
@@ -889,25 +891,67 @@ static void cpu_pmu_destroy(struct arm_pmu *cpu_pmu)
 }
 
 /*
- * CPU PMU identification and probing.
+ * CPU PMU identification and probing. Its possible to have
+ * multiple CPU types in an ARM machine. Assure that we are
+ * picking the right PMU types based on the CPU in question
  */
-static int probe_current_pmu(struct arm_pmu *pmu,
-			     const struct pmu_probe_info *info)
+static int probe_plat_pmu(struct arm_pmu *pmu,
+			     const struct pmu_probe_info *info,
+			     unsigned int pmuid)
 {
-	int cpu = get_cpu();
-	unsigned int cpuid = read_cpuid_id();
 	int ret = -ENODEV;
+	int cpu;
+	int aff_ctr = 0;
+	static int duplicate_pmus;
+	struct platform_device *pdev = pmu->plat_device;
+	int irq = platform_get_irq(pdev, 0);
 
-	pr_info("probing PMU on CPU %d\n", cpu);
+	if (irq >= 0 && !irq_is_percpu(irq)) {
+		pmu->irq_affinity = kcalloc(pdev->num_resources, sizeof(int),
+					    GFP_KERNEL);
+		if (!pmu->irq_affinity)
+			return -ENOMEM;
+	}
 
+	for_each_possible_cpu(cpu) {
+		unsigned int cpuid = read_specific_cpuid(cpu);
+
+		if (cpuid == pmuid) {
+			cpumask_set_cpu(cpu, &pmu->supported_cpus);
+			if (pmu->irq_affinity) {
+				pmu->irq_affinity[aff_ctr] = cpu;
+				aff_ctr++;
+			}
+		}
+	}
+
+	/* find the type of PMU given the CPU */
 	for (; info->init != NULL; info++) {
-		if ((cpuid & info->mask) != info->cpuid)
+		if ((pmuid & info->mask) != info->cpuid)
 			continue;
 		ret = info->init(pmu);
+		/*
+		 * if this pmu declaration is unspecified and we have
+		 * previously found a PMU on this platform then append
+		 * a PMU number to the pmu name. This avoids changing
+		 * the names of PMUs that are specific to a class of CPUs.
+		 * The assumption is that if we match a specific PMU in the
+		 * provided pmu_probe_info then it's unique, and another PMU
+		 * in the system will match a different entry rather than
+		 * needing the _number to assure its unique.
+		 */
+		if ((!info->cpuid) && (duplicate_pmus)) {
+			pmu->name = kasprintf(GFP_KERNEL, "%s_%d",
+					    pmu->name, duplicate_pmus);
+			if (!pmu->name) {
+				kfree(pmu->irq_affinity);
+				ret = -ENOMEM;
+			}
+		}
+		duplicate_pmus++;
 		break;
 	}
 
-	put_cpu();
 	return ret;
 }
 
@@ -1043,8 +1087,7 @@ int arm_pmu_device_probe(struct platform_device *pdev,
 		if (!ret)
 			ret = init_fn(pmu);
 	} else if (probe_table) {
-		cpumask_setall(&pmu->supported_cpus);
-		ret = probe_current_pmu(pmu, probe_table);
+		ret = probe_plat_pmu(pmu, probe_table, read_cpuid_id());
 	}
 
 	if (ret) {
-- 
2.5.5

[PATCH V10 5/6] arm64: pmu: Detect and enable multiple PMUs in an ACPI system

[Jeremy Linton]

Its possible that an ACPI system has multiple CPU types in it
with differing PMU counters. Iterate the CPU's and make a determination
about how many of each type exist in the system. Then take and create
a PMU platform device for each type, and assign it the interrupts parsed
from the MADT. Creating a platform device is necessary because the PMUs
are not described as devices in the DSDT table.

This code is loosely based on earlier work by Mark Salter.

Signed-off-by: Jeremy Linton <jeremy.linton@arm.com>
---
 drivers/perf/arm_pmu.c      |   8 +-
 drivers/perf/arm_pmu_acpi.c | 234 +++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 240 insertions(+), 2 deletions(-)

diff --git a/drivers/perf/arm_pmu.c b/drivers/perf/arm_pmu.c
index 6008be9..07e1404 100644
--- a/drivers/perf/arm_pmu.c
+++ b/drivers/perf/arm_pmu.c
@@ -1087,7 +1087,13 @@ int arm_pmu_device_probe(struct platform_device *pdev,
 		if (!ret)
 			ret = init_fn(pmu);
 	} else if (probe_table) {
-		ret = probe_plat_pmu(pmu, probe_table, read_cpuid_id());
+		if (acpi_disabled) {
+			/* use the current cpu. */
+			ret = probe_plat_pmu(pmu, probe_table,
+					     read_cpuid_id());
+		} else {
+			ret = probe_plat_pmu(pmu, probe_table, pdev->id);
+		}
 	}
 
 	if (ret) {
diff --git a/drivers/perf/arm_pmu_acpi.c b/drivers/perf/arm_pmu_acpi.c
index 135851c..eecf1c1 100644
--- a/drivers/perf/arm_pmu_acpi.c
+++ b/drivers/perf/arm_pmu_acpi.c
@@ -2,13 +2,17 @@
  * ARM ACPI PMU support
  *
  * Copyright (C) 2015 Red Hat Inc.
+ * Copyright (C) 2016 ARM Ltd.
  * Author: Mark Salter <msalter@redhat.com>
+ *	   Jeremy Linton <jeremy.linton@arm.com>
  *
  * This work is licensed under the terms of the GNU GPL, version 2.  See
  * the COPYING file in the top-level directory.
  *
  */
 
+#define pr_fmt(fmt) "ACPI-PMU: " fmt
+
 #include <asm/cpu.h>
 #include <linux/acpi.h>
 #include <linux/irq.h>
@@ -20,7 +24,14 @@
 struct pmu_irq {
 	int  gsi;
 	int  trigger;
-	bool registered;
+	int  irq;
+	bool used;
+};
+
+struct pmu_types {
+	struct list_head list;
+	int		 cpu_type;
+	int		 cpu_count;
 };
 
 static struct pmu_irq pmu_irqs[NR_CPUS];
@@ -38,3 +49,224 @@ void __init arm_pmu_parse_acpi(int cpu, struct acpi_madt_generic_interrupt *gic)
 	else
 		pmu_irqs[cpu].trigger = ACPI_LEVEL_SENSITIVE;
 }
+
+static void __init arm_pmu_acpi_handle_alloc_failure(struct list_head *pmus)
+{
+	int i;
+	struct pmu_types *pmu, *safe_temp;
+
+	list_for_each_entry_safe(pmu, safe_temp, pmus, list) {
+		list_del(&pmu->list);
+		kfree(pmu);
+	}
+
+	for_each_possible_cpu(i)
+		if (pmu_irqs[i].irq > 0)
+			acpi_unregister_gsi(pmu_irqs[i].gsi);
+}
+
+/*
+ * Count number and type of CPU cores in the system. Returns the number
+ * of "unused" MADT entries we could not associate with a PMU. This can
+ * be the result of CPU's not being online,  or errors in the MADT.
+ * Under normal circumstances this will be 0.
+ */
+static int __init arm_pmu_acpi_determine_cpu_types(struct list_head *pmus)
+{
+	int i;
+	int unused_madt_entries = 0;
+
+	for_each_possible_cpu(i) {
+		struct cpuinfo_arm64 *cinfo = per_cpu_ptr(&cpu_data, i);
+		struct pmu_types *pmu;
+
+		/*
+		 * Ignore GSI registration failure for now, as
+		 * some of the MADT entries may not be used.
+		 */
+		pmu_irqs[i].irq = acpi_register_gsi(NULL, pmu_irqs[i].gsi,
+						    pmu_irqs[i].trigger,
+						    ACPI_ACTIVE_HIGH);
+		/* likely not online */
+		if (cinfo->reg_midr == 0) {
+			unused_madt_entries++;
+			continue;
+		}
+
+		list_for_each_entry(pmu, pmus, list) {
+			if (pmu->cpu_type == cinfo->reg_midr) {
+				pmu->cpu_count++;
+				break;
+			}
+		}
+
+		/* we didn't find the CPU type, add an entry to identify it */
+		if (&pmu->list == pmus) {
+			pmu = kzalloc(sizeof(struct pmu_types), GFP_KERNEL);
+			if (!pmu) {
+				pr_err("Unable to allocate pmu_types\n");
+				arm_pmu_acpi_handle_alloc_failure(pmus);
+				break;
+			}
+			pmu->cpu_type = cinfo->reg_midr;
+			pmu->cpu_count++;
+			list_add_tail(&pmu->list, pmus);
+		}
+	}
+
+	return unused_madt_entries;
+}
+
+static int __init arm_pmu_acpi_register_device(int count, struct resource *res,
+					       int cpu_id)
+{
+	struct platform_device *pdev;
+	int err = -ENOMEM;
+
+	pdev = platform_device_alloc(ARMV8_PMU_PDEV_NAME, cpu_id);
+	if (pdev) {
+		err = platform_device_add_resources(pdev, res, count);
+		if (!err)
+			err = platform_device_add(pdev);
+		if (err) {
+			pr_warn("Unable to register PMU device\n");
+			platform_device_put(pdev);
+		}
+	} else {
+	    pr_warn("Unable to allocate platform device\n");
+	}
+
+	return err;
+}
+
+static void __init arm_pmu_acpi_unregister_pmu_gsi(int cpu_id)
+{
+	int i;
+
+	for_each_possible_cpu(i) {
+		struct cpuinfo_arm64 *cinfo = per_cpu_ptr(&cpu_data, i);
+
+		if (cinfo->reg_midr == cpu_id) {
+			pmu_irqs[i].used = false;
+			if (pmu_irqs[i].irq > 0)
+				acpi_unregister_gsi(pmu_irqs[i].gsi);
+			pmu_irqs[i].gsi = -ENODEV;
+		}
+	}
+}
+
+/*
+ * Registers the group of PMU interfaces which correspond to the 'cpu_id'.
+ * This group utilizes 'count' resources in the 'res'.
+ */
+static int __init arm_pmu_acpi_register_pmu(int count, struct resource *res,
+					    int cpu_id)
+{
+	int err;
+
+	err = arm_pmu_acpi_register_device(count, res, cpu_id);
+
+	/* unmark and unregister GSIs for this PMU */
+	if (err)
+		arm_pmu_acpi_unregister_pmu_gsi(cpu_id);
+
+	return err;
+}
+
+int arm_pmu_acpi_retrieve_irq(struct resource *res, int cpu)
+{
+	int irq = -ENODEV;
+
+	if (pmu_irqs[cpu].used) {
+		pr_info("CPU %d's interrupt is already used\n", cpu);
+	} else {
+		pmu_irqs[cpu].used = true;
+		res->start = pmu_irqs[cpu].irq;
+		res->end = pmu_irqs[cpu].irq;
+		res->flags = IORESOURCE_IRQ;
+		if (pmu_irqs[cpu].trigger == ACPI_EDGE_SENSITIVE)
+			res->flags |= IORESOURCE_IRQ_HIGHEDGE;
+		else
+			res->flags |= IORESOURCE_IRQ_HIGHLEVEL;
+	}
+	return irq;
+}
+
+/*
+ * For the given cpu/pmu type, walk all known GSIs, register them, and add
+ * them to the resource structure. Return the number of GSI's contained
+ * in the res structure, and the id of the last CPU/PMU we added.
+ */
+static int __init arm_pmu_acpi_gsi_res(struct pmu_types *pmus,
+				       struct resource *res)
+{
+	int i, count;
+
+	/* lets group all the PMU's from similar CPU's together */
+	count = 0;
+	for_each_possible_cpu(i) {
+		struct cpuinfo_arm64 *cinfo = per_cpu_ptr(&cpu_data, i);
+
+		if (pmus->cpu_type == cinfo->reg_midr) {
+			if ((pmu_irqs[i].gsi == 0) && (cinfo->reg_midr != 0))
+				continue;
+
+			/* likely not online */
+			if (!cinfo->reg_midr)
+				continue;
+
+			arm_pmu_acpi_retrieve_irq(&res[count], i);
+			count++;
+		}
+	}
+	return count;
+}
+
+static int __init pmu_acpi_register(struct pmu_types *pmu)
+{
+	int count;
+	int err = -ENOMEM;
+	struct resource	*res;
+
+	res = kcalloc(pmu->cpu_count, sizeof(struct resource), GFP_KERNEL);
+
+	/* for a given PMU type, collect all the GSIs. */
+	if (res) {
+		count = arm_pmu_acpi_gsi_res(pmu, res);
+		/* register this set of interrupts with a new PMU device */
+		err = arm_pmu_acpi_register_pmu(pmu->cpu_count, res,
+						pmu->cpu_type);
+		if (!err)
+			pr_info("Register %d devices for %X\n", count,
+				pmu->cpu_type);
+		kfree(res);
+	} else {
+		pr_warn("PMU unable to allocate interrupt resource\n");
+		arm_pmu_acpi_unregister_pmu_gsi(pmu->cpu_type);
+	}
+	return err;
+}
+
+static int __init pmu_acpi_init(void)
+{
+	struct pmu_types *pmu, *safe_temp;
+	bool unused_madt_entries;
+	LIST_HEAD(pmus);
+
+	if (acpi_disabled)
+		return 0;
+
+	unused_madt_entries = arm_pmu_acpi_determine_cpu_types(&pmus);
+
+	list_for_each_entry_safe(pmu, safe_temp, &pmus, list) {
+		pmu->cpu_count += unused_madt_entries;
+		pmu_acpi_register(pmu);
+
+		list_del(&pmu->list);
+		kfree(pmu);
+	}
+
+	return 0;
+}
+
+arch_initcall(pmu_acpi_init);
-- 
2.5.5

[PATCH V10 6/6] arm: pmu: Add PMU definitions for cores not initially online

[Jeremy Linton]

ACPI CPUs aren't associated with a PMU until they have been put
online. This means that we potentially have to update a PMU
definition the first time a CPU is hot added to the machine.

Signed-off-by: Jeremy Linton <jeremy.linton@arm.com>
---
 drivers/perf/arm_pmu.c       | 38 ++++++++++++++++++++++++++++++++++++--
 include/linux/perf/arm_pmu.h |  4 ++++
 2 files changed, 40 insertions(+), 2 deletions(-)

diff --git a/drivers/perf/arm_pmu.c b/drivers/perf/arm_pmu.c
index 07e1404..775551c 100644
--- a/drivers/perf/arm_pmu.c
+++ b/drivers/perf/arm_pmu.c
@@ -711,6 +711,30 @@ static int cpu_pmu_request_irq(struct arm_pmu *cpu_pmu, irq_handler_t handler)
 	return 0;
 }
 
+static DEFINE_SPINLOCK(arm_pmu_resource_lock);
+
+static void arm_perf_associate_new_cpu(struct arm_pmu *lpmu, unsigned int cpu)
+{
+	struct platform_device *pdev = lpmu->plat_device;
+	struct resource *res;
+	struct pmu_hw_events *events;
+	int num_res;
+
+	spin_lock(&arm_pmu_resource_lock);
+	for (num_res = 0; num_res < pdev->num_resources; num_res++) {
+		if (!pdev->resource[num_res].flags)
+			break;
+	}
+	res = &pdev->resource[num_res];
+	arm_pmu_acpi_retrieve_irq(res, cpu);
+	events = per_cpu_ptr(lpmu->hw_events, cpu);
+	cpumask_set_cpu(cpu, &lpmu->supported_cpus);
+	if (lpmu->irq_affinity)
+		lpmu->irq_affinity[num_res] = cpu;
+	events->percpu_pmu = lpmu;
+	spin_unlock(&arm_pmu_resource_lock);
+}
+
 /*
  * PMU hardware loses all context when a CPU goes offline.
  * When a CPU is hotplugged back in, since some hardware registers are
@@ -721,10 +745,18 @@ static int arm_perf_starting_cpu(unsigned int cpu, struct hlist_node *node)
 {
 	struct arm_pmu *pmu = hlist_entry_safe(node, struct arm_pmu, node);
 
-	if (!cpumask_test_cpu(cpu, &pmu->supported_cpus))
-		return 0;
+	if (!cpumask_test_cpu(cpu, &pmu->supported_cpus)) {
+		unsigned int cpuid = read_specific_cpuid(cpu);
+
+		if (acpi_disabled)
+			return 0;
+		if (cpuid != pmu->id)
+			return 0;
+		arm_perf_associate_new_cpu(pmu, cpu);
+	}
 	if (pmu->reset)
 		pmu->reset(pmu);
+
 	return 0;
 }
 
@@ -906,6 +938,8 @@ static int probe_plat_pmu(struct arm_pmu *pmu,
 	struct platform_device *pdev = pmu->plat_device;
 	int irq = platform_get_irq(pdev, 0);
 
+	pmu->id = pmuid;
+
 	if (irq >= 0 && !irq_is_percpu(irq)) {
 		pmu->irq_affinity = kcalloc(pdev->num_resources, sizeof(int),
 					    GFP_KERNEL);
diff --git a/include/linux/perf/arm_pmu.h b/include/linux/perf/arm_pmu.h
index df1ba55..ed82b8f 100644
--- a/include/linux/perf/arm_pmu.h
+++ b/include/linux/perf/arm_pmu.h
@@ -112,6 +112,7 @@ struct arm_pmu {
 	struct mutex	reserve_mutex;
 	u64		max_period;
 	bool		secure_access; /* 32-bit ARM only */
+	unsigned int	id;
 #define ARMV8_PMUV3_MAX_COMMON_EVENTS 0x40
 	DECLARE_BITMAP(pmceid_bitmap, ARMV8_PMUV3_MAX_COMMON_EVENTS);
 	struct platform_device	*plat_device;
@@ -167,8 +168,11 @@ int arm_pmu_device_probe(struct platform_device *pdev,
 #ifdef CONFIG_ARM_PMU_ACPI
 struct acpi_madt_generic_interrupt;
 void arm_pmu_parse_acpi(int cpu, struct acpi_madt_generic_interrupt *gic);
+int arm_pmu_acpi_retrieve_irq(struct resource *pdev, int cpu);
 #else
 #define arm_pmu_parse_acpi(a, b) do { } while (0)
+#define arm_pmu_acpi_retrieve_irq(pdev, cpu) \
+	do { } while (0)
 #endif /* CONFIG_ARM_PMU_ACPI */
 
 #endif /* __ARM_PMU_H__ */
-- 
2.5.5

[PATCH V10 0/6] Enable PMUs in ACPI Systems

[Punit Agrawal]

Hi Jeremy,

Jeremy Linton <jeremy.linton@arm.com> writes:

> This patch expands and reworks the patches published by Mark Salter
> in order to clean up a few of the previous review comments, as well as
> add support for newer CPUs and big/little configurations.
>
> v10:
> - Rebase to 4.9
> - Rework the arm_perf_start_cpu changes to support the 4.9 hotplug
>   changes.
> - Remove the call to acpi_register_gsi() from the cpu online code path.
>   Instead the GSI's are registered during the initcall. This changes
>   the error handling a bit because we now try to clean up the
>   previously registered GSIs in a couple important places. This
>   was also a result of the rebase.
> - Dropped the MIDR partnumber usage, its no longer necessary to
>   differentiate by only the partnum, so this helps to clarify the code
>   a bit. 
> - Shuffle some code around and rename a few variables.
> - Added a few comments to hopefully clarify some questions people have
>   previously had about unused MADT entries, skipping processing cores
>   with MIDR=0, etc.
>

I've had a look at the code and tested the patches on a Juno and a
Seattle based system I have and the pmus seem to be behaving themselves.

FWIW,

Acked-by: Punit Agrawal <punit.agrawal@arm.com>

Tested-by: Punit Agrawal <punit.agrawal@arm.com>

> v9:
> - Add/cleanup an additional hotplug patch I've had sitting around. This
>   patch brings the ACPI PMU mostly on par with the DT functionality with
>   respect to having CPUs offline during boot. This should help clarify
>   some of the code structuring.
> - Cleanup the list of PMU types early if we fail to allocate memory for an
>   additional pmu type.
>
> v8:
> - Rebase to 4.8rc4
> - Assorted minor comment/hunk placement/etc tweaks per Punit Agrawal
>
> v7:
> - Rebase to 4.8rc3
> - Remove cpu affinity sysfs entry. While providing a CPU mask for
>   ARMv8 PMU's is really helpful in big/little environments, reworking
>   the PMU code to support the cpumask attribute for !arm64 PMUs is out
>   of the scope of this patch set.
> - Fix CPU miscount problem where an alloc failure followed by successfully
>   allocating the structure can result in under counting the CPUs associated
>   with the PMU. This bug was created in v6 with the conversion to a linked
>   list.
> -  Remove initial platform device creation code by Mark Salter, and re-squash
>    multiple platform device creation code together with helper routines.
>    Other minor tweakage.
>
> v6:
> - Added cpu affinity sysfs entry
> - Converted pmu_types array, to linked list
> - Restrict use of the armv8_pmu_probe_table to ACPI systems
> - Rename MADT parsing routines in smp.c
> - Convert sysfs PMU name to use index rather than partnum
> - Remove pr_devel statements
> - Other Minor cleanups
> - Add Partial Ack-by Will Deacon
>
> v5:
> - Remove list of CPU types for ACPI systems. We now match a generic
>   event list, and use the PMCIED[01] to select events which exist on
>   the given PMU. This avoids the need to update the kernel every time
>   a new CPU is released.
> - Update the maintainers list to include the new file.
>
> v4:
> - Correct build issues with ARM (!ARM64) kernels.
> - Add ThunderX to list of PMU types.
>
> v3:
> - Enable ARM performance monitoring units on ACPI/arm64 machines.
>
> Jeremy Linton (5):
>   arm64: Rename the common MADT parse routine
>   arm: arm64: Add routine to determine cpuid of other cpus
>   arm: arm64: pmu: Assign platform PMU CPU affinity
>   arm64: pmu: Detect and enable multiple PMUs in an ACPI system
>   arm: pmu: Add PMU definitions for cores not initially online
>
> Mark Salter (1):
>   arm64: pmu: Cache PMU interrupt numbers from MADT parse
>
>  arch/arm/include/asm/cputype.h   |   2 +
>  arch/arm64/include/asm/cputype.h |   3 +
>  arch/arm64/kernel/smp.c          |  18 ++-
>  drivers/perf/Kconfig             |   4 +
>  drivers/perf/Makefile            |   1 +
>  drivers/perf/arm_pmu.c           | 107 +++++++++++++--
>  drivers/perf/arm_pmu_acpi.c      | 272 +++++++++++++++++++++++++++++++++++++++
>  include/linux/perf/arm_pmu.h     |  11 ++
>  8 files changed, 400 insertions(+), 18 deletions(-)
>  create mode 100644 drivers/perf/arm_pmu_acpi.c

[PATCH V10 5/6] arm64: pmu: Detect and enable multiple PMUs in an ACPI system

[Will Deacon]

On Wed, Nov 09, 2016 at 05:39:52PM -0600, Jeremy Linton wrote:
> Its possible that an ACPI system has multiple CPU types in it
> with differing PMU counters. Iterate the CPU's and make a determination
> about how many of each type exist in the system. Then take and create
> a PMU platform device for each type, and assign it the interrupts parsed
> from the MADT. Creating a platform device is necessary because the PMUs
> are not described as devices in the DSDT table.
> 
> This code is loosely based on earlier work by Mark Salter.
> 
> Signed-off-by: Jeremy Linton <jeremy.linton@arm.com>
> ---
>  drivers/perf/arm_pmu.c      |   8 +-
>  drivers/perf/arm_pmu_acpi.c | 234 +++++++++++++++++++++++++++++++++++++++++++-
>  2 files changed, 240 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/perf/arm_pmu.c b/drivers/perf/arm_pmu.c
> index 6008be9..07e1404 100644
> --- a/drivers/perf/arm_pmu.c
> +++ b/drivers/perf/arm_pmu.c
> @@ -1087,7 +1087,13 @@ int arm_pmu_device_probe(struct platform_device *pdev,
>  		if (!ret)
>  			ret = init_fn(pmu);
>  	} else if (probe_table) {
> -		ret = probe_plat_pmu(pmu, probe_table, read_cpuid_id());
> +		if (acpi_disabled) {
> +			/* use the current cpu. */
> +			ret = probe_plat_pmu(pmu, probe_table,
> +					     read_cpuid_id());
> +		} else {
> +			ret = probe_plat_pmu(pmu, probe_table, pdev->id);
> +		}
>  	}
>  
>  	if (ret) {
> diff --git a/drivers/perf/arm_pmu_acpi.c b/drivers/perf/arm_pmu_acpi.c
> index 135851c..eecf1c1 100644
> --- a/drivers/perf/arm_pmu_acpi.c
> +++ b/drivers/perf/arm_pmu_acpi.c
> @@ -2,13 +2,17 @@
>   * ARM ACPI PMU support
>   *
>   * Copyright (C) 2015 Red Hat Inc.
> + * Copyright (C) 2016 ARM Ltd.
>   * Author: Mark Salter <msalter@redhat.com>
> + *	   Jeremy Linton <jeremy.linton@arm.com>
>   *
>   * This work is licensed under the terms of the GNU GPL, version 2.  See
>   * the COPYING file in the top-level directory.
>   *
>   */
>  
> +#define pr_fmt(fmt) "ACPI-PMU: " fmt
> +
>  #include <asm/cpu.h>
>  #include <linux/acpi.h>
>  #include <linux/irq.h>
> @@ -20,7 +24,14 @@
>  struct pmu_irq {
>  	int  gsi;
>  	int  trigger;
> -	bool registered;
> +	int  irq;
> +	bool used;
> +};
> +
> +struct pmu_types {
> +	struct list_head list;
> +	int		 cpu_type;
> +	int		 cpu_count;
>  };
>  
>  static struct pmu_irq pmu_irqs[NR_CPUS];
> @@ -38,3 +49,224 @@ void __init arm_pmu_parse_acpi(int cpu, struct acpi_madt_generic_interrupt *gic)
>  	else
>  		pmu_irqs[cpu].trigger = ACPI_LEVEL_SENSITIVE;
>  }
> +
> +static void __init arm_pmu_acpi_handle_alloc_failure(struct list_head *pmus)
> +{
> +	int i;
> +	struct pmu_types *pmu, *safe_temp;
> +
> +	list_for_each_entry_safe(pmu, safe_temp, pmus, list) {
> +		list_del(&pmu->list);
> +		kfree(pmu);
> +	}
> +
> +	for_each_possible_cpu(i)
> +		if (pmu_irqs[i].irq > 0)
> +			acpi_unregister_gsi(pmu_irqs[i].gsi);
> +}
> +
> +/*
> + * Count number and type of CPU cores in the system. Returns the number
> + * of "unused" MADT entries we could not associate with a PMU. This can
> + * be the result of CPU's not being online,  or errors in the MADT.
> + * Under normal circumstances this will be 0.
> + */
> +static int __init arm_pmu_acpi_determine_cpu_types(struct list_head *pmus)
> +{
> +	int i;
> +	int unused_madt_entries = 0;
> +
> +	for_each_possible_cpu(i) {
> +		struct cpuinfo_arm64 *cinfo = per_cpu_ptr(&cpu_data, i);
> +		struct pmu_types *pmu;
> +
> +		/*
> +		 * Ignore GSI registration failure for now, as
> +		 * some of the MADT entries may not be used.
> +		 */
> +		pmu_irqs[i].irq = acpi_register_gsi(NULL, pmu_irqs[i].gsi,
> +						    pmu_irqs[i].trigger,
> +						    ACPI_ACTIVE_HIGH);
> +		/* likely not online */
> +		if (cinfo->reg_midr == 0) {

I appreciate that this code only gets built for arm64 at the moment, but
given that you've introduced the read_specific_cpuid macro, it seems like
it would be better to use it here too and avoid the reference to struct
cpuinfo_arm64 altogether. Similarly for the other references in this file.

Will

[PATCH V10 2/6] arm: arm64: Add routine to determine cpuid of other cpus

[Will Deacon]

[adding Russell]

On Wed, Nov 09, 2016 at 05:39:49PM -0600, Jeremy Linton wrote:
> It is helpful if we can read the cpuid/midr of other CPUs
> in the system independent of arm/arm64.
> 
> Signed-off-by: Jeremy Linton <jeremy.linton@arm.com>
> ---
>  arch/arm/include/asm/cputype.h   | 2 ++
>  arch/arm64/include/asm/cputype.h | 3 +++
>  2 files changed, 5 insertions(+)
> 
> diff --git a/arch/arm/include/asm/cputype.h b/arch/arm/include/asm/cputype.h
> index 522b5fe..31fb273 100644
> --- a/arch/arm/include/asm/cputype.h
> +++ b/arch/arm/include/asm/cputype.h
> @@ -235,6 +235,8 @@ static inline unsigned int __attribute_const__ read_cpuid_mpidr(void)
>  #define cpu_is_sa1100() (read_cpuid_part() == ARM_CPU_PART_SA1100)
>  #define cpu_is_sa1110() (read_cpuid_part() == ARM_CPU_PART_SA1110)
>  
> +#define read_specific_cpuid(cpu_num) per_cpu_ptr(&cpu_data, cpu_num)->cpuid
> +
>  /*
>   * Intel's XScale3 core supports some v6 features (supersections, L2)
>   * but advertises itself as v5 as it does not support the v6 ISA.  For

Russell -- are you ok with adding this macro to arch/arm/? It will get used
by the CPU PMU driver, which needs a portable (i.e. between arm and arm64)
way to convert a logical CPU ID into the MIDR register for that CPU.

Thanks,

Will

[PATCH V10 2/6] arm: arm64: Add routine to determine cpuid of other cpus

[Russell King - ARM Linux]

On Tue, Nov 29, 2016 at 10:31:12AM +0000, Will Deacon wrote:
> [adding Russell]
> 
> On Wed, Nov 09, 2016 at 05:39:49PM -0600, Jeremy Linton wrote:
> > It is helpful if we can read the cpuid/midr of other CPUs
> > in the system independent of arm/arm64.
> > 
> > Signed-off-by: Jeremy Linton <jeremy.linton@arm.com>
> > ---
> >  arch/arm/include/asm/cputype.h   | 2 ++
> >  arch/arm64/include/asm/cputype.h | 3 +++
> >  2 files changed, 5 insertions(+)
> > 
> > diff --git a/arch/arm/include/asm/cputype.h b/arch/arm/include/asm/cputype.h
> > index 522b5fe..31fb273 100644
> > --- a/arch/arm/include/asm/cputype.h
> > +++ b/arch/arm/include/asm/cputype.h
> > @@ -235,6 +235,8 @@ static inline unsigned int __attribute_const__ read_cpuid_mpidr(void)
> >  #define cpu_is_sa1100() (read_cpuid_part() == ARM_CPU_PART_SA1100)
> >  #define cpu_is_sa1110() (read_cpuid_part() == ARM_CPU_PART_SA1110)
> >  
> > +#define read_specific_cpuid(cpu_num) per_cpu_ptr(&cpu_data, cpu_num)->cpuid
> > +
> >  /*
> >   * Intel's XScale3 core supports some v6 features (supersections, L2)
> >   * but advertises itself as v5 as it does not support the v6 ISA.  For
> 
> Russell -- are you ok with adding this macro to arch/arm/? It will get used
> by the CPU PMU driver, which needs a portable (i.e. between arm and arm64)
> way to convert a logical CPU ID into the MIDR register for that CPU.

No, because we don't set the cpuid member in uniprocessor configurations,
so it's going to be a fragile macro - it'll return zero for kernels
configured without SMP support.

I'd ideally like cpuid for the boot CPU to be set early, so that we can
get rid of many ifdefs in this area, but haven't convinced myself that
it's safe to do so in all configurations with the percpu stuff not being
up and running in setup_arch().

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

[PATCH V10 4/6] arm: arm64: pmu: Assign platform PMU CPU affinity

[Will Deacon]

On Wed, Nov 09, 2016 at 05:39:51PM -0600, Jeremy Linton wrote:
> On systems with multiple PMU types the PMU to CPU affinity
> needs to be detected and set. The CPU to interrupt affinity
> should also be set.
> 
> Signed-off-by: Jeremy Linton <jeremy.linton@arm.com>
> ---
>  drivers/perf/arm_pmu.c | 63 ++++++++++++++++++++++++++++++++++++++++++--------
>  1 file changed, 53 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/perf/arm_pmu.c b/drivers/perf/arm_pmu.c
> index b37b572..6008be9 100644
> --- a/drivers/perf/arm_pmu.c
> +++ b/drivers/perf/arm_pmu.c
> @@ -11,6 +11,7 @@
>   */
>  #define pr_fmt(fmt) "hw perfevents: " fmt
>  
> +#include <linux/acpi.h>
>  #include <linux/bitmap.h>
>  #include <linux/cpumask.h>
>  #include <linux/cpu_pm.h>
> @@ -24,6 +25,7 @@
>  #include <linux/irq.h>
>  #include <linux/irqdesc.h>
>  
> +#include <asm/cpu.h>
>  #include <asm/cputype.h>
>  #include <asm/irq_regs.h>
>  
> @@ -889,25 +891,67 @@ static void cpu_pmu_destroy(struct arm_pmu *cpu_pmu)
>  }
>  
>  /*
> - * CPU PMU identification and probing.
> + * CPU PMU identification and probing. Its possible to have
> + * multiple CPU types in an ARM machine. Assure that we are
> + * picking the right PMU types based on the CPU in question
>   */
> -static int probe_current_pmu(struct arm_pmu *pmu,
> -			     const struct pmu_probe_info *info)
> +static int probe_plat_pmu(struct arm_pmu *pmu,
> +			     const struct pmu_probe_info *info,
> +			     unsigned int pmuid)
>  {
> -	int cpu = get_cpu();
> -	unsigned int cpuid = read_cpuid_id();
>  	int ret = -ENODEV;
> +	int cpu;
> +	int aff_ctr = 0;
> +	static int duplicate_pmus;
> +	struct platform_device *pdev = pmu->plat_device;
> +	int irq = platform_get_irq(pdev, 0);
>  
> -	pr_info("probing PMU on CPU %d\n", cpu);
> +	if (irq >= 0 && !irq_is_percpu(irq)) {
> +		pmu->irq_affinity = kcalloc(pdev->num_resources, sizeof(int),
> +					    GFP_KERNEL);
> +		if (!pmu->irq_affinity)
> +			return -ENOMEM;
> +	}
>  
> +	for_each_possible_cpu(cpu) {
> +		unsigned int cpuid = read_specific_cpuid(cpu);
> +
> +		if (cpuid == pmuid) {
> +			cpumask_set_cpu(cpu, &pmu->supported_cpus);
> +			if (pmu->irq_affinity) {
> +				pmu->irq_affinity[aff_ctr] = cpu;
> +				aff_ctr++;
> +			}
> +		}
> +	}
> +
> +	/* find the type of PMU given the CPU */
>  	for (; info->init != NULL; info++) {
> -		if ((cpuid & info->mask) != info->cpuid)
> +		if ((pmuid & info->mask) != info->cpuid)
>  			continue;
>  		ret = info->init(pmu);
> +		/*
> +		 * if this pmu declaration is unspecified and we have
> +		 * previously found a PMU on this platform then append
> +		 * a PMU number to the pmu name. This avoids changing
> +		 * the names of PMUs that are specific to a class of CPUs.
> +		 * The assumption is that if we match a specific PMU in the
> +		 * provided pmu_probe_info then it's unique, and another PMU
> +		 * in the system will match a different entry rather than
> +		 * needing the _number to assure its unique.
> +		 */
> +		if ((!info->cpuid) && (duplicate_pmus)) {

This is a bit grim: if you had a PMU with a non-zero info->cpuid, then you
later found a PMU with a zeroed info->cpuid, the latter would get a
redundant suffix. This doesn't happen in reality, because the ACPI case
always has info->cpuid == 0, but if somebody extends armv8_pmu_probe_table
then we'd get this and probably not realise.

I think the duplicate_pmus counter needs to be tied explicitly to the
"default type" (i.e. when info->cpuid == 0, but see my next comment).

> +			pmu->name = kasprintf(GFP_KERNEL, "%s_%d",
> +					    pmu->name, duplicate_pmus);
> +			if (!pmu->name) {
> +				kfree(pmu->irq_affinity);
> +				ret = -ENOMEM;
> +			}
> +		}

This code doesn't run for the device-tree probing case, but I think it would
be useful to do the same numbering trick for e.g. systems with multiple PMUs
that all end up matching on armv8_pmuv3.

Will

[PATCH V10 2/6] arm: arm64: Add routine to determine cpuid of other cpus

[Jeremy Linton]

Hi,

On 11/29/2016 04:46 AM, Russell King - ARM Linux wrote:
> On Tue, Nov 29, 2016 at 10:31:12AM +0000, Will Deacon wrote:
>> [adding Russell]
>>
>> On Wed, Nov 09, 2016 at 05:39:49PM -0600, Jeremy Linton wrote:
>>> It is helpful if we can read the cpuid/midr of other CPUs
>>> in the system independent of arm/arm64.
>>>
>>> Signed-off-by: Jeremy Linton <jeremy.linton@arm.com>
>>> ---
>>>  arch/arm/include/asm/cputype.h   | 2 ++
>>>  arch/arm64/include/asm/cputype.h | 3 +++
>>>  2 files changed, 5 insertions(+)
>>>
>>> diff --git a/arch/arm/include/asm/cputype.h b/arch/arm/include/asm/cputype.h
>>> index 522b5fe..31fb273 100644
>>> --- a/arch/arm/include/asm/cputype.h
>>> +++ b/arch/arm/include/asm/cputype.h
>>> @@ -235,6 +235,8 @@ static inline unsigned int __attribute_const__ read_cpuid_mpidr(void)
>>>  #define cpu_is_sa1100() (read_cpuid_part() == ARM_CPU_PART_SA1100)
>>>  #define cpu_is_sa1110() (read_cpuid_part() == ARM_CPU_PART_SA1110)
>>>
>>> +#define read_specific_cpuid(cpu_num) per_cpu_ptr(&cpu_data, cpu_num)->cpuid
>>> +
>>>  /*
>>>   * Intel's XScale3 core supports some v6 features (supersections, L2)
>>>   * but advertises itself as v5 as it does not support the v6 ISA.  For
>>
>> Russell -- are you ok with adding this macro to arch/arm/? It will get used
>> by the CPU PMU driver, which needs a portable (i.e. between arm and arm64)
>> way to convert a logical CPU ID into the MIDR register for that CPU.
>
> No, because we don't set the cpuid member in uniprocessor configurations,
> so it's going to be a fragile macro - it'll return zero for kernels
> configured without SMP support.

By itself is easy enough to fix, it could be tweaked to do the same 
thing as c_show()

#define read_specific_cpuid(cpu_num) is_smp() ? per_cpu(&cpu_data, 
cpu_num)->cpuid : read_cpuid_id()

c_show could then be updated to use read_specific_cpuid().

>
> I'd ideally like cpuid for the boot CPU to be set early, so that we can
> get rid of many ifdefs in this area, but haven't convinced myself that
> it's safe to do so in all configurations with the percpu stuff not being
> up and running in setup_arch().

As the above change is a little ugly... It seems something like 
smp_prepare_boot_cpu() call would be perfect if it were available for 
!SMP. Although this is getting a little off base for the PMU changes.

The alternative to all this, is yet another #ifdef in the pmu code, 
which is IMHO the worse choice.

Suggestions?

[PATCH V10 4/6] arm: arm64: pmu: Assign platform PMU CPU affinity

[Jeremy Linton]

Hi,

On 11/29/2016 04:52 AM, Will Deacon wrote:
> On Wed, Nov 09, 2016 at 05:39:51PM -0600, Jeremy Linton wrote:
>> On systems with multiple PMU types the PMU to CPU affinity
>> needs to be detected and set. The CPU to interrupt affinity
>> should also be set.
>>

(trimming)

>> +static int probe_plat_pmu(struct arm_pmu *pmu,
>> +			     const struct pmu_probe_info *info,
>> +			     unsigned int pmuid)
>>  {
>> -	int cpu = get_cpu();
>> -	unsigned int cpuid = read_cpuid_id();
>>  	int ret = -ENODEV;
>> +	int cpu;
>> +	int aff_ctr = 0;
>> +	static int duplicate_pmus;
>> +	struct platform_device *pdev = pmu->plat_device;
>> +	int irq = platform_get_irq(pdev, 0);
>>
>> -	pr_info("probing PMU on CPU %d\n", cpu);
>> +	if (irq >= 0 && !irq_is_percpu(irq)) {
>> +		pmu->irq_affinity = kcalloc(pdev->num_resources, sizeof(int),
>> +					    GFP_KERNEL);
>> +		if (!pmu->irq_affinity)
>> +			return -ENOMEM;
>> +	}
>>
>> +	for_each_possible_cpu(cpu) {
>> +		unsigned int cpuid = read_specific_cpuid(cpu);
>> +
>> +		if (cpuid == pmuid) {
>> +			cpumask_set_cpu(cpu, &pmu->supported_cpus);
>> +			if (pmu->irq_affinity) {
>> +				pmu->irq_affinity[aff_ctr] = cpu;
>> +				aff_ctr++;
>> +			}
>> +		}
>> +	}
>> +
>> +	/* find the type of PMU given the CPU */
>>  	for (; info->init != NULL; info++) {
>> -		if ((cpuid & info->mask) != info->cpuid)
>> +		if ((pmuid & info->mask) != info->cpuid)
>>  			continue;
>>  		ret = info->init(pmu);
>> +		/*
>> +		 * if this pmu declaration is unspecified and we have
>> +		 * previously found a PMU on this platform then append
>> +		 * a PMU number to the pmu name. This avoids changing
>> +		 * the names of PMUs that are specific to a class of CPUs.
>> +		 * The assumption is that if we match a specific PMU in the
>> +		 * provided pmu_probe_info then it's unique, and another PMU
>> +		 * in the system will match a different entry rather than
>> +		 * needing the _number to assure its unique.
>> +		 */
>> +		if ((!info->cpuid) && (duplicate_pmus)) {
>
> This is a bit grim: if you had a PMU with a non-zero info->cpuid, then you
> later found a PMU with a zeroed info->cpuid, the latter would get a
> redundant suffix. This doesn't happen in reality, because the ACPI case
> always has info->cpuid == 0, but if somebody extends armv8_pmu_probe_table
> then we'd get this and probably not realise.

Hoisting the duplicate_pmus inside the !info->cpuid fixes that...

>
> I think the duplicate_pmus counter needs to be tied explicitly to the
> "default type" (i.e. when info->cpuid == 0, but see my next comment).
>
>> +			pmu->name = kasprintf(GFP_KERNEL, "%s_%d",
>> +					    pmu->name, duplicate_pmus);
>> +			if (!pmu->name) {
>> +				kfree(pmu->irq_affinity);
>> +				ret = -ENOMEM;
>> +			}
>> +		}
>
> This code doesn't run for the device-tree probing case, but I think it would
> be useful to do the same numbering trick for e.g. systems with multiple PMUs
> that all end up matching on armv8_pmuv3.

Ok, its pretty straightforward to move the check into 
arm_pmu_device_probe() itself and do a string compare against 
DEFAULT_V8_PMU define rather than !cpuid.

Ok, I will do that and post v11 as soon as I hear from Russell about 
what he wants to do with read_specific_cpuid().