LinuxPPC-Dev Archive on lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 7/7] P2020ds: add event button handler
From: Li Yang @ 2010-12-03 12:34 UTC (permalink / raw)
  To: linuxppc-dev
In-Reply-To: <1291379651-8822-6-git-send-email-leoli@freescale.com>

This can be used as a wakeup source for power management.

Signed-off-by: Li Yang <leoli@freescale.com>
---
 arch/powerpc/boot/dts/p2020ds.dts        |    9 ++++++++-
 arch/powerpc/platforms/85xx/mpc85xx_ds.c |   26 +++++++++++++++++++++++++-
 2 files changed, 33 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/boot/dts/p2020ds.dts b/arch/powerpc/boot/dts/p2020ds.dts
index 1101914..f4c6520 100644
--- a/arch/powerpc/boot/dts/p2020ds.dts
+++ b/arch/powerpc/boot/dts/p2020ds.dts
@@ -1,7 +1,7 @@
 /*
  * P2020 DS Device Tree Source
  *
- * Copyright 2009 Freescale Semiconductor Inc.
+ * Copyright 2009-2010 Freescale Semiconductor Inc.
  *
  * This program is free software; you can redistribute  it and/or modify it
  * under  the terms of  the GNU General  Public License as published by the
@@ -155,6 +155,13 @@
 			compatible = "fsl,elbc-fcm-nand";
 			reg = <0x6 0x0 0x40000>;
 		};
+
+		ngpixis@3,0 {
+			compatible = "fsl,p2020ds-fpga";
+			reg = <0x3 0 0x30>;
+			interrupt-parent = <&mpic>;
+			interrupts = <0 0>;
+		};
 	};
 
 	soc@ffe00000 {
diff --git a/arch/powerpc/platforms/85xx/mpc85xx_ds.c b/arch/powerpc/platforms/85xx/mpc85xx_ds.c
index 8190bc2..a8807fe 100644
--- a/arch/powerpc/platforms/85xx/mpc85xx_ds.c
+++ b/arch/powerpc/platforms/85xx/mpc85xx_ds.c
@@ -4,7 +4,7 @@
  * Author Xianghua Xiao (x.xiao@freescale.com)
  * Roy Zang <tie-fei.zang@freescale.com>
  * 	- Add PCI/PCI Exprees support
- * Copyright 2007 Freescale Semiconductor Inc.
+ * Copyright 2007-2010 Freescale Semiconductor Inc.
  *
  * This program is free software; you can redistribute  it and/or modify it
  * under  the terms of  the GNU General  Public License as published by the
@@ -200,6 +200,30 @@ static void __init mpc85xx_ds_setup_arch(void)
 	printk("MPC85xx DS board from Freescale Semiconductor\n");
 }
 
+static irqreturn_t event_isr(int irq, void *dev_id)
+{
+
+	printk(KERN_INFO "MPC85xxDS: Event button been pushed.\n");
+	return IRQ_HANDLED;
+}
+
+static int __init p2020ds_ngpixis_init(void)
+{
+	int event_irq, ret;
+	struct device_node *np;
+
+	np = of_find_compatible_node(NULL, NULL, "fsl,p2020ds-fpga");
+	if (np) {
+		event_irq = irq_of_parse_and_map(np, 0);
+		ret = request_irq(event_irq, event_isr, 0, "event", NULL);
+		if (ret)
+			printk(KERN_ERR "Can't request board event int\n");
+		of_node_put(np);
+	}
+	return 0;
+}
+machine_device_initcall(p2020_ds, p2020ds_ngpixis_init);
+
 /*
  * Called very early, device-tree isn't unflattened
  */
-- 
1.6.6-rc1.GIT

^ permalink raw reply related

* [PATCH 2/7] powerpc/85xx: add HOTPLUG_CPU support
From: Li Yang @ 2010-12-03 12:34 UTC (permalink / raw)
  To: linuxppc-dev
In-Reply-To: <1291379651-8822-1-git-send-email-leoli@freescale.com>

Add support to disable and re-enable individual cores at runtime
on MPC85xx/QorIQ SMP machines.

This makes suspend/resume possible for SMP systems, as the power management
code on SMP always disable non-boot cpus on suspend.

MPC85xx machines use ePAPR spin-table in boot page for CPU kick-off.
This patch brings the bootpage and spin-table from bootloader into
kernel because the bootpage in bootloader might have been lost at
runtime.  Also add support to boot from physical address larger than
32-bit.

Signed-off-by: Yutaka Ando <y.ando@freescale.com>
Signed-off-by: Li Yang <leoli@freescale.com>
---
 arch/powerpc/Kconfig                   |    2 +-
 arch/powerpc/kernel/Makefile           |    2 +-
 arch/powerpc/kernel/head_fsl_booke.S   |   32 +++++
 arch/powerpc/kernel/smp.c              |    4 +-
 arch/powerpc/platforms/85xx/Makefile   |    4 +-
 arch/powerpc/platforms/85xx/bootpage.S |  206 +++++++++++++++++++++++++++++
 arch/powerpc/platforms/85xx/smp.c      |  222 ++++++++++++++++++++++++++------
 7 files changed, 428 insertions(+), 44 deletions(-)
 create mode 100644 arch/powerpc/platforms/85xx/bootpage.S

diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index e625e9e..b1982dd 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -320,7 +320,7 @@ config SWIOTLB
 
 config HOTPLUG_CPU
 	bool "Support for enabling/disabling CPUs"
-	depends on SMP && HOTPLUG && EXPERIMENTAL && (PPC_PSERIES || PPC_PMAC)
+	depends on SMP && HOTPLUG && EXPERIMENTAL && (PPC_PSERIES || PPC_PMAC || E500)
 	---help---
 	  Say Y here to be able to disable and re-enable individual
 	  CPUs at runtime on SMP machines.
diff --git a/arch/powerpc/kernel/Makefile b/arch/powerpc/kernel/Makefile
index 36c30f3..bb20496 100644
--- a/arch/powerpc/kernel/Makefile
+++ b/arch/powerpc/kernel/Makefile
@@ -56,7 +56,7 @@ obj-$(CONFIG_IBMEBUS)           += ibmebus.o
 obj-$(CONFIG_GENERIC_TBSYNC)	+= smp-tbsync.o
 obj-$(CONFIG_CRASH_DUMP)	+= crash_dump.o
 ifeq ($(CONFIG_PPC32),y)
-obj-$(CONFIG_E500)		+= idle_e500.o
+obj-$(CONFIG_E500)		+= idle_e500.o l2cr_85xx.o
 endif
 obj-$(CONFIG_6xx)		+= idle_6xx.o l2cr_6xx.o cpu_setup_6xx.o
 obj-$(CONFIG_TAU)		+= tau_6xx.o
diff --git a/arch/powerpc/kernel/head_fsl_booke.S b/arch/powerpc/kernel/head_fsl_booke.S
index 529b817..61d9c46 100644
--- a/arch/powerpc/kernel/head_fsl_booke.S
+++ b/arch/powerpc/kernel/head_fsl_booke.S
@@ -23,6 +23,7 @@
  *	PowerPC 44x support, Matt Porter <mporter@kernel.crashing.org>
  *    Copyright 2004 Freescale Semiconductor, Inc
  *	PowerPC e500 modifications, Kumar Gala <galak@kernel.crashing.org>
+ *    Copyright 2008, 2010 Freescale Semiconductor, Inc.
  *
  * This program is free software; you can redistribute  it and/or modify it
  * under  the terms of  the GNU General  Public License as published by the
@@ -254,6 +255,37 @@ _ENTRY(__early_start)
 	lwz	r11, 0(r12);		/* Get Linux PTE */
 #endif
 
+_GLOBAL(flush_disable_L1)
+/*
+ * Flush L1 d-cache, invalidate and disable d-cache,
+ * invalidate and disable i-cache
+ */
+	mflr	r10
+	bl	flush_dcache_L1	/* Flush L1 d-cache */
+	mtlr	r10
+
+	mfspr	r4, SPRN_L1CSR0	/* Invalidate and disable d-cache */
+	li	r5, 2
+	rlwimi	r4, r5, 0, 3
+
+	msync
+	isync
+	mtspr	SPRN_L1CSR0, r4
+	isync
+
+1:	mfspr	r4, SPRN_L1CSR0	/* Wait for the invalidate to finish */
+	andi.	r4, r4, 2
+	bne	1b
+
+	mfspr	r4, SPRN_L1CSR1	/* Invalidate and disable i-cache */
+	li	r5, 2
+	rlwimi	r4, r5, 0, 3
+
+	mtspr	SPRN_L1CSR1, r4
+	isync
+
+	blr
+
 /*
  * Interrupt vector entry code
  *
diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
index 68034bb..321cf2e 100644
--- a/arch/powerpc/kernel/smp.c
+++ b/arch/powerpc/kernel/smp.c
@@ -317,6 +317,8 @@ int generic_cpu_disable(void)
 	set_cpu_online(cpu, false);
 #ifdef CONFIG_PPC64
 	vdso_data->processorCount--;
+#endif
+#if defined(CONFIG_PPC64) || defined(CONFIG_E500)
 	fixup_irqs(cpu_online_mask);
 #endif
 	return 0;
@@ -336,7 +338,7 @@ int generic_cpu_enable(unsigned int cpu)
 	while (!cpu_online(cpu))
 		cpu_relax();
 
-#ifdef CONFIG_PPC64
+#if defined(CONFIG_PPC64) || defined(CONFIG_E500)
 	fixup_irqs(cpu_online_mask);
 	/* counter the irq disable in fixup_irqs */
 	local_irq_enable();
diff --git a/arch/powerpc/platforms/85xx/Makefile b/arch/powerpc/platforms/85xx/Makefile
index dd70db7..6bbcf22 100644
--- a/arch/powerpc/platforms/85xx/Makefile
+++ b/arch/powerpc/platforms/85xx/Makefile
@@ -1,7 +1,9 @@
 #
 # Makefile for the PowerPC 85xx linux kernel.
 #
-obj-$(CONFIG_SMP) += smp.o
+obj-$(CONFIG_SMP)         += smp.o
+obj-$(CONFIG_HOTPLUG_CPU) += bootpage.o
+obj-$(CONFIG_SUSPEND)     += suspend-asm.o
 
 obj-$(CONFIG_MPC8540_ADS) += mpc85xx_ads.o
 obj-$(CONFIG_MPC8560_ADS) += mpc85xx_ads.o
diff --git a/arch/powerpc/platforms/85xx/bootpage.S b/arch/powerpc/platforms/85xx/bootpage.S
new file mode 100644
index 0000000..ff0ca10
--- /dev/null
+++ b/arch/powerpc/platforms/85xx/bootpage.S
@@ -0,0 +1,206 @@
+/*
+ * Copyright 2008-2010 Freescale Semiconductor, Inc.
+ * Kumar Gala <kumar.gala@freescale.com>
+ * This file is taken from u-boot
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation;
+ */
+#include <linux/init.h>
+#include <linux/threads.h>
+#include <asm/processor.h>
+#include <asm/page.h>
+#include <asm/mmu.h>
+#include <asm/pgtable.h>
+#include <asm/cputable.h>
+#include <asm/thread_info.h>
+#include <asm/ppc_asm.h>
+#include <asm/asm-offsets.h>
+#include <asm/cache.h>
+
+/* To boot secondary cpus, we need a place for them to start up.
+ * Normally, they start at 0xfffffffc, but that's usually the
+ * firmware, and we don't want to have to run the firmware again.
+ * Instead, the primary cpu will set the BPTR to point here to
+ * this page.  We then set up the core, and head to
+ * start_secondary.  Note that this means that the code below
+ * must never exceed 1023 instructions (the branch at the end
+ * would then be the 1024th).
+ */
+	.globl	__secondary_start_page
+	.align	12
+__secondary_start_page:
+	lis	r3, 0x8000		/* enable machine check */
+#ifndef CONFIG_PPC_E500MC
+	ori	r3,r3,0x4000		/* enable Timebase */
+#endif
+#ifdef CONFIG_PHYS_64BIT
+	/* for 36-bit addressing */
+	ori	r3,r3,0x0080		/* enable MAS7 updates */
+#endif
+	mtspr	SPRN_HID0,r3
+
+#ifndef CONFIG_PPC_E500MC
+	li	r3,0x3000		/* Addr streaming & broadcast */
+	mtspr	SPRN_HID1,r3
+#endif
+
+	/* Enable branch prediction */
+	li	r3,0x201
+	mtspr	SPRN_BUCSR,r3
+
+	/* Ensure TB is 0 */
+	li	r3,0
+	mttbl	r3
+	mttbu	r3
+
+	mfspr	r0,SPRN_L1CSR1
+	ori	r0,r0,0x0003		/* Enable/invalidate the I-Cache */
+	mtspr	SPRN_L1CSR1,r0
+	isync
+
+
+	mfspr	r0,SPRN_L1CSR0
+	ori	r0,r0,0x0003		/* Enable/invalidate the D-Cache */
+	msync
+	isync
+	mtspr	SPRN_L1CSR0,r0
+	isync
+
+#define toreset(x) (x - __secondary_start_page + 0xfffff000)
+
+	/* get our PIR to figure out our table entry */
+	lis	r3,toreset(__spin_table)@h
+	ori	r3,r3,toreset(__spin_table)@l
+
+	/* r10 has the base address for the entry */
+	mfspr	r0,SPRN_PIR
+#ifdef CONFIG_PPC_E500MC
+	rlwinm	r4,r0,27,27,31
+#else
+	mr	r4,r0
+#endif
+	slwi	r8,r4,5
+	add	r10,r3,r8
+
+#define EPAPR_MAGIC		(0x45504150)
+#define ENTRY_ADDR_UPPER	0
+#define ENTRY_ADDR_LOWER	4
+#define ENTRY_R3_UPPER		8
+#define ENTRY_R3_LOWER		12
+#define ENTRY_RESV		16
+#define ENTRY_PIR		20
+#define ENTRY_R6_UPPER		24
+#define ENTRY_R6_LOWER		28
+#define ENTRY_SIZE		32
+
+	/* setup the entry */
+	li	r3,0
+	li	r8,1
+	stw	r0,ENTRY_PIR(r10)
+	stw	r3,ENTRY_ADDR_UPPER(r10)
+	stw	r8,ENTRY_ADDR_LOWER(r10)
+	stw	r3,ENTRY_R3_UPPER(r10)
+	stw	r4,ENTRY_R3_LOWER(r10)
+	stw	r3,ENTRY_R6_UPPER(r10)
+	stw	r3,ENTRY_R6_LOWER(r10)
+
+	/* setup mapping for AS = 1, and jump there */
+	lis	r11,(MAS0_TLBSEL(1)|MAS0_ESEL(1))@h
+	mtspr	SPRN_MAS0,r11
+	lis	r11,(MAS1_VALID|MAS1_IPROT)@h
+	ori	r11,r11,(MAS1_TS|MAS1_TSIZE(BOOK3E_PAGESZ_4K))@l
+	mtspr	SPRN_MAS1,r11
+	lis	r11,(0xfffff000|MAS2_I|MAS2_G)@h
+	ori	r11,r11,(0xfffff000|MAS2_I|MAS2_G)@l
+	mtspr	SPRN_MAS2,r11
+	lis	r11,(0xfffff000|MAS3_SX|MAS3_SW|MAS3_SR)@h
+	ori	r11,r11,(0xfffff000|MAS3_SX|MAS3_SW|MAS3_SR)@l
+	mtspr	SPRN_MAS3,r11
+	tlbwe
+
+	bl	1f
+1:	mflr	r11
+	addi	r11,r11,28
+	mfmsr	r13
+	ori	r12,r13,MSR_IS|MSR_DS@l
+
+	mtspr	SPRN_SRR0,r11
+	mtspr	SPRN_SRR1,r12
+	rfi
+
+	/* spin waiting for addr */
+2:
+	lwz	r4,ENTRY_ADDR_LOWER(r10)
+	andi.	r11,r4,1
+	bne	2b
+	isync
+
+	/* get the upper bits of the addr */
+	lwz	r11,ENTRY_ADDR_UPPER(r10)
+
+	/* setup branch addr */
+	mtspr	SPRN_SRR0,r4
+
+	/* mark the entry as released */
+	li	r8,3
+	stw	r8,ENTRY_ADDR_LOWER(r10)
+
+	/* mask by ~64M to setup our tlb we will jump to */
+	rlwinm	r12,r4,0,0,5
+
+	/* setup r3, r4, r5, r6, r7, r8, r9 */
+	lwz	r3,ENTRY_R3_LOWER(r10)
+	li	r4,0
+	li	r5,0
+	lwz	r6,ENTRY_R6_LOWER(r10)
+	lis	r7,(64*1024*1024)@h
+	li	r8,0
+	li	r9,0
+
+	/* load up the pir */
+	lwz	r0,ENTRY_PIR(r10)
+	mtspr	SPRN_PIR,r0
+	mfspr	r0,SPRN_PIR
+	stw	r0,ENTRY_PIR(r10)
+
+	mtspr	SPRN_IVPR,r12
+/*
+ * Coming here, we know the cpu has one TLB mapping in TLB1[0]
+ * which maps 0xfffff000-0xffffffff one-to-one.  We set up a
+ * second mapping that maps addr 1:1 for 64M, and then we jump to
+ * addr
+ */
+	lis	r10,(MAS0_TLBSEL(1)|MAS0_ESEL(0))@h
+	mtspr	SPRN_MAS0,r10
+	lis	r10,(MAS1_VALID|MAS1_IPROT)@h
+	ori	r10,r10,(MAS1_TSIZE(BOOK3E_PAGESZ_64M))@l
+	mtspr	SPRN_MAS1,r10
+	/* WIMGE = 0b00000 for now */
+	mtspr	SPRN_MAS2,r12
+	ori	r12,r12,(MAS3_SX|MAS3_SW|MAS3_SR)
+	mtspr	SPRN_MAS3,r12
+#ifdef CONFIG_PHYS_64BIT
+	mtspr	SPRN_MAS7,r11
+#endif
+	tlbwe
+
+/* Now we have another mapping for this page, so we jump to that
+ * mapping
+ */
+	mtspr	SPRN_SRR1,r13
+	rfi
+
+	.align L1_CACHE_SHIFT
+	.globl __spin_table
+__spin_table:
+	.space NR_CPUS*ENTRY_SIZE
+
+	/* Fill in the empty space.  The actual reset vector is
+	 * the last word of the page */
+__secondary_start_code_end:
+	.space 4092 - (__secondary_start_code_end - __secondary_start_page)
+
+__secondary_reset_vector:
+	b	__secondary_start_page
diff --git a/arch/powerpc/platforms/85xx/smp.c b/arch/powerpc/platforms/85xx/smp.c
index 1e8aec8..2ef3e8e 100644
--- a/arch/powerpc/platforms/85xx/smp.c
+++ b/arch/powerpc/platforms/85xx/smp.c
@@ -17,6 +17,7 @@
 #include <linux/of.h>
 #include <linux/kexec.h>
 #include <linux/highmem.h>
+#include <linux/cpu.h>
 
 #include <asm/machdep.h>
 #include <asm/pgtable.h>
@@ -28,26 +29,116 @@
 #include <sysdev/fsl_soc.h>
 #include <sysdev/mpic.h>
 
+#define MPC85xx_BPTR_OFF		0x00020
+#define MPC85xx_BPTR_EN			0x80000000
+#define MPC85xx_ECM_EEBPCR_OFF		0x01010
+#define MPC85xx_PIC_PIR_OFF		0x41090
+
+extern void mpc85xx_cpu_down(void) __attribute__((noreturn));
 extern void __early_start(void);
+extern void __secondary_start_page(void);
+extern volatile unsigned long __spin_table;
+
+struct epapr_entry {
+	u32	addr_h;
+	u32	addr_l;
+	u32	r3_h;
+	u32	r3_l;
+	u32	reserved;
+	u32	pir;
+	u32	r6_h;
+	u32	r6_l;
+};
 
-#define BOOT_ENTRY_ADDR_UPPER	0
-#define BOOT_ENTRY_ADDR_LOWER	1
-#define BOOT_ENTRY_R3_UPPER	2
-#define BOOT_ENTRY_R3_LOWER	3
-#define BOOT_ENTRY_RESV		4
-#define BOOT_ENTRY_PIR		5
-#define BOOT_ENTRY_R6_UPPER	6
-#define BOOT_ENTRY_R6_LOWER	7
-#define NUM_BOOT_ENTRY		8
-#define SIZE_BOOT_ENTRY		(NUM_BOOT_ENTRY * sizeof(u32))
+/* access per cpu vars from generic smp.c */
+DECLARE_PER_CPU(int, cpu_state);
 
-static void __init
+#ifdef CONFIG_HOTPLUG_CPU
+static void __cpuinit
+smp_85xx_mach_cpu_die(void)
+{
+	__get_cpu_var(cpu_state) = CPU_DEAD;
+	smp_wmb();
+
+	local_irq_disable();
+	idle_task_exit();
+	mtspr(SPRN_TSR, TSR_ENW | TSR_WIS | TSR_DIS | TSR_FIS);
+	mtspr(SPRN_TCR, 0);
+	mpc85xx_cpu_down();
+}
+
+static void __cpuinit
+smp_85xx_reset_core(int nr)
+{
+	__iomem u32 *ecm_vaddr;
+	__iomem u32 *pic_vaddr;
+	u32 pcr, pir, cpu;
+
+	cpu = (1 << 24) << nr;
+	ecm_vaddr = ioremap(get_immrbase() + MPC85xx_ECM_EEBPCR_OFF, 4);
+	pcr = in_be32(ecm_vaddr);
+	if (pcr & cpu) {
+		pic_vaddr = ioremap(get_immrbase() + MPC85xx_PIC_PIR_OFF, 4);
+		pir = in_be32(pic_vaddr);
+		/* reset assert */
+		pir |= (1 << nr);
+		out_be32(pic_vaddr, pir);
+		pir = in_be32(pic_vaddr);
+		pir &= ~(1 << nr);
+		/* reset negate */
+		out_be32(pic_vaddr, pir);
+		(void)in_be32(pic_vaddr);
+		iounmap(pic_vaddr);
+	} else {
+		out_be32(ecm_vaddr, pcr | cpu);
+		(void)in_be32(ecm_vaddr);
+	}
+	iounmap(ecm_vaddr);
+}
+
+static int __cpuinit
+smp_85xx_map_bootpg(unsigned long pa)
+{
+	__iomem u32 *bootpg_ptr;
+	u32 bptr;
+
+	/* Get the BPTR */
+	bootpg_ptr = ioremap(get_immrbase() + MPC85xx_BPTR_OFF, 4);
+
+	/* Set the BPTR to the secondary boot page */
+	(void)in_be32(bootpg_ptr);
+
+	bptr = (MPC85xx_BPTR_EN | (pa >> 12));
+	out_be32(bootpg_ptr, bptr);
+	(void)in_be32(bootpg_ptr);
+	iounmap(bootpg_ptr);
+	return 0;
+}
+
+static int __cpuinit
+smp_85xx_unmap_bootpg(void)
+{
+	__iomem u32 *bootpg_ptr;
+
+	/* Get the BPTR */
+	bootpg_ptr = ioremap(get_immrbase() + MPC85xx_BPTR_OFF, 4);
+
+	/* Restore the BPTR */
+	if (in_be32(bootpg_ptr) & MPC85xx_BPTR_EN) {
+		out_be32(bootpg_ptr, 0);
+		(void)in_be32(bootpg_ptr);
+	}
+	iounmap(bootpg_ptr);
+	return 0;
+}
+#endif
+
+static void __cpuinit
 smp_85xx_kick_cpu(int nr)
 {
 	unsigned long flags;
-	const u64 *cpu_rel_addr;
-	__iomem u32 *bptr_vaddr;
-	struct device_node *np;
+	phys_addr_t cpu_rel_addr;
+	__iomem struct epapr_entry *epapr;
 	int n = 0;
 	int ioremappable;
 
@@ -55,41 +146,83 @@ smp_85xx_kick_cpu(int nr)
 
 	pr_debug("smp_85xx_kick_cpu: kick CPU #%d\n", nr);
 
-	np = of_get_cpu_node(nr, NULL);
-	cpu_rel_addr = of_get_property(np, "cpu-release-addr", NULL);
+	if (system_state < SYSTEM_RUNNING) {
+		/* booting, using __spin_table from u-boot */
+		struct device_node *np;
+		const u64 *prop;
 
-	if (cpu_rel_addr == NULL) {
-		printk(KERN_ERR "No cpu-release-addr for cpu %d\n", nr);
-		return;
-	}
+		np = of_get_cpu_node(nr, NULL);
+		if (np == NULL)
+			return;
 
-	/*
-	 * A secondary core could be in a spinloop in the bootpage
-	 * (0xfffff000), somewhere in highmem, or somewhere in lowmem.
-	 * The bootpage and highmem can be accessed via ioremap(), but
-	 * we need to directly access the spinloop if its in lowmem.
-	 */
-	ioremappable = *cpu_rel_addr > virt_to_phys(high_memory);
+		prop = of_get_property(np, "cpu-release-addr", NULL);
+		if (prop == NULL) {
+			of_node_put(np);
+			printk(KERN_ERR "No cpu-release-addr for cpu %d\n", nr);
+			return;
+		}
+		cpu_rel_addr = (phys_addr_t)*prop;
+		of_node_put(np);
+
+		/*
+		 * A secondary core could be in a spinloop in the bootpage
+		 * (0xfffff000), somewhere in highmem, or somewhere in lowmem.
+		 * The bootpage and highmem can be accessed via ioremap(), but
+		 * we need to directly access the spinloop if its in lowmem.
+		 */
+		ioremappable = cpu_rel_addr > virt_to_phys(high_memory);
+
+		if (ioremappable)
+			epapr = ioremap(cpu_rel_addr,
+						sizeof(struct epapr_entry));
+		else
+			epapr = phys_to_virt(cpu_rel_addr);
+
+		local_irq_save(flags);
+	} else {
+#ifdef CONFIG_HOTPLUG_CPU
+		/* spin table in kernel, no need to remap */
+		ioremappable = 0;
+		epapr = (void *)&__spin_table + nr * sizeof(struct epapr_entry);
 
-	/* Map the spin table */
-	if (ioremappable)
-		bptr_vaddr = ioremap(*cpu_rel_addr, SIZE_BOOT_ENTRY);
-	else
-		bptr_vaddr = phys_to_virt(*cpu_rel_addr);
+		/* prevent bootpage from being accessed by others */
+		local_irq_save(flags);
+
+		smp_85xx_map_bootpg(__pa(__secondary_start_page));
 
-	local_irq_save(flags);
+		smp_85xx_reset_core(nr);
 
-	out_be32(bptr_vaddr + BOOT_ENTRY_PIR, nr);
+		/* wait until core(nr) is ready... */
+		while ((in_be32(&epapr->addr_l) != 1) && (++n < 1000))
+			udelay(100);
+
+		if (n == 1000) {
+			pr_err("timeout waiting for core%d to reset\n",
+					nr);
+			goto out;
+		}
+#else
+		pr_err("runtime kick cpu not supported\n");
+		return;
+#endif
+	}
+
+	out_be32(&epapr->pir, nr);
 #ifdef CONFIG_PPC32
-	out_be32(bptr_vaddr + BOOT_ENTRY_ADDR_LOWER, __pa(__early_start));
+	/* clear the acknowledge status */
+	__secondary_hold_acknowledge = -1;
+	out_be32(&epapr->addr_l, __pa(__early_start));
 
 	if (!ioremappable)
-		flush_dcache_range((ulong)bptr_vaddr,
-				(ulong)(bptr_vaddr + SIZE_BOOT_ENTRY));
+		flush_dcache_range((ulong)epapr,
+				(ulong)epapr + sizeof(struct epapr_entry));
 
 	/* Wait a bit for the CPU to ack. */
+	n = 0;
 	while ((__secondary_hold_acknowledge != nr) && (++n < 1000))
 		mdelay(1);
+	if (n == 1000)
+		pr_err("timeout waiting for core%d to ack\n", nr);
 #else
 	out_be64((u64 *)(bptr_vaddr + BOOT_ENTRY_ADDR_UPPER),
 		__pa((u64)*((unsigned long long *) generic_secondary_smp_init)));
@@ -97,12 +230,15 @@ smp_85xx_kick_cpu(int nr)
 	smp_generic_kick_cpu(nr);
 #endif
 
+out:
+#ifdef CONFIG_HOTPLUG_CPU
+	if (system_state >= SYSTEM_RUNNING)
+		smp_85xx_unmap_bootpg();
+#endif
 	local_irq_restore(flags);
 
 	if (ioremappable)
-		iounmap(bptr_vaddr);
-
-	pr_debug("waited %d msecs for CPU #%d.\n", n, nr);
+		iounmap(epapr);
 }
 
 static void __init
@@ -232,6 +368,12 @@ void __init mpc85xx_smp_init(void)
 
 	BUG_ON(!smp_85xx_ops.message_pass);
 
+#ifdef CONFIG_HOTPLUG_CPU
+	smp_85xx_ops.cpu_disable   = generic_cpu_disable;
+	smp_85xx_ops.cpu_die	= generic_cpu_die;
+	ppc_md.cpu_die		= smp_85xx_mach_cpu_die;
+#endif
+
 	smp_ops = &smp_85xx_ops;
 
 #ifdef CONFIG_KEXEC
-- 
1.6.6-rc1.GIT

^ permalink raw reply related

* [PATCH 1/7] powerpc/85xx: re-enable timebase sync disabled by KEXEC patch
From: Li Yang @ 2010-12-03 12:34 UTC (permalink / raw)
  To: linuxppc-dev

The timebase sync is not only necessary when using KEXEC.  It should also
be used by normal boot up and cpu hotplug.  Remove the ifdef added by
the KEXEC patch.  Fix a problem that cpu hotplugging freezes the whole system.

Signed-off-by: Jin Qing <b24347@freescale.com>
Singed-off-by: Li Yang <leoli@freescale.com>
---
 arch/powerpc/platforms/85xx/smp.c |    4 +---
 1 files changed, 1 insertions(+), 3 deletions(-)

diff --git a/arch/powerpc/platforms/85xx/smp.c b/arch/powerpc/platforms/85xx/smp.c
index 5c91a99..1e8aec8 100644
--- a/arch/powerpc/platforms/85xx/smp.c
+++ b/arch/powerpc/platforms/85xx/smp.c
@@ -2,7 +2,7 @@
  * Author: Andy Fleming <afleming@freescale.com>
  * 	   Kumar Gala <galak@kernel.crashing.org>
  *
- * Copyright 2006-2008 Freescale Semiconductor Inc.
+ * Copyright 2006-2010 Freescale Semiconductor Inc.
  *
  * This program is free software; you can redistribute  it and/or modify it
  * under  the terms of  the GNU General  Public License as published by the
@@ -115,10 +115,8 @@ smp_85xx_setup_cpu(int cpu_nr)
 
 struct smp_ops_t smp_85xx_ops = {
 	.kick_cpu = smp_85xx_kick_cpu,
-#ifdef CONFIG_KEXEC
 	.give_timebase	= smp_generic_give_timebase,
 	.take_timebase	= smp_generic_take_timebase,
-#endif
 };
 
 #ifdef CONFIG_KEXEC
-- 
1.6.6-rc1.GIT

^ permalink raw reply related

* Re: linux-next: build failure after merge of the final tree (powerpc tree related)
From: Josh Boyer @ 2010-12-03 11:26 UTC (permalink / raw)
  To: Benjamin Herrenschmidt
  Cc: Stephen Rothwell, linux-kernel, linux-next, Paul Mackerras,
	Anton Blanchard, linuxppc-dev
In-Reply-To: <1291355998.32570.417.camel@pasglop>

On Fri, Dec 03, 2010 at 04:59:58PM +1100, Benjamin Herrenschmidt wrote:
>On Fri, 2010-12-03 at 16:32 +1100, Stephen Rothwell wrote:
>> 
>> After merging the  tree, today's linux-next build (powerpc allmodconfig)
>> failed like this:
>> 
>> arch/powerpc/lib/hweight_64.S: Assembler messages:
>> arch/powerpc/lib/hweight_64.S:52: Error: Unrecognized opcode: `popcntw'
>> arch/powerpc/lib/hweight_64.S:77: Error: Unrecognized opcode: `popcntw'
>> arch/powerpc/lib/hweight_64.S:106: Error: Unrecognized opcode: `popcntd'
>> 
>> This is with:
>> 
>> powerpc64-linux-gcc (GCC) 4.4.0
>> GNU assembler (GNU Binutils) 2.19.1
>> 
>> Caused by commit 64ff31287693c1f325cb9cb049569c1611438ef1 ("powerpc: Add
>> support for popcnt instructions"). 
>
>
>This toolchain is a bit ancient I suppose... Anton, do you reckon we

SLES 11 SP1 still uses gcc 4.3.  4.4.0 is not ancient by any means.  Neither is binutils 2.19.1.

>should use .long based macros for these for the time being or just
>require a newer binutils ?

.long macros sound like the proper solution.

josh

^ permalink raw reply

* Re: linux-next: build failure after merge of the final tree (powerpc tree related)
From: Stephen Rothwell @ 2010-12-03  6:08 UTC (permalink / raw)
  To: Benjamin Herrenschmidt
  Cc: linuxppc-dev, linux-next, Paul Mackerras, linux-kernel,
	Anton Blanchard
In-Reply-To: <1291355998.32570.417.camel@pasglop>

[-- Attachment #1: Type: text/plain, Size: 434 bytes --]

On Fri, 03 Dec 2010 16:59:58 +1100 Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:
>
> This toolchain is a bit ancient I suppose... Anton, do you reckon we
> should use .long based macros for these for the time being or just
> require a newer binutils ?

The currently documented minimum binutils is 2.12 ...

-- 
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au
http://www.canb.auug.org.au/~sfr/

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

^ permalink raw reply

* Re: linux-next: build failure after merge of the final tree (powerpc tree related)
From: Benjamin Herrenschmidt @ 2010-12-03  5:59 UTC (permalink / raw)
  To: Stephen Rothwell
  Cc: linuxppc-dev, linux-next, Paul Mackerras, linux-kernel,
	Anton Blanchard
In-Reply-To: <20101203163206.72258fb8.sfr@canb.auug.org.au>

On Fri, 2010-12-03 at 16:32 +1100, Stephen Rothwell wrote:
> 
> After merging the  tree, today's linux-next build (powerpc allmodconfig)
> failed like this:
> 
> arch/powerpc/lib/hweight_64.S: Assembler messages:
> arch/powerpc/lib/hweight_64.S:52: Error: Unrecognized opcode: `popcntw'
> arch/powerpc/lib/hweight_64.S:77: Error: Unrecognized opcode: `popcntw'
> arch/powerpc/lib/hweight_64.S:106: Error: Unrecognized opcode: `popcntd'
> 
> This is with:
> 
> powerpc64-linux-gcc (GCC) 4.4.0
> GNU assembler (GNU Binutils) 2.19.1
> 
> Caused by commit 64ff31287693c1f325cb9cb049569c1611438ef1 ("powerpc: Add
> support for popcnt instructions"). 


This toolchain is a bit ancient I suppose... Anton, do you reckon we
should use .long based macros for these for the time being or just
require a newer binutils ?

Cheers,
Ben.

^ permalink raw reply

* linux-next: build failure after merge of the final tree (powerpc tree related)
From: Stephen Rothwell @ 2010-12-03  5:32 UTC (permalink / raw)
  To: Benjamin Herrenschmidt, Paul Mackerras, linuxppc-dev
  Cc: linux-next, linux-kernel, Anton Blanchard

[-- Attachment #1: Type: text/plain, Size: 688 bytes --]

Hi all,

After merging the  tree, today's linux-next build (powerpc allmodconfig)
failed like this:

arch/powerpc/lib/hweight_64.S: Assembler messages:
arch/powerpc/lib/hweight_64.S:52: Error: Unrecognized opcode: `popcntw'
arch/powerpc/lib/hweight_64.S:77: Error: Unrecognized opcode: `popcntw'
arch/powerpc/lib/hweight_64.S:106: Error: Unrecognized opcode: `popcntd'

This is with:

powerpc64-linux-gcc (GCC) 4.4.0
GNU assembler (GNU Binutils) 2.19.1

Caused by commit 64ff31287693c1f325cb9cb049569c1611438ef1 ("powerpc: Add
support for popcnt instructions").

-- 
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au
http://www.canb.auug.org.au/~sfr/

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

^ permalink raw reply

* RE: [PATCH 2/2][v3] rapidio, powerpc/85xx: Error interrupt handler for sRIO.
From: Xie Shaohui-B21989 @ 2010-12-03  3:29 UTC (permalink / raw)
  To: Bounine, Alexandre, linuxppc-dev
  Cc: akpm, Gala Kumar-B11780, Li Yang-R58472, Zang Roy-R61911
In-Reply-To: <0CE8B6BE3C4AD74AB97D9D29BD24E552015F5E40@CORPEXCH1.na.ads.idt.com>

[-- Attachment #1: Type: text/plain, Size: 1324 bytes --]

Hi Alex,

 

May I ask when would these patches be applied to mainline?

 

 

Best Regards, 
Shaohui Xie 

 

From: Bounine, Alexandre [mailto:Alexandre.Bounine@idt.com] 
Sent: Wednesday, December 01, 2010 4:49 AM
To: Xie Shaohui-B21989; linuxppc-dev@lists.ozlabs.org
Cc: akpm@linux-foundation.org; Li Yang-R58472; Gala Kumar-B11780; Zang
Roy-R61911
Subject: RE: [PATCH 2/2][v3] rapidio, powerpc/85xx: Error interrupt
handler for sRIO.

 

Applies correctly now. Lab tested on 8548/RIO setup.

Alex.

> -----Original Message-----
> From: Shaohui Xie [mailto:b21989@freescale.com]
> Sent: Thursday, November 18, 2010 1:58 AM
> To: linuxppc-dev@lists.ozlabs.org
> Cc: akpm@linux-foundation.org; Shaohui Xie; Li Yang; Kumar Gala; Roy
Zang; Bounine, Alexandre
> Subject: [PATCH 2/2][v3] rapidio, powerpc/85xx: Error interrupt
handler for sRIO.
>
> The sRIO controller reports errors to the core with one signal, it
uses
> register EPWISR to provides the core quick access to where the error
occurred.
> The EPWISR indicates that there are 4 interrupts sources, port1,
port2, message
> unit and port write receive, but the sRIO driver does not support
port2
> for now, still the handler takes care of port2.
> Currently the handler only clear error status without any recovery.


[-- Attachment #2: Type: text/html, Size: 7931 bytes --]

^ permalink raw reply

* Re: Problem with MPC5121 custom board.
From: Wolfgang Denk @ 2010-12-02 22:06 UTC (permalink / raw)
  To: CTAG / Moisés Domínguez; +Cc: linuxppc-dev
In-Reply-To: <765B188B547A48E7951D86ACC3544C54@ctag>

Dear =?iso-8859-1?Q?CTAG_/_Mois=E9s_Dom=EDnguez?=,

In message <765B188B547A48E7951D86ACC3544C54@ctag> you wrote:
> 
> We have a problem with custom design based on ads5121. May be you could
> provide us any clue about what may be happening.
> 
> Sometimes, when the board is powered-on:
> 
> - U-boot does not start (at least serial port does not output anything, I
> don't know if some piece of u-boot is executed).

In an overwhelming majority of such cases this is an indication of
incorrect and/or incomplete initialization of your system RAM. See the
FAQ at http://www.denx.de/wiki/view/DULG/UBootCrashAfterRelocation

Note that U-Boot related questions (and this is one) should better go
to the U-Boot mailing list instead (see
http://lists.denx.de/mailman/listinfo/u-boot)

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
You can only live once, but if you do it right, once is enough.

^ permalink raw reply

* Re: Getting the IRQ number (Was: Basic driver devel questions ?)
From: Benjamin Herrenschmidt @ 2010-12-02 20:22 UTC (permalink / raw)
  To: Guillaume Dargaud; +Cc: linuxppc-dev
In-Reply-To: <201012021847.09367.dargaud@lpsc.in2p3.fr>

On Thu, 2010-12-02 at 18:47 +0100, Guillaume Dargaud wrote:
> 
> OK, learning by example then.
> 
> Another basic question: 
> what's the difference between of_register_platform_driver() and 
> platform_driver_register() ?
> I assume I must use the first one if I want to use
> irq_of_parse_and_map... 

No. of_platform_drivers are more/less obsolete. Normal platform drivers
can now be associated with a device-tree node just fine.

Cheers,
Ben.

^ permalink raw reply

* Re: Getting the IRQ number (Was: Basic driver devel questions ?)
From: Guillaume Dargaud @ 2010-12-02 17:47 UTC (permalink / raw)
  To: linuxppc-dev
In-Reply-To: <AANLkTinnXoU2kPEB-OmDWoWom18Nu9r0jOT==zWmRp4k@mail.gmail.com>

>      grep -r irq_of_parse_and_map *
> 
> will give you plenty of examples.

OK, learning by example then.

Another basic question: 
what's the difference between of_register_platform_driver() and 
platform_driver_register() ?
I assume I must use the first one if I want to use irq_of_parse_and_map...
-- 
Guillaume Dargaud
http://www.gdargaud.net/

^ permalink raw reply

* RE: [v4] ppc44x:PHY fixup for USB on canyonlands board
From: Rupjyoti Sarmah @ 2010-12-02 16:50 UTC (permalink / raw)
  To: Milton Miller; +Cc: linuxppc-dev, linux-kernel
In-Reply-To: <mdm-canyonlands-reply2@mdm.bga.com>

>>I prepared these comments for v1, but aparently forgot to send them,
>>so I'll do so now.

Thanks for your inputs. I will resubmit an updated patch soon with the
changes.

Regards,
Rup

^ permalink raw reply

* Re: Getting the IRQ number (Was: Basic driver devel questions ?)
From: Timur Tabi @ 2010-12-02 16:17 UTC (permalink / raw)
  To: Guillaume Dargaud; +Cc: linuxppc-dev
In-Reply-To: <201012021636.05963.dargaud@lpsc.in2p3.fr>

On Thu, Dec 2, 2010 at 9:36 AM, Guillaume Dargaud <dargaud@lpsc.in2p3.fr> wrote:

> Sounds like what I need... But I don't find irq_of_parse_and_map() in LDD 3rd
> ed and very few info in google. Are there some examples of use somewhere ?

     grep -r irq_of_parse_and_map *

will give you plenty of examples.

-- 
Timur Tabi
Linux kernel developer at Freescale

^ permalink raw reply

* Re: Getting the IRQ number (Was: Basic driver devel questions ?)
From: Guillaume Dargaud @ 2010-12-02 15:36 UTC (permalink / raw)
  To: linuxppc-dev
In-Reply-To: <20101201124142.15f75af7@udp111988uds.am.freescale.net>

> How was your driver probed?  If you can get a pointer to the device
> node, use irq_of_parse_and_map() to get a virtual irq that you can pass
> to request_irq().

Sounds like what I need... But I don't find irq_of_parse_and_map() in LDD 3rd 
ed and very few info in google. Are there some examples of use somewhere ?

> Can your driver tolerate it being shared?  If so, request it as shared.
I think so (not sure).
-- 
Guillaume Dargaud
http://www.gdargaud.net/

^ permalink raw reply

* Problem with MPC5121 custom board.
From: CTAG / Moisés Domínguez @ 2010-12-02  8:48 UTC (permalink / raw)
  To: linuxppc-dev


Hi all,

We have a problem with custom design based on ads5121. May be you could
provide us any clue about what may be happening.

Sometimes, when the board is powered-on:

- U-boot does not start (at least serial port does not output anything, =
I
don't know if some piece of u-boot is executed).
- LPC_DIV and IPS_DIV are not configured so u-boot does not reach this =
point
either.
- The only way to get the board run once reached this point is power-off =
and
power-on again (That's: 3.3V, 1.8V. 1.4V).
- If we only turn-off 1.8V and 1.4V that's not enough to get the board
working.

When this happens we:

- Measured lpc_clock which is running at 33 MHZ so PLLs are working so =
reset
word is being read right.
- Measured all voltages (3.3V, 1.8V. 1.4V) and are OK.
- We read RCWL and RCWH with JTAG and have the value we expect.
- Could erase the flash with JTAG but we couldn't reprogram it.

We don't know what else to test in order to detect where the problem is. =
We
suppose that, as the only way to run again is reset the 3.3V, there must =
be
any peripheral that is not being working properly, but we don't know how =
to
find it and why this is happening.

Some clue is much appreciated.

Regards,

Moises.

^ permalink raw reply

* Re: [PATCH] of/address: use propper endianess in get_flags
From: Benjamin Herrenschmidt @ 2010-12-01 23:15 UTC (permalink / raw)
  To: Sebastian Andrzej Siewior; +Cc: sodaville, linuxppc-dev
In-Reply-To: <20101201095446.GA26877@www.tglx.de>

On Wed, 2010-12-01 at 10:54 +0100, Sebastian Andrzej Siewior wrote:
> This patch changes u32 to __be32 for all "ranges", "prop" and "addr" and
> such. Those variables are pointing to the device tree which containts
> intergers in big endian format.
> Most functions are doing it right because of_read_number() is doing the
> right thing for them. of_bus_isa_get_flags(), of_bus_pci_get_flags() and
> of_bus_isa_map() were accessing the data directly and were doing it wrong.
> 
> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>

Acked-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---

> ---
>  arch/powerpc/include/asm/prom.h |    2 +-
>  drivers/of/address.c            |   54 ++++++++++++++++++++------------------
>  include/linux/of_address.h      |    6 ++--
>  3 files changed, 32 insertions(+), 30 deletions(-)
> 
> diff --git a/arch/powerpc/include/asm/prom.h b/arch/powerpc/include/asm/prom.h
> index ae26f2e..ab34f60 100644
> --- a/arch/powerpc/include/asm/prom.h
> +++ b/arch/powerpc/include/asm/prom.h
> @@ -42,7 +42,7 @@ extern void pci_create_OF_bus_map(void);
>  
>  /* Translate a DMA address from device space to CPU space */
>  extern u64 of_translate_dma_address(struct device_node *dev,
> -				    const u32 *in_addr);
> +				    const __be32 *in_addr);
>  
>  #ifdef CONFIG_PCI
>  extern unsigned long pci_address_to_pio(phys_addr_t address);
> diff --git a/drivers/of/address.c b/drivers/of/address.c
> index 3a1c7e7..b4559c5 100644
> --- a/drivers/of/address.c
> +++ b/drivers/of/address.c
> @@ -12,13 +12,13 @@
>  			(ns) > 0)
>  
>  static struct of_bus *of_match_bus(struct device_node *np);
> -static int __of_address_to_resource(struct device_node *dev, const u32 *addrp,
> -				    u64 size, unsigned int flags,
> +static int __of_address_to_resource(struct device_node *dev,
> +		const __be32 *addrp, u64 size, unsigned int flags,
>  				    struct resource *r);
>  
>  /* Debug utility */
>  #ifdef DEBUG
> -static void of_dump_addr(const char *s, const u32 *addr, int na)
> +static void of_dump_addr(const char *s, const __be32 *addr, int na)
>  {
>  	printk(KERN_DEBUG "%s", s);
>  	while (na--)
> @@ -26,7 +26,7 @@ static void of_dump_addr(const char *s, const u32 *addr, int na)
>  	printk("\n");
>  }
>  #else
> -static void of_dump_addr(const char *s, const u32 *addr, int na) { }
> +static void of_dump_addr(const char *s, const __be32 *addr, int na) { }
>  #endif
>  
>  /* Callbacks for bus specific translators */
> @@ -36,10 +36,10 @@ struct of_bus {
>  	int		(*match)(struct device_node *parent);
>  	void		(*count_cells)(struct device_node *child,
>  				       int *addrc, int *sizec);
> -	u64		(*map)(u32 *addr, const u32 *range,
> +	u64		(*map)(u32 *addr, const __be32 *range,
>  				int na, int ns, int pna);
>  	int		(*translate)(u32 *addr, u64 offset, int na);
> -	unsigned int	(*get_flags)(const u32 *addr);
> +	unsigned int	(*get_flags)(const __be32 *addr);
>  };
>  
>  /*
> @@ -55,7 +55,7 @@ static void of_bus_default_count_cells(struct device_node *dev,
>  		*sizec = of_n_size_cells(dev);
>  }
>  
> -static u64 of_bus_default_map(u32 *addr, const u32 *range,
> +static u64 of_bus_default_map(u32 *addr, const __be32 *range,
>  		int na, int ns, int pna)
>  {
>  	u64 cp, s, da;
> @@ -85,7 +85,7 @@ static int of_bus_default_translate(u32 *addr, u64 offset, int na)
>  	return 0;
>  }
>  
> -static unsigned int of_bus_default_get_flags(const u32 *addr)
> +static unsigned int of_bus_default_get_flags(const __be32 *addr)
>  {
>  	return IORESOURCE_MEM;
>  }
> @@ -110,10 +110,10 @@ static void of_bus_pci_count_cells(struct device_node *np,
>  		*sizec = 2;
>  }
>  
> -static unsigned int of_bus_pci_get_flags(const u32 *addr)
> +static unsigned int of_bus_pci_get_flags(const __be32 *addr)
>  {
>  	unsigned int flags = 0;
> -	u32 w = addr[0];
> +	u32 w = be32_to_cpup(addr);
>  
>  	switch((w >> 24) & 0x03) {
>  	case 0x01:
> @@ -129,7 +129,8 @@ static unsigned int of_bus_pci_get_flags(const u32 *addr)
>  	return flags;
>  }
>  
> -static u64 of_bus_pci_map(u32 *addr, const u32 *range, int na, int ns, int pna)
> +static u64 of_bus_pci_map(u32 *addr, const __be32 *range, int na, int ns,
> +		int pna)
>  {
>  	u64 cp, s, da;
>  	unsigned int af, rf;
> @@ -160,7 +161,7 @@ static int of_bus_pci_translate(u32 *addr, u64 offset, int na)
>  	return of_bus_default_translate(addr + 1, offset, na - 1);
>  }
>  
> -const u32 *of_get_pci_address(struct device_node *dev, int bar_no, u64 *size,
> +const __be32 *of_get_pci_address(struct device_node *dev, int bar_no, u64 *size,
>  			unsigned int *flags)
>  {
>  	const __be32 *prop;
> @@ -207,7 +208,7 @@ EXPORT_SYMBOL(of_get_pci_address);
>  int of_pci_address_to_resource(struct device_node *dev, int bar,
>  			       struct resource *r)
>  {
> -	const u32	*addrp;
> +	const __be32	*addrp;
>  	u64		size;
>  	unsigned int	flags;
>  
> @@ -237,12 +238,13 @@ static void of_bus_isa_count_cells(struct device_node *child,
>  		*sizec = 1;
>  }
>  
> -static u64 of_bus_isa_map(u32 *addr, const u32 *range, int na, int ns, int pna)
> +static u64 of_bus_isa_map(u32 *addr, const __be32 *range, int na, int ns,
> +		int pna)
>  {
>  	u64 cp, s, da;
>  
>  	/* Check address type match */
> -	if ((addr[0] ^ range[0]) & 0x00000001)
> +	if ((addr[0] ^ range[0]) & cpu_to_be32(1))
>  		return OF_BAD_ADDR;
>  
>  	/* Read address values, skipping high cell */
> @@ -264,10 +266,10 @@ static int of_bus_isa_translate(u32 *addr, u64 offset, int na)
>  	return of_bus_default_translate(addr + 1, offset, na - 1);
>  }
>  
> -static unsigned int of_bus_isa_get_flags(const u32 *addr)
> +static unsigned int of_bus_isa_get_flags(const __be32 *addr)
>  {
>  	unsigned int flags = 0;
> -	u32 w = addr[0];
> +	u32 w = be32_to_cpup(addr);
>  
>  	if (w & 1)
>  		flags |= IORESOURCE_IO;
> @@ -330,7 +332,7 @@ static int of_translate_one(struct device_node *parent, struct of_bus *bus,
>  			    struct of_bus *pbus, u32 *addr,
>  			    int na, int ns, int pna, const char *rprop)
>  {
> -	const u32 *ranges;
> +	const __be32 *ranges;
>  	unsigned int rlen;
>  	int rone;
>  	u64 offset = OF_BAD_ADDR;
> @@ -398,7 +400,7 @@ static int of_translate_one(struct device_node *parent, struct of_bus *bus,
>   * that can be mapped to a cpu physical address). This is not really specified
>   * that way, but this is traditionally the way IBM at least do things
>   */
> -u64 __of_translate_address(struct device_node *dev, const u32 *in_addr,
> +u64 __of_translate_address(struct device_node *dev, const __be32 *in_addr,
>  			   const char *rprop)
>  {
>  	struct device_node *parent = NULL;
> @@ -475,22 +477,22 @@ u64 __of_translate_address(struct device_node *dev, const u32 *in_addr,
>  	return result;
>  }
>  
> -u64 of_translate_address(struct device_node *dev, const u32 *in_addr)
> +u64 of_translate_address(struct device_node *dev, const __be32 *in_addr)
>  {
>  	return __of_translate_address(dev, in_addr, "ranges");
>  }
>  EXPORT_SYMBOL(of_translate_address);
>  
> -u64 of_translate_dma_address(struct device_node *dev, const u32 *in_addr)
> +u64 of_translate_dma_address(struct device_node *dev, const __be32 *in_addr)
>  {
>  	return __of_translate_address(dev, in_addr, "dma-ranges");
>  }
>  EXPORT_SYMBOL(of_translate_dma_address);
>  
> -const u32 *of_get_address(struct device_node *dev, int index, u64 *size,
> +const __be32 *of_get_address(struct device_node *dev, int index, u64 *size,
>  		    unsigned int *flags)
>  {
> -	const u32 *prop;
> +	const __be32 *prop;
>  	unsigned int psize;
>  	struct device_node *parent;
>  	struct of_bus *bus;
> @@ -525,8 +527,8 @@ const u32 *of_get_address(struct device_node *dev, int index, u64 *size,
>  }
>  EXPORT_SYMBOL(of_get_address);
>  
> -static int __of_address_to_resource(struct device_node *dev, const u32 *addrp,
> -				    u64 size, unsigned int flags,
> +static int __of_address_to_resource(struct device_node *dev,
> +		const __be32 *addrp, u64 size, unsigned int flags,
>  				    struct resource *r)
>  {
>  	u64 taddr;
> @@ -564,7 +566,7 @@ static int __of_address_to_resource(struct device_node *dev, const u32 *addrp,
>  int of_address_to_resource(struct device_node *dev, int index,
>  			   struct resource *r)
>  {
> -	const u32	*addrp;
> +	const __be32	*addrp;
>  	u64		size;
>  	unsigned int	flags;
>  
> diff --git a/include/linux/of_address.h b/include/linux/of_address.h
> index 8aea06f..2feda6e 100644
> --- a/include/linux/of_address.h
> +++ b/include/linux/of_address.h
> @@ -3,7 +3,7 @@
>  #include <linux/ioport.h>
>  #include <linux/of.h>
>  
> -extern u64 of_translate_address(struct device_node *np, const u32 *addr);
> +extern u64 of_translate_address(struct device_node *np, const __be32 *addr);
>  extern int of_address_to_resource(struct device_node *dev, int index,
>  				  struct resource *r);
>  extern void __iomem *of_iomap(struct device_node *device, int index);
> @@ -21,7 +21,7 @@ static inline unsigned long pci_address_to_pio(phys_addr_t addr) { return -1; }
>  #endif
>  
>  #ifdef CONFIG_PCI
> -extern const u32 *of_get_pci_address(struct device_node *dev, int bar_no,
> +extern const __be32 *of_get_pci_address(struct device_node *dev, int bar_no,
>  			       u64 *size, unsigned int *flags);
>  extern int of_pci_address_to_resource(struct device_node *dev, int bar,
>  				      struct resource *r);
> @@ -32,7 +32,7 @@ static inline int of_pci_address_to_resource(struct device_node *dev, int bar,
>  	return -ENOSYS;
>  }
>  
> -static inline const u32 *of_get_pci_address(struct device_node *dev,
> +static inline const __be32 *of_get_pci_address(struct device_node *dev,
>  		int bar_no, u64 *size, unsigned int *flags)
>  {
>  	return NULL;

^ permalink raw reply

* [PATCH] mpc52xx: gpt: include fs.h
From: Wolfram Sang @ 2010-12-01 23:00 UTC (permalink / raw)
  To: linuxppc-dev

Fix build errors like these from a randconfig:

src/arch/powerpc/platforms/52xx/mpc52xx_gpt.c:549: error: dereferencing pointer to incomplete type: 1 errors in 1 logs
src/arch/powerpc/platforms/52xx/mpc52xx_gpt.c:636: error: implicit declaration of function 'nonseekable_open': 1 errors in 1 logs
src/arch/powerpc/platforms/52xx/mpc52xx_gpt.c:657: error: variable 'mpc52xx_wdt_fops' has initializer but incomplete type: 1 errors in 1 logs
src/arch/powerpc/platforms/52xx/mpc52xx_gpt.c:658: error: excess elements in struct initializer: 1 errors in 1 logs
src/arch/powerpc/platforms/52xx/mpc52xx_gpt.c:658: error: unknown field 'owner' specified in initializer: 1 errors in 1 logs
...

Reported-by: Geert Uytterhoeven <geert@linux-m68k.org>
Signed-off-by: Wolfram Sang <w.sang@pengutronix.de>
Cc: Grant Likely <grant.likely@secretlab.ca>
---
 arch/powerpc/platforms/52xx/mpc52xx_gpt.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/arch/powerpc/platforms/52xx/mpc52xx_gpt.c b/arch/powerpc/platforms/52xx/mpc52xx_gpt.c
index fea833e..e0d703c 100644
--- a/arch/powerpc/platforms/52xx/mpc52xx_gpt.c
+++ b/arch/powerpc/platforms/52xx/mpc52xx_gpt.c
@@ -63,6 +63,7 @@
 #include <linux/of_gpio.h>
 #include <linux/kernel.h>
 #include <linux/slab.h>
+#include <linux/fs.h>
 #include <linux/watchdog.h>
 #include <linux/miscdevice.h>
 #include <linux/uaccess.h>
-- 
1.7.2.3

^ permalink raw reply related

* [PATCH V3 3/3] powerpc: Disable VPHN polling during a suspend operation
From: Jesse Larrew @ 2010-12-01 22:31 UTC (permalink / raw)
  To: linuxppc-dev; +Cc: markn, pmac, tbreeds, lkessler, Jesse Larrew, mjwolf
In-Reply-To: <20101201223052.15119.61473.sendpatchset@manic8ball.ltc.austin.ibm.com>

From: Jesse Larrew <jlarrew@linux.vnet.ibm.com>

Tie the polling mechanism into the ibm,suspend-me rtas call to
stop/restart polling before/after a suspend, hibernate, migrate,
or checkpoint restart operation. This ensures that the system has a
chance to disable the polling if the partition is migrated to a system
that does not support VPHN (and vice versa).

Signed-off-by: Jesse Larrew <jlarrew@linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/topology.h |   10 ++++++++++
 arch/powerpc/kernel/rtas.c          |    3 +++
 2 files changed, 13 insertions(+), 0 deletions(-)

diff --git a/arch/powerpc/include/asm/topology.h b/arch/powerpc/include/asm/topology.h
index afe4aaa..aed188b 100644
--- a/arch/powerpc/include/asm/topology.h
+++ b/arch/powerpc/include/asm/topology.h
@@ -93,6 +93,8 @@ extern void __init dump_numa_cpu_topology(void);
 extern int sysfs_add_device_to_node(struct sys_device *dev, int nid);
 extern void sysfs_remove_device_from_node(struct sys_device *dev, int nid);
 
+extern int start_topology_update(void);
+extern int stop_topology_update(void);
 #else
 
 static inline void dump_numa_cpu_topology(void) {}
@@ -107,6 +109,14 @@ static inline void sysfs_remove_device_from_node(struct sys_device *dev,
 {
 }
 
+static inline int start_topology_update(void)
+{
+	return 0;
+}
+static inline int stop_topology_update(void)
+{
+	return 0;
+}
 #endif /* CONFIG_NUMA */
 
 #include <asm-generic/topology.h>
diff --git a/arch/powerpc/kernel/rtas.c b/arch/powerpc/kernel/rtas.c
index 8fe8bc6..2097f2b 100644
--- a/arch/powerpc/kernel/rtas.c
+++ b/arch/powerpc/kernel/rtas.c
@@ -41,6 +41,7 @@
 #include <asm/atomic.h>
 #include <asm/time.h>
 #include <asm/mmu.h>
+#include <asm/topology.h>
 
 struct rtas_t rtas = {
 	.lock = __ARCH_SPIN_LOCK_UNLOCKED
@@ -713,6 +714,7 @@ static int __rtas_suspend_last_cpu(struct rtas_suspend_me_data *data, int wake_w
 	int cpu;
 
 	slb_set_size(SLB_MIN_SIZE);
+	stop_topology_update();
 	printk(KERN_DEBUG "calling ibm,suspend-me on cpu %i\n", smp_processor_id());
 
 	while (rc == H_MULTI_THREADS_ACTIVE && !atomic_read(&data->done) &&
@@ -728,6 +730,7 @@ static int __rtas_suspend_last_cpu(struct rtas_suspend_me_data *data, int wake_w
 		rc = atomic_read(&data->error);
 
 	atomic_set(&data->error, rc);
+	start_topology_update();
 
 	if (wake_when_done) {
 		atomic_set(&data->done, 1);
-- 
1.7.2.3

^ permalink raw reply related

* [PATCH V3 2/3] powerpc: Poll VPA for topology changes and update NUMA maps
From: Jesse Larrew @ 2010-12-01 22:31 UTC (permalink / raw)
  To: linuxppc-dev; +Cc: markn, pmac, tbreeds, lkessler, Jesse Larrew, mjwolf
In-Reply-To: <20101201223052.15119.61473.sendpatchset@manic8ball.ltc.austin.ibm.com>

From: Jesse Larrew <jlarrew@linux.vnet.ibm.com>

This patch sets a timer during boot that will periodically poll the
associativity change counters in the VPA. When a change in 
associativity is detected, it retrieves the new associativity domain 
information via the H_HOME_NODE_ASSOCIATIVITY hcall and updates the 
NUMA node maps and sysfs entries accordingly. Note that since the 
ibm,associativity device tree property does not exist on configurations 
with both NUMA and SPLPAR enabled, no device tree updates are necessary.

Signed-off-by: Jesse Larrew <jlarrew@linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/lppaca.h |    5 +-
 arch/powerpc/mm/numa.c            |  277 +++++++++++++++++++++++++++++++++++--
 2 files changed, 271 insertions(+), 11 deletions(-)

diff --git a/arch/powerpc/include/asm/lppaca.h b/arch/powerpc/include/asm/lppaca.h
index 7f5e0fe..380d48b 100644
--- a/arch/powerpc/include/asm/lppaca.h
+++ b/arch/powerpc/include/asm/lppaca.h
@@ -62,7 +62,10 @@ struct lppaca {
 	volatile u32 dyn_pir;		// Dynamic ProcIdReg value	x20-x23
 	u32	dsei_data;           	// DSEI data                  	x24-x27
 	u64	sprg3;               	// SPRG3 value                	x28-x2F
-	u8	reserved3[80];		// Reserved			x30-x7F
+	u8	reserved3[40];		// Reserved			x30-x57
+	volatile u8 vphn_assoc_counts[8]; // Virtual processor home node
+					// associativity change counters x58-x5F
+	u8	reserved4[32];		// Reserved			x60-x7F
 
 //=============================================================================
 // CACHE_LINE_2 0x0080 - 0x00FF Contains local read-write data
diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index 74505b2..42aa7d1 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c
@@ -20,10 +20,14 @@
 #include <linux/memblock.h>
 #include <linux/of.h>
 #include <linux/pfn.h>
+#include <linux/cpuset.h>
+#include <linux/node.h>
 #include <asm/sparsemem.h>
 #include <asm/prom.h>
 #include <asm/system.h>
 #include <asm/smp.h>
+#include <asm/firmware.h>
+#include <asm/paca.h>
 
 static int numa_enabled = 1;
 
@@ -246,32 +250,41 @@ static void initialize_distance_lookup_table(int nid,
 /* Returns nid in the range [0..MAX_NUMNODES-1], or -1 if no useful numa
  * info is found.
  */
-static int of_node_to_nid_single(struct device_node *device)
+static int associativity_to_nid(const unsigned int *associativity)
 {
 	int nid = -1;
-	const unsigned int *tmp;
 
 	if (min_common_depth == -1)
 		goto out;
 
-	tmp = of_get_associativity(device);
-	if (!tmp)
-		goto out;
-
-	if (tmp[0] >= min_common_depth)
-		nid = tmp[min_common_depth];
+	if (associativity[0] >= min_common_depth)
+		nid = associativity[min_common_depth];
 
 	/* POWER4 LPAR uses 0xffff as invalid node */
 	if (nid == 0xffff || nid >= MAX_NUMNODES)
 		nid = -1;
 
-	if (nid > 0 && tmp[0] >= distance_ref_points_depth)
-		initialize_distance_lookup_table(nid, tmp);
+	if (nid > 0 && associativity[0] >= distance_ref_points_depth)
+		initialize_distance_lookup_table(nid, associativity);
 
 out:
 	return nid;
 }
 
+/* Returns the nid associated with the given device tree node,
+ * or -1 if not found.
+ */
+static int of_node_to_nid_single(struct device_node *device)
+{
+	int nid = -1;
+	const unsigned int *tmp;
+
+	tmp = of_get_associativity(device);
+	if (tmp)
+		nid = associativity_to_nid(tmp);
+	return nid;
+}
+
 /* Walk the device tree upwards, looking for an associativity id */
 int of_node_to_nid(struct device_node *device)
 {
@@ -1248,3 +1261,247 @@ int hot_add_scn_to_nid(unsigned long scn_addr)
 }
 
 #endif /* CONFIG_MEMORY_HOTPLUG */
+
+/* Vrtual Processor Home Node (VPHN) support */
+#define VPHN_NR_CHANGE_CTRS (8)
+static u8 vphn_cpu_change_counts[NR_CPUS][VPHN_NR_CHANGE_CTRS];
+static cpumask_t cpu_associativity_changes_mask;
+static int vphn_enabled;
+static void set_topology_timer(void);
+int stop_topology_update(void);
+
+/*
+ * Store the current values of the associativity change counters in the
+ * hypervisor.
+ */
+static void setup_cpu_associativity_change_counters(void)
+{
+	int cpu = 0;
+
+	for_each_possible_cpu(cpu) {
+		int i = 0;
+		u8 *counts = vphn_cpu_change_counts[cpu];
+		volatile u8 *hypervisor_counts = lppaca[cpu].vphn_assoc_counts;
+
+		for (i = 0; i < VPHN_NR_CHANGE_CTRS; i++) {
+			counts[i] = hypervisor_counts[i];
+		}
+	}
+}
+
+/*
+ * The hypervisor maintains a set of 8 associativity change counters in
+ * the VPA of each cpu that correspond to the associativity levels in the
+ * ibm,associativity-reference-points property. When an associativity
+ * level changes, the corresponding counter is incremented.
+ *
+ * Set a bit in cpu_associativity_changes_mask for each cpu whose home
+ * node associativity levels have changed.
+ *
+ * Returns the number of cpus with unhandled associativity changes.
+ */
+static int update_cpu_associativity_changes_mask(void)
+{
+	int cpu = 0, nr_cpus = 0;
+	cpumask_t *changes = &cpu_associativity_changes_mask;
+
+	cpumask_clear(changes);
+
+	for_each_possible_cpu(cpu) {
+		int i, changed = 0;
+		u8 *counts = vphn_cpu_change_counts[cpu];
+		volatile u8 *hypervisor_counts = lppaca[cpu].vphn_assoc_counts;
+
+		for (i = 0; i < VPHN_NR_CHANGE_CTRS; i++) {
+			if (hypervisor_counts[i] > counts[i]) {
+				counts[i] = hypervisor_counts[i];
+				changed = 1;
+			}
+		}
+		if (changed) {
+			cpumask_set_cpu(cpu, changes);
+			nr_cpus++;
+		}
+	}
+
+	return nr_cpus;
+}
+
+/* 6 64-bit registers unpacked into 12 32-bit associativity values */
+#define VPHN_ASSOC_BUFSIZE (6*sizeof(u64)/sizeof(u32))
+
+/*
+ * Convert the associativity domain numbers returned from the hypervisor
+ * to the sequence they would appear in the ibm,associativity property.
+ */
+static int vphn_unpack_associativity(const long *packed, unsigned int *unpacked)
+{
+	int i = 0;
+	int nr_assoc_doms = 0;
+	const u16 *field = (const u16*) packed;
+
+#define VPHN_FIELD_UNUSED	(0xffff)
+#define VPHN_FIELD_MSB		(0x8000)
+#define VPHN_FIELD_MASK		(~VPHN_FIELD_MSB)
+
+	for (i = 0; i < VPHN_ASSOC_BUFSIZE; i++) {
+		if (*field == VPHN_FIELD_UNUSED) {
+			/* All significant fields processed, and remaining
+			 * fields contain the reserved value of all 1's.
+			 * Just store them.
+			 */
+			unpacked[i] = *((u32*)field);
+			field += 2;
+		}
+		else if (*field & VPHN_FIELD_MSB) {
+			/* Data is in the lower 15 bits of this field */
+			unpacked[i] = *field & VPHN_FIELD_MASK;
+			field++;
+			nr_assoc_doms++;
+		}
+		else {
+			/* Data is in the lower 15 bits of this field
+			 * concatenated with the next 16 bit field
+			 */
+			unpacked[i] = *((u32*)field);
+			field += 2;
+			nr_assoc_doms++;
+		}
+	}
+
+	return nr_assoc_doms;
+}
+
+/*
+ * Retrieve the new associativity information for a virtual processor's
+ * home node.
+ */
+static long hcall_vphn(unsigned long cpu, unsigned int *associativity)
+{
+	long rc = 0;
+	long retbuf[PLPAR_HCALL9_BUFSIZE] = {0};
+	u64 flags = 1;
+	int hwcpu = get_hard_smp_processor_id(cpu);
+
+	rc = plpar_hcall9(H_HOME_NODE_ASSOCIATIVITY, retbuf, flags, hwcpu);
+	vphn_unpack_associativity(retbuf, associativity);
+
+	return rc;
+}
+
+static long vphn_get_associativity(unsigned long cpu,
+					unsigned int *associativity)
+{
+	long rc = 0;
+
+	rc = hcall_vphn(cpu, associativity);
+
+	switch (rc) {
+	case H_FUNCTION:
+		printk(KERN_INFO
+			"VPHN is not supported. Disabling polling...\n");
+		stop_topology_update();
+		break;
+	case H_HARDWARE:
+		printk(KERN_ERR
+			"hcall_vphn() experienced a hardware fault "
+			"preventing VPHN. Disabling polling...\n");
+		stop_topology_update();
+	}
+
+	return rc;
+}
+
+/*
+ * Update the node maps and sysfs entries for each cpu whose home node
+ * has changed.
+ */
+int arch_update_cpu_topology(void)
+{
+	int cpu = 0, nid = 0, old_nid = 0;
+	unsigned int associativity[VPHN_ASSOC_BUFSIZE] = {0};
+	struct sys_device *sysdev = NULL;
+
+	for_each_cpu_mask(cpu, cpu_associativity_changes_mask) {
+		vphn_get_associativity(cpu, associativity);
+		nid = associativity_to_nid(associativity);
+
+		if (nid < 0 || !node_online(nid))
+			nid = first_online_node;
+
+		old_nid = numa_cpu_lookup_table[cpu];
+
+		/* Disable hotplug while we update the cpu
+		 * masks and sysfs.
+		 */
+		get_online_cpus();
+		unregister_cpu_under_node(cpu, old_nid);
+		unmap_cpu_from_node(cpu);
+		map_cpu_to_node(cpu, nid);
+		register_cpu_under_node(cpu, nid);
+		put_online_cpus();
+
+		sysdev = get_cpu_sysdev(cpu);
+		if (sysdev)
+			kobject_uevent(&sysdev->kobj, KOBJ_CHANGE);
+	}
+
+	return 1;
+}
+
+static void topology_work_fn(struct work_struct *work)
+{
+	rebuild_sched_domains();
+}
+static DECLARE_WORK(topology_work, topology_work_fn);
+
+void topology_schedule_update(void)
+{
+	schedule_work(&topology_work);
+}
+
+static void topology_timer_fn(unsigned long ignored)
+{
+	if (!vphn_enabled)
+		return;
+	if (update_cpu_associativity_changes_mask() > 0)
+		topology_schedule_update();
+	set_topology_timer();
+}
+static struct timer_list topology_timer =
+	TIMER_INITIALIZER(topology_timer_fn, 0, 0);
+
+static void set_topology_timer(void)
+{
+	topology_timer.data = 0;
+	topology_timer.expires = jiffies + 60 * HZ;
+	add_timer(&topology_timer);
+}
+
+/*
+ * Start polling for VPHN associativity changes.
+ */
+int start_topology_update(void)
+{
+	int rc = 0;
+
+	if (firmware_has_feature(FW_FEATURE_VPHN)) {
+		vphn_enabled = 1;
+		setup_cpu_associativity_change_counters();
+		init_timer_deferrable(&topology_timer);
+		set_topology_timer();
+		rc = 1;
+	}
+
+	return rc;
+}
+__initcall(start_topology_update);
+
+/*
+ * Disable polling for VPHN associativity changes.
+ */
+int stop_topology_update(void)
+{
+	vphn_enabled = 0;
+	return del_timer_sync(&topology_timer);
+}
-- 
1.7.2.3

^ permalink raw reply related

* [PATCH V3 1/3] powerpc: Add VPHN firmware feature
From: Jesse Larrew @ 2010-12-01 22:31 UTC (permalink / raw)
  To: linuxppc-dev; +Cc: markn, pmac, tbreeds, lkessler, Jesse Larrew, mjwolf
In-Reply-To: <20101201223052.15119.61473.sendpatchset@manic8ball.ltc.austin.ibm.com>

From: Jesse Larrew <jlarrew@linux.vnet.ibm.com>

This simple patch adds the firmware feature for VPHN to the firmware 
features bitmask.

Signed-off-by: Jesse Larrew <jlarrew@linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/firmware.h       |    3 ++-
 arch/powerpc/include/asm/hvcall.h         |    3 ++-
 arch/powerpc/platforms/pseries/firmware.c |    1 +
 3 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/include/asm/firmware.h b/arch/powerpc/include/asm/firmware.h
index 20778a4..4ef662e 100644
--- a/arch/powerpc/include/asm/firmware.h
+++ b/arch/powerpc/include/asm/firmware.h
@@ -46,6 +46,7 @@
 #define FW_FEATURE_PS3_LV1	ASM_CONST(0x0000000000800000)
 #define FW_FEATURE_BEAT		ASM_CONST(0x0000000001000000)
 #define FW_FEATURE_CMO		ASM_CONST(0x0000000002000000)
+#define FW_FEATURE_VPHN		ASM_CONST(0x0000000004000000)
 
 #ifndef __ASSEMBLY__
 
@@ -59,7 +60,7 @@ enum {
 		FW_FEATURE_VIO | FW_FEATURE_RDMA | FW_FEATURE_LLAN |
 		FW_FEATURE_BULK_REMOVE | FW_FEATURE_XDABR |
 		FW_FEATURE_MULTITCE | FW_FEATURE_SPLPAR | FW_FEATURE_LPAR |
-		FW_FEATURE_CMO,
+		FW_FEATURE_CMO | FW_FEATURE_VPHN,
 	FW_FEATURE_PSERIES_ALWAYS = 0,
 	FW_FEATURE_ISERIES_POSSIBLE = FW_FEATURE_ISERIES | FW_FEATURE_LPAR,
 	FW_FEATURE_ISERIES_ALWAYS = FW_FEATURE_ISERIES | FW_FEATURE_LPAR,
diff --git a/arch/powerpc/include/asm/hvcall.h b/arch/powerpc/include/asm/hvcall.h
index de03ca5..6de1e5f 100644
--- a/arch/powerpc/include/asm/hvcall.h
+++ b/arch/powerpc/include/asm/hvcall.h
@@ -232,7 +232,8 @@
 #define H_GET_EM_PARMS		0x2B8
 #define H_SET_MPP		0x2D0
 #define H_GET_MPP		0x2D4
-#define MAX_HCALL_OPCODE	H_GET_MPP
+#define H_HOME_NODE_ASSOCIATIVITY 0x2EC
+#define MAX_HCALL_OPCODE	H_HOME_NODE_ASSOCIATIVITY
 
 #ifndef __ASSEMBLY__
 
diff --git a/arch/powerpc/platforms/pseries/firmware.c b/arch/powerpc/platforms/pseries/firmware.c
index 0a14d8c..0b0eff0 100644
--- a/arch/powerpc/platforms/pseries/firmware.c
+++ b/arch/powerpc/platforms/pseries/firmware.c
@@ -55,6 +55,7 @@ firmware_features_table[FIRMWARE_MAX_FEATURES] = {
 	{FW_FEATURE_XDABR,		"hcall-xdabr"},
 	{FW_FEATURE_MULTITCE,		"hcall-multi-tce"},
 	{FW_FEATURE_SPLPAR,		"hcall-splpar"},
+	{FW_FEATURE_VPHN,		"hcall-vphn"},
 };
 
 /* Build up the firmware features bitmask using the contents of
-- 
1.7.2.3

^ permalink raw reply related

* [PATCH V3 0/3][RFC] Add Support for Virtual Processor Home Node (VPHN)
From: Jesse Larrew @ 2010-12-01 22:30 UTC (permalink / raw)
  To: linuxppc-dev; +Cc: markn, pmac, tbreeds, lkessler, Jesse Larrew, mjwolf

From: Jesse Larrew <jlarrew@linux.vnet.ibm.com>

The SPLPAR option allows the platform to dispatch virtual processors on
physical processors that, due to the variable nature of work loads, are
temporarily free, thus improving the utilization of computing resources.
However, SPLPAR implies inconsistent mapping of virtual to physical
processors, thus defeating resource allocation software that attempts to
optimize performance on platforms that implement the NUMA option.

To bridge the gap between these two options, the VPHN option maintains a
substantially consistent mapping of a given virtual processor to a physical
processor or set of processors within a given associativity domain. When
allocating computing resources, the kernel can take advantage of this
statistically consistent mapping to improve processing performance.

VPHN mappings are substantially consistent but not static. For any given
dispatch cycle, a best effort is made by the hypervisor to dispatch the
virtual processor on a physical processor within a targeted associativity
domain (the virtual processor's home node). However, if processing capacity
within the home node is not available, some other physical processor is
assigned to meet the processing capacity entitlement. From time to time,
to optimize the total platform performance, it may be necessary for the
platform to change the home node of a given virtual processor.

The Virtual Processor Home Node feature addresses this by adding the
H_HOME_NODE_ASSOCIATIVITY hcall to retrieve the current associativity
domain information directly from the hypervisor for a given virtual
processor's home node. It also exposes a set of associativity change
counters in the Virtual Processor Area (VPA) of each processor to indicate
when associativity changes occur.

This patch set sets a timer during boot that will periodically poll the
associativity change counters. When a change in associativity is detected,
it retrieves the new associativity domain information via the
H_HOME_NODE_ASSOCIATIVITY hcall and updates the NUMA node maps and sysfs
entries accordingly. The polling mechanism is also tied into the
ibm,suspend-me rtas call to stop/restart polling before/after a suspend,
hibernate, migrate, or checkpoint restart operation.

This patch set applies to v2.6.37-rc4 and includes the following:

[PATCH 1/3] powerpc: Add VPHN firmware feature
[PATCH 2/3] powerpc: Poll VPA for topology changes and update NUMA maps
[PATCH 3/3] powerpc: Disable VPHN polling during a suspend operation

Changes since V2:

* Rebased on 2.6.37-rc4.
* Rearranged work declarations and timer initializations to eliminate
  unnecessary function declarations.
* Eliminated redundant cpumask operations in 
  update_cpu_associativity_changes_mask().
* Eliminated unnecessary return statments from functions with void
  return types.
* Coding-style cleanups.
* Replaced del_timer() with del_timer_sync() in stop_topology_update(),
  and added the "vphn_enabled" flag to prevent the timer function from
  reinstalling itself.

Signed-off-by: Jesse Larrew <jlarrew@linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/firmware.h       |    3 +-
 arch/powerpc/include/asm/hvcall.h         |    3 +-
 arch/powerpc/include/asm/lppaca.h         |    5 +-
 arch/powerpc/include/asm/topology.h       |   10 +
 arch/powerpc/kernel/rtas.c                |    3 +
 arch/powerpc/mm/numa.c                    |  274 +++++++++++++++++++++++++++-
 arch/powerpc/platforms/pseries/firmware.c |    1 +
 7 files changed, 286 insertions(+), 13 deletions(-)

^ permalink raw reply

* Re: [PATCH V2 2/3] powerpc: Poll VPA for topology changes and update NUMA maps
From: Benjamin Herrenschmidt @ 2010-12-01 21:54 UTC (permalink / raw)
  To: Jesse Larrew; +Cc: markn, pmac, tbreeds, lkessler, mjwolf, linuxppc-dev
In-Reply-To: <4CF6C32A.8000801@linux.vnet.ibm.com>

On Wed, 2010-12-01 at 16:50 -0500, Jesse Larrew wrote:
> 
> Hmmm... Good point. That would eliminate a lot of complexity, and if
> we wrap the code in the timer interrupt so that it only executes on
> systems with the VPHN feature, then partition migration pretty much
> takes care of itself as well. :) I'll repost this patch set with the
> tweaks that you mentioned above, then I'll post a separate patch to
> remove the cpumask and timer.

Right. First fixup that patch and we can merge that, then we can look at
the "better approach" as a separate step.

Cheers,
Ben.

^ permalink raw reply

* Re: [PATCH V2 2/3] powerpc: Poll VPA for topology changes and update NUMA maps
From: Jesse Larrew @ 2010-12-01 21:50 UTC (permalink / raw)
  To: Benjamin Herrenschmidt
  Cc: markn, pmac, tbreeds, lkessler, mjwolf, linuxppc-dev
In-Reply-To: <1291002259.32570.274.camel@pasglop>

On 11/28/2010 10:44 PM, Benjamin Herrenschmidt wrote:
> On Tue, 2010-11-09 at 16:25 -0700, Jesse Larrew wrote:
>> From: Jesse Larrew <jlarrew@linux.vnet.ibm.com>
>>
>> This patch sets a timer during boot that will periodically poll the
>> associativity change counters in the VPA. When a change in 
>> associativity is detected, it retrieves the new associativity domain 
>> information via the H_HOME_NODE_ASSOCIATIVITY hcall and updates the 
>> NUMA node maps and sysfs entries accordingly. Note that since the 
>> ibm,associativity device tree property does not exist on configurations 
>> with both NUMA and SPLPAR enabled, no device tree updates are necessary.
>>
>> Signed-off-by: Jesse Larrew <jlarrew@linux.vnet.ibm.com>
>> ---
> 
> No fundamental objection, just quick nits before I merge:

Thanks for the review!

>> +
>> +/* Vrtual Processor Home Node (VPHN) support */
>> +#define VPHN_NR_CHANGE_CTRS (8)
>> +static u8 vphn_cpu_change_counts[NR_CPUS][VPHN_NR_CHANGE_CTRS];
>> +static cpumask_t cpu_associativity_changes_mask;
>> +static void topology_work_fn(struct work_struct *work);
>> +static DECLARE_WORK(topology_work, topology_work_fn);
> 
> Remove the prototype for topology_work_fn() and puts the DECLARE_WORK
> right below the function itself.
> 

No problem.

>> +static void topology_timer_fn(unsigned long ignored);
>> +static struct timer_list topology_timer =
>> +	TIMER_INITIALIZER(topology_timer_fn, 0, 0);
> 
> Same deal.
> 

Ditto.

>> +static void set_topology_timer(void);
>> +int stop_topology_update(void);
>> +
>> +/*
>> + * Store the current values of the associativity change counters in the
>> + * hypervisor.
>> + */
>> +static void setup_cpu_associativity_change_counters(void)
>> +{
>> +	int cpu = 0;
>> +
>> +	for_each_possible_cpu(cpu) {
>> +		int i = 0;
>> +		u8 *counts = vphn_cpu_change_counts[cpu];
>> +		volatile u8 *hypervisor_counts = lppaca[cpu].vphn_assoc_counts;
>> +
>> +		for (i = 0; i < VPHN_NR_CHANGE_CTRS; i++) {
>> +			counts[i] = hypervisor_counts[i];
>> +		}
>> +	}
>> +
>> +	return;
>> +}
>> +
>> +/*
>> + * The hypervisor maintains a set of 8 associativity change counters in
>> + * the VPA of each cpu that correspond to the associativity levels in the
>> + * ibm,associativity-reference-points property. When an associativity
>> + * level changes, the corresponding counter is incremented.
>> + *
>> + * Set a bit in cpu_associativity_changes_mask for each cpu whose home
>> + * node associativity levels have changed.
>> + */
>> +static void update_cpu_associativity_changes_mask(void)
>> +{
>> +	int cpu = 0;
>> +	cpumask_t *changes = &cpu_associativity_changes_mask;
>> +
>> +	cpumask_clear(changes);
>> +
>> +	for_each_possible_cpu(cpu) {
>> +		int i;
>> +		u8 *counts = vphn_cpu_change_counts[cpu];
>> +		volatile u8 *hypervisor_counts = lppaca[cpu].vphn_assoc_counts;
>> +
>> +		for (i = 0; i < VPHN_NR_CHANGE_CTRS; i++) {
>> +			if (hypervisor_counts[i] > counts[i]) {
>> +				counts[i] = hypervisor_counts[i];
>> +
>> +				if (!(cpumask_test_cpu(cpu, changes)))
>> +					cpumask_set_cpu(cpu, changes);
>> +			}
>> +		}
> 
> This is a tad sub-optimal. I'd just set a local variable, and
> after the inside loop set the cpumask bit when that variable is set.
> 
> Also, keep another variable that accumulate all bits set and return
> its value so you don't have to re-check the mask in the caller.
> 
> cpumask operations can be expensive.
> 

You're right. That's more efficient.

>> +	}
>> +
>> +	return;
> 
> You don't need a return; at the end of a function.
> 

Ok.

>> +}
>> +
>> +/* 6 64-bit registers unpacked into 12 32-bit associativity values */
>> +#define VPHN_ASSOC_BUFSIZE (6*sizeof(u64)/sizeof(u32))
>> +
>> +/*
>> + * Convert the associativity domain numbers returned from the hypervisor
>> + * to the sequence they would appear in the ibm,associativity property.
>> + */
>> +static int vphn_unpack_associativity(const long *packed, unsigned int *unpacked)
>> +{
>> +	int i = 0;
>> +	int nr_assoc_doms = 0;
>> +	const u16 *field = (const u16*) packed;
>> +
>> +#define VPHN_FIELD_UNUSED	(0xffff)
>> +#define VPHN_FIELD_MSB		(0x8000)
>> +#define VPHN_FIELD_MASK		(~VPHN_FIELD_MSB)
>> +
>> +	for (i = 0; i < VPHN_ASSOC_BUFSIZE; i++) {
>> +		if (*field == VPHN_FIELD_UNUSED) {
>> +			/* All significant fields processed, and remaining
>> +			 * fields contain the reserved value of all 1's.
>> +			 * Just store them.
>> +			 */
>> +			unpacked[i] = *((u32*)field);
>> +			field += 2;
>> +		}
>> +		else if (*field & VPHN_FIELD_MSB) {
>> +			/* Data is in the lower 15 bits of this field */
>> +			unpacked[i] = *field & VPHN_FIELD_MASK;
>> +			field++;
>> +			nr_assoc_doms++;
>> +		}
>> +		else {
>> +			/* Data is in the lower 15 bits of this field
>> +			 * concatenated with the next 16 bit field
>> +			 */
>> +			unpacked[i] = *((u32*)field);
>> +			field += 2;
>> +			nr_assoc_doms++;
>> +		}
>> +	}
>> +
>> +	return nr_assoc_doms;
>> +}
>> +
>> +/*
>> + * Retrieve the new associativity information for a virtual processor's
>> + * home node.
>> + */
>> +static long hcall_vphn(unsigned long cpu, unsigned int *associativity)
>> +{
>> +	long rc = 0;
>> +	long retbuf[PLPAR_HCALL9_BUFSIZE] = {0};
>> +	u64 flags = 1;
>> +	int hwcpu = get_hard_smp_processor_id(cpu);
>> +
>> +	rc = plpar_hcall9(H_HOME_NODE_ASSOCIATIVITY, retbuf, flags, hwcpu);
>> +	vphn_unpack_associativity(retbuf, associativity);
>> +
>> +	return rc;
>> +}
>> +
>> +static long
>> +vphn_get_associativity(unsigned long cpu, unsigned int *associativity)
>> +{
> 
> Nowadays, we prefer keeping the "static long" and the function name on
> the same line. If you really want to avoid >80 col (I don't care myself
> much as long as you stick below 100) then move the second argument down
> one line.
> 

Ok. I wasn't sure which way was preferred. :P

>> +	long rc = 0;
>> +
>> +	rc = hcall_vphn(cpu, associativity);
>> +
>> +	switch (rc) {
>> +	case H_FUNCTION:
>> +		printk(KERN_INFO
>> +			"VPHN is not supported. Disabling polling...\n");
>> +		stop_topology_update();
>> +		break;
>> +	case H_HARDWARE:
>> +		printk(KERN_ERR
>> +			"hcall_vphn() experienced a hardware fault "
>> +			"preventing VPHN. Disabling polling...\n");
>> +		stop_topology_update();
>> +	}
>> +
>> +	return rc;
>> +}
>> +
>> +/*
>> + * Update the node maps and sysfs entries for each cpu whose home node
>> + * has changed.
>> + */
>> +int arch_update_cpu_topology(void)
>> +{
>> +	int cpu = 0, nid = 0, old_nid = 0;
>> +	unsigned int associativity[VPHN_ASSOC_BUFSIZE] = {0};
>> +	struct sys_device *sysdev = NULL;
>> +
>> +	for_each_cpu_mask(cpu, cpu_associativity_changes_mask) {
>> +		vphn_get_associativity(cpu, associativity);
>> +		nid = associativity_to_nid(associativity);
>> +
>> +		if (nid < 0 || !node_online(nid))
>> +			nid = first_online_node;
>> +
>> +		old_nid = numa_cpu_lookup_table[cpu];
>> +
>> +		/* Disable hotplug while we update the cpu
>> +		 * masks and sysfs.
>> +		 */
>> +		get_online_cpus();
>> +		unregister_cpu_under_node(cpu, old_nid);
>> +		unmap_cpu_from_node(cpu);
>> +		map_cpu_to_node(cpu, nid);
>> +		register_cpu_under_node(cpu, nid);
>> +		put_online_cpus();
>> +
>> +		sysdev = get_cpu_sysdev(cpu);
>> +		if (sysdev)
>> +			kobject_uevent(&sysdev->kobj, KOBJ_CHANGE);
>> +	}
>> +
>> +	return 1;
>> +}
> 
> That looks terribly expensive. Might be worth considering adding a way
> to sysfs to "mv" an object around in the future.
> 

That's a good idea. I'll look into it once we get VPHN finalized.

>> +static void topology_work_fn(struct work_struct *work)
>> +{
>> +	rebuild_sched_domains();
>> +}
>> +
>> +void topology_schedule_update(void)
>> +{
>> +	schedule_work(&topology_work);
>> +}
>> +
>> +static void topology_timer_fn(unsigned long ignored)
>> +{
>> +	update_cpu_associativity_changes_mask();
>> +	if (!cpumask_empty(&cpu_associativity_changes_mask))
>> +		topology_schedule_update();
>> +	set_topology_timer();
>> +}
> 
> I wonder if we really need that cpumask and timer overall. We might be
> better off just having a per-cpu copy of the counters and check them in
> the timer interrupt, it's low enough overhead don't you think ?
> 

Hmmm... Good point. That would eliminate a lot of complexity, and if we wrap the code in the timer interrupt so that it only executes on systems with the VPHN feature, then partition migration pretty much takes care of itself as well. :) I'll repost this patch set with the tweaks that you mentioned above, then I'll post a separate patch to remove the cpumask and timer.

>> +static void set_topology_timer(void)
>> +{
>> +	topology_timer.data = 0;
>> +	topology_timer.expires = jiffies + 60 * HZ;
>> +	add_timer(&topology_timer);
>> +}
>> +
>> +/*
>> + * Start polling for VPHN associativity changes.
>> + */
>> +int start_topology_update(void)
>> +{
>> +	int rc = 0;
>> +
>> +	if (firmware_has_feature(FW_FEATURE_VPHN)) {
>> +		setup_cpu_associativity_change_counters();
>> +		init_timer_deferrable(&topology_timer);
>> +		set_topology_timer();
>> +		rc = 1;
>> +	}
>> +
>> +	return rc;
>> +}
>> +__initcall(start_topology_update);
>> +
>> +/*
>> + * Disable polling for VPHN associativity changes.
>> + */
>> +int stop_topology_update(void)
>> +{
>> +	return del_timer(&topology_timer);
>> +}
> 
> del_timer_sync() ?
> 

Ah, good catch! I think the proper way to do this is to use del_timer_sync() and add a "shutting down" flag to the timer function to ensure that it doesn't reschedule itself.

> Cheers,
> Ben.
> 


-- 

Jesse Larrew
Software Engineer, Linux on Power Kernel Team
IBM Linux Technology Center
Phone: (512) 973-2052 (T/L: 363-2052)
jlarrew@linux.vnet.ibm.com

^ permalink raw reply

* Re: [MPC52xx]Latency issue with DMA on FEC
From: Scott Wood @ 2010-12-01 21:16 UTC (permalink / raw)
  To: David Laight
  Cc: linuxppc-dev, Eric Dumazet, Jean-Michel Hautbois, Steven Rostedt,
	linux-rt-users
In-Reply-To: <AE90C24D6B3A694183C094C60CF0A2F6D8ABDB@saturn3.aculab.com>

On Wed, 1 Dec 2010 15:09:54 +0000
David Laight <David.Laight@ACULAB.COM> wrote:

> The in_le32() not only contains the unwanted 'sync', but also
> a 'twi' (trap immediate - NFI exactly what this does) and 'isync'.

It turns a data dependency into a flow dependency.  It's basically equivalent to:

lwz	rX, ...
cmpw	rX, rX
bne	1f
1: isync

> The 'isync' is particularly horrid and unnecessary (aborts
> the instruction queue and refeches the opcode bytes)

The isync makes sure that the twi has completed before proceeding.

Note that the guarded, cache-inhibited load itself can be pretty
painful -- the core can't restart it, so it must complete before you
can take an interrupt.

> The very slow in_le32() might be there to give semi-synchronous
> traps on address fault - but unless the hardware is being probed
> that really isn't necessary.

There are times when you really want to be sure that the I/O is
finished before proceeding with something that isn't a load/store and
thus can't be serialized with normal barriers.

E.g. you're about to execute instructions in a physical address window
that you just set up (or even just create a non-guarded mapping to it
-- could get speculative accesses any time), or you just masked an
interrupt at the PIC (with a readback to flush) and are about to enable
MSR[EE].

Most of the time, though, it's overkill.  It should probably be an
alternate accessor form, or maybe a wait_for_io() wrapper -- if it can
be shown to make a real performance difference.

-Scott

^ permalink raw reply

* Re: _extending_ platform support options?
From: Benjamin Herrenschmidt @ 2010-12-01 20:57 UTC (permalink / raw)
  To: Josh Boyer; +Cc: linuxppc-dev, Joachim Foerster
In-Reply-To: <20101201134719.GA1053@zod.rchland.ibm.com>

On Wed, 2010-12-01 at 08:47 -0500, Josh Boyer wrote:
> >Though, to me, it does not seem really OK to assign ppc_md members
> >that way. The original struct machdep for "virtex" (which is defined
> >in virtex.c with define_machine()) is not adjusted either. Ok, we
> >could modify that one, too.
> >Especially I'm not sure if it is OK to use machine_core_initcall()
> for such modifications.
> >
> >So my question is: Is there any recommended way for doing such
> >"extensions"? Or is it OK to just duplicate virtex.c (which does not
> >seem really OK, too)?
> 
> Duplicate it as you have done, naming the file something unique.  We
> try
> to prevent unnecessary duplication of code, but sometimes it's cleaner
> to just have a separate board file instead. 

Right. Best way is to turn the common code in virtex.c into "library"
code that you can hookup from your platform's ppc_md, so you avoid
duplication that way for most things.

You an do that by just linking in virtex.c and changing the stuff you
want to be non-static, or better if that becomes a habit, separate
virtex-lib.c (for example) from virtex-simple.c (generic platform for
example). You don't have to follow my proposed names :-)

Cheers,
Ben.

^ permalink raw reply


This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox