[PATCH v5 0/7] xen/arm: move to mach-virt and support SMP

[PATCH v5 0/7] xen/arm: move to mach-virt and support SMP

[Stefano Stabellini]

Hi all,
this patch series, based on 3.9-rc3, moves xenvm to mach-virt,
introduces SMP support in Xen on ARM and implements machine reboot and
power off via Xen sched_op hypercalls.

Each patch comes with a detailed changelog.


The merge window is approaching, this patch series only contains Xen
specific changes for ARM, if anybody has any objections speak now
please.

I would like an ack from an ARM maintainer on patch #4 and #5 since they
touch generic ARM Makefiles.


Changes in v5:
- allocate xen_vcpu_info dynamically, aligning it to the size of the
struct;
- use VCPUOP_register_vcpu_info on cpu0 too;
- set pm_power_off and arm_pm_restart from the Xen specific
intialization code.



Stefano Stabellini (7):
      xen/arm: actually pass a non-NULL percpu pointer to request_percpu_irq
      xen/arm: implement HYPERVISOR_vcpu_op
      xen/arm: SMP support
      xen: move the xenvm machine to mach-virt
      xen/arm: XEN selects ARM_PSCI
      xenvm: add a simple PSCI node and a second cpu
      xen/arm: use sched_op hypercalls for machine reboot and power off

 arch/arm/Kconfig                     |    1 +
 arch/arm/boot/dts/Makefile           |    4 +-
 arch/arm/boot/dts/xenvm-4.2.dts      |   13 ++++++
 arch/arm/include/asm/xen/hypercall.h |    1 +
 arch/arm/mach-vexpress/v2m.c         |    1 -
 arch/arm/mach-virt/virt.c            |    1 +
 arch/arm/xen/enlighten.c             |   75 ++++++++++++++++++++++++++++++---
 arch/arm/xen/hypercall.S             |    1 +
 8 files changed, 87 insertions(+), 10 deletions(-)


git://git.kernel.org/pub/scm/linux/kernel/git/sstabellini/xen.git 3.9-rc3-smp-5-tag

Cheers,

Stefano

[PATCH v5 1/7] xen/arm: actually pass a non-NULL percpu pointer to request_percpu_irq

[Stefano Stabellini]

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Reviewed-by: Ian Campbell <ian.camjpbell@citrix.com>
CC: stable@vger.kernel.org
---
 arch/arm/xen/enlighten.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/arch/arm/xen/enlighten.c b/arch/arm/xen/enlighten.c
index 8dc0605..99ce189 100644
--- a/arch/arm/xen/enlighten.c
+++ b/arch/arm/xen/enlighten.c
@@ -239,7 +239,7 @@ static int __init xen_init_events(void)
 	xen_init_IRQ();
 
 	if (request_percpu_irq(xen_events_irq, xen_arm_callback,
-			"events", xen_vcpu)) {
+			"events", &xen_vcpu)) {
 		pr_err("Error requesting IRQ %d\n", xen_events_irq);
 		return -EINVAL;
 	}
-- 
1.7.2.5

[PATCH v5 2/7] xen/arm: implement HYPERVISOR_vcpu_op

[Stefano Stabellini]

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Reviewed-by: Ian Campbell <ian.campbell@citrix.com>
---
 arch/arm/include/asm/xen/hypercall.h |    1 +
 arch/arm/xen/enlighten.c             |    1 +
 arch/arm/xen/hypercall.S             |    1 +
 3 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/arch/arm/include/asm/xen/hypercall.h b/arch/arm/include/asm/xen/hypercall.h
index 8a82325..799f42e 100644
--- a/arch/arm/include/asm/xen/hypercall.h
+++ b/arch/arm/include/asm/xen/hypercall.h
@@ -46,6 +46,7 @@ int HYPERVISOR_event_channel_op(int cmd, void *arg);
 unsigned long HYPERVISOR_hvm_op(int op, void *arg);
 int HYPERVISOR_memory_op(unsigned int cmd, void *arg);
 int HYPERVISOR_physdev_op(int cmd, void *arg);
+int HYPERVISOR_vcpu_op(int cmd, int vcpuid, void *extra_args);
 
 static inline void
 MULTI_update_va_mapping(struct multicall_entry *mcl, unsigned long va,
diff --git a/arch/arm/xen/enlighten.c b/arch/arm/xen/enlighten.c
index 99ce189..82d5e63 100644
--- a/arch/arm/xen/enlighten.c
+++ b/arch/arm/xen/enlighten.c
@@ -259,4 +259,5 @@ EXPORT_SYMBOL_GPL(HYPERVISOR_sched_op);
 EXPORT_SYMBOL_GPL(HYPERVISOR_hvm_op);
 EXPORT_SYMBOL_GPL(HYPERVISOR_memory_op);
 EXPORT_SYMBOL_GPL(HYPERVISOR_physdev_op);
+EXPORT_SYMBOL_GPL(HYPERVISOR_vcpu_op);
 EXPORT_SYMBOL_GPL(privcmd_call);
diff --git a/arch/arm/xen/hypercall.S b/arch/arm/xen/hypercall.S
index 71f7239..199cb2d 100644
--- a/arch/arm/xen/hypercall.S
+++ b/arch/arm/xen/hypercall.S
@@ -87,6 +87,7 @@ HYPERCALL2(event_channel_op);
 HYPERCALL2(hvm_op);
 HYPERCALL2(memory_op);
 HYPERCALL2(physdev_op);
+HYPERCALL3(vcpu_op);
 
 ENTRY(privcmd_call)
 	stmdb sp!, {r4}
-- 
1.7.2.5

[PATCH v5 3/7] xen/arm: SMP support

[Stefano Stabellini]

Map vcpu_info using VCPUOP_register_vcpu_info on all the online vcpus,
make sure the allocated struct doesn't cross a page boundary.

Call enable_percpu_irq on every cpu.

Changes in v5:
- allocate xen_vcpu_info dynamically, aligning it to the size of the
struct;
- use VCPUOP_register_vcpu_info on cpu0 too.

Changed in v2:
- move the percpu variable argument fix to a separate patch;
- remove unused variable.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
---
 arch/arm/xen/enlighten.c |   48 ++++++++++++++++++++++++++++++++++++++++-----
 1 files changed, 42 insertions(+), 6 deletions(-)

diff --git a/arch/arm/xen/enlighten.c b/arch/arm/xen/enlighten.c
index 82d5e63..6c87d11 100644
--- a/arch/arm/xen/enlighten.c
+++ b/arch/arm/xen/enlighten.c
@@ -2,6 +2,7 @@
 #include <xen/events.h>
 #include <xen/grant_table.h>
 #include <xen/hvm.h>
+#include <xen/interface/vcpu.h>
 #include <xen/interface/xen.h>
 #include <xen/interface/memory.h>
 #include <xen/interface/hvm/params.h>
@@ -32,6 +33,7 @@ struct shared_info xen_dummy_shared_info;
 struct shared_info *HYPERVISOR_shared_info = (void *)&xen_dummy_shared_info;
 
 DEFINE_PER_CPU(struct vcpu_info *, xen_vcpu);
+static struct vcpu_info __percpu *xen_vcpu_info;
 
 /* These are unused until we support booting "pre-ballooned" */
 unsigned long xen_released_pages;
@@ -148,6 +150,29 @@ int xen_unmap_domain_mfn_range(struct vm_area_struct *vma,
 }
 EXPORT_SYMBOL_GPL(xen_unmap_domain_mfn_range);
 
+static int __init xen_secondary_init(unsigned int cpu)
+{
+	struct vcpu_register_vcpu_info info;
+	struct vcpu_info *vcpup;
+	int err;
+
+	pr_info("Xen: initializing cpu%d\n", cpu);
+	vcpup = per_cpu_ptr(xen_vcpu_info, cpu);
+
+	info.mfn = __pa(vcpup) >> PAGE_SHIFT;
+	info.offset = offset_in_page(vcpup);
+
+	err = HYPERVISOR_vcpu_op(VCPUOP_register_vcpu_info, cpu, &info);
+	if (err) {
+		pr_debug("register_vcpu_info failed: err=%d\n", err);
+	} else {
+		/* This cpu is using the registered vcpu info, even if
+		   later ones fail to. */
+		per_cpu(xen_vcpu, cpu) = vcpup;
+	}
+	return 0;
+}
+
 /*
  * see Documentation/devicetree/bindings/arm/xen.txt for the
  * documentation of the Xen Device Tree format.
@@ -163,6 +188,7 @@ static int __init xen_guest_init(void)
 	const char *version = NULL;
 	const char *xen_prefix = "xen,xen-";
 	struct resource res;
+	int i;
 
 	node = of_find_compatible_node(NULL, NULL, "xen,xen");
 	if (!node) {
@@ -209,13 +235,18 @@ static int __init xen_guest_init(void)
 
 	/* xen_vcpu is a pointer to the vcpu_info struct in the shared_info
 	 * page, we use it in the event channel upcall and in some pvclock
-	 * related functions. We don't need the vcpu_info placement
-	 * optimizations because we don't use any pv_mmu or pv_irq op on
-	 * HVM.
+	 * related functions. 
 	 * The shared info contains exactly 1 CPU (the boot CPU). The guest
 	 * is required to use VCPUOP_register_vcpu_info to place vcpu info
-	 * for secondary CPUs as they are brought up. */
-	per_cpu(xen_vcpu, 0) = &HYPERVISOR_shared_info->vcpu_info[0];
+	 * for secondary CPUs as they are brought up.
+	 * For uniformity we use VCPUOP_register_vcpu_info even on cpu0.
+	 */
+	xen_vcpu_info = __alloc_percpu(sizeof(struct vcpu_info),
+			                       sizeof(struct vcpu_info));
+	if (xen_vcpu_info == NULL)
+		return -ENOMEM;
+	for_each_online_cpu(i)
+		xen_secondary_init(i);
 
 	gnttab_init();
 	if (!xen_initial_domain())
@@ -231,6 +262,11 @@ static irqreturn_t xen_arm_callback(int irq, void *arg)
 	return IRQ_HANDLED;
 }
 
+static __init void xen_percpu_enable_events(void *unused)
+{
+	enable_percpu_irq(xen_events_irq, 0);
+}
+
 static int __init xen_init_events(void)
 {
 	if (!xen_domain() || xen_events_irq < 0)
@@ -244,7 +280,7 @@ static int __init xen_init_events(void)
 		return -EINVAL;
 	}
 
-	enable_percpu_irq(xen_events_irq, 0);
+	on_each_cpu(xen_percpu_enable_events, NULL, 0);
 
 	return 0;
 }
-- 
1.7.2.5

[PATCH v5 4/7] xen: move the xenvm machine to mach-virt

[Stefano Stabellini]

xenvm is based on mach-vexpress, move it to mach-virt.

Changes in v4:
- update the dts Makefile too.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
CC: Marc Zyngier <marc.zyngier@arm.com>
CC: will.deacon@arm.com
CC: arnd@arndb.de
CC: rob.herring@calxeda.com
---
 arch/arm/boot/dts/Makefile   |    4 ++--
 arch/arm/mach-vexpress/v2m.c |    1 -
 arch/arm/mach-virt/virt.c    |    1 +
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/arm/boot/dts/Makefile b/arch/arm/boot/dts/Makefile
index 9c62558..b6289b7 100644
--- a/arch/arm/boot/dts/Makefile
+++ b/arch/arm/boot/dts/Makefile
@@ -168,8 +168,8 @@ dtb-$(CONFIG_ARCH_TEGRA) += tegra20-harmony.dtb \
 dtb-$(CONFIG_ARCH_VEXPRESS) += vexpress-v2p-ca5s.dtb \
 	vexpress-v2p-ca9.dtb \
 	vexpress-v2p-ca15-tc1.dtb \
-	vexpress-v2p-ca15_a7.dtb \
-	xenvm-4.2.dtb
+	vexpress-v2p-ca15_a7.dtb
+dtb-$(CONFIG_ARCH_VIRT) += xenvm-4.2.dtb
 dtb-$(CONFIG_ARCH_VT8500) += vt8500-bv07.dtb \
 	wm8505-ref.dtb \
 	wm8650-mid.dtb \
diff --git a/arch/arm/mach-vexpress/v2m.c b/arch/arm/mach-vexpress/v2m.c
index 915683c..c43ec78 100644
--- a/arch/arm/mach-vexpress/v2m.c
+++ b/arch/arm/mach-vexpress/v2m.c
@@ -469,7 +469,6 @@ static void __init v2m_dt_init(void)
 
 static const char * const v2m_dt_match[] __initconst = {
 	"arm,vexpress",
-	"xen,xenvm",
 	NULL,
 };
 
diff --git a/arch/arm/mach-virt/virt.c b/arch/arm/mach-virt/virt.c
index 31666f6..528c05e 100644
--- a/arch/arm/mach-virt/virt.c
+++ b/arch/arm/mach-virt/virt.c
@@ -40,6 +40,7 @@ static void __init virt_timer_init(void)
 
 static const char *virt_dt_match[] = {
 	"linux,dummy-virt",
+	"xen,xenvm",
 	NULL
 };
 
-- 
1.7.2.5

[PATCH v5 5/7] xen/arm: XEN selects ARM_PSCI

[Stefano Stabellini]

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
---
 arch/arm/Kconfig |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 2c3bdce..344e299 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -1892,6 +1892,7 @@ config XEN
 	depends on ARM && AEABI && OF
 	depends on CPU_V7 && !CPU_V6
 	depends on !GENERIC_ATOMIC64
+	select ARM_PSCI
 	help
 	  Say Y if you want to run Linux in a Virtual Machine on Xen on ARM.
 
-- 
1.7.2.5

[PATCH v5 6/7] xenvm: add a simple PSCI node and a second cpu

[Stefano Stabellini]

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
CC: rob.herring@calxeda.com
CC: will.deacon@arm.com
CC: arnd@arndb.de
---
 arch/arm/boot/dts/xenvm-4.2.dts |   13 +++++++++++++
 1 files changed, 13 insertions(+), 0 deletions(-)

diff --git a/arch/arm/boot/dts/xenvm-4.2.dts b/arch/arm/boot/dts/xenvm-4.2.dts
index ec3f952..3369151 100644
--- a/arch/arm/boot/dts/xenvm-4.2.dts
+++ b/arch/arm/boot/dts/xenvm-4.2.dts
@@ -29,6 +29,19 @@
 			compatible = "arm,cortex-a15";
 			reg = <0>;
 		};
+
+		cpu@1 {
+			device_type = "cpu";
+			compatible = "arm,cortex-a15";
+			reg = <1>;
+		};
+	};
+
+	psci {
+		compatible      = "arm,psci";
+		method          = "hvc";
+		cpu_off         = <1>;
+		cpu_on          = <2>;
 	};
 
 	memory@80000000 {
-- 
1.7.2.5

[PATCH v5 7/7] xen/arm: use sched_op hypercalls for machine reboot and power off

[Stefano Stabellini]

Changes in v5:
- set pm_power_off and arm_pm_restart from the Xen specific
intialization code.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
---
 arch/arm/xen/enlighten.c |   24 ++++++++++++++++++++++++
 1 files changed, 24 insertions(+), 0 deletions(-)

diff --git a/arch/arm/xen/enlighten.c b/arch/arm/xen/enlighten.c
index 6c87d11..9a0a917 100644
--- a/arch/arm/xen/enlighten.c
+++ b/arch/arm/xen/enlighten.c
@@ -10,9 +10,12 @@
 #include <xen/platform_pci.h>
 #include <xen/xenbus.h>
 #include <xen/page.h>
+#include <xen/xen.h>
+#include <xen/interface/sched.h>
 #include <xen/xen-ops.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
+#include <asm/system_misc.h>
 #include <linux/interrupt.h>
 #include <linux/irqreturn.h>
 #include <linux/module.h>
@@ -173,6 +176,24 @@ static int __init xen_secondary_init(unsigned int cpu)
 	return 0;
 }
 
+static void xen_restart(char str, const char *cmd)
+{
+	struct sched_shutdown r = { .reason = SHUTDOWN_reboot };
+	int rc;
+	rc = HYPERVISOR_sched_op(SCHEDOP_shutdown, &r);
+	if (rc)
+		BUG();
+}
+
+static void xen_power_off(void)
+{
+	struct sched_shutdown r = { .reason = SHUTDOWN_poweroff };
+	int rc;
+	rc = HYPERVISOR_sched_op(SCHEDOP_shutdown, &r);
+	if (rc)
+		BUG();
+}
+
 /*
  * see Documentation/devicetree/bindings/arm/xen.txt for the
  * documentation of the Xen Device Tree format.
@@ -252,6 +273,9 @@ static int __init xen_guest_init(void)
 	if (!xen_initial_domain())
 		xenbus_probe(NULL);
 
+	pm_power_off = xen_power_off;
+	arm_pm_restart = xen_restart;
+
 	return 0;
 }
 core_initcall(xen_guest_init);
-- 
1.7.2.5

Re: [PATCH v5 0/7] xen/arm: move to mach-virt and support SMP

[Stefano Stabellini]

Arnd, Olof,
during the last few merge windows Konrad has always sent the Xen ARM
patches to Linus via his tree, but this time (3.10 merge window), after
consulting with Konrad I am thinking of sending to Linus the pull
request myself.

Arnd, are you OK with this patch series?

It's all very Xen specific and it is already in linux-next.
The only non-Xen files that are touched are arch/arm/Kconfig, to select
ARM_PSCI on XEN, and arch/arm/boot/dts/Makefile, to build the xenvm DTB
if CONFIG_ARCH_VIRT.

Cheers,

Stefano


On Thu, 25 Apr 2013, Stefano Stabellini wrote:
> Hi all,
> this patch series, based on 3.9-rc3, moves xenvm to mach-virt,
> introduces SMP support in Xen on ARM and implements machine reboot and
> power off via Xen sched_op hypercalls.
> 
> Each patch comes with a detailed changelog.
> 
> 
> The merge window is approaching, this patch series only contains Xen
> specific changes for ARM, if anybody has any objections speak now
> please.
> 
> I would like an ack from an ARM maintainer on patch #4 and #5 since they
> touch generic ARM Makefiles.
> 
> 
> Changes in v5:
> - allocate xen_vcpu_info dynamically, aligning it to the size of the
> struct;
> - use VCPUOP_register_vcpu_info on cpu0 too;
> - set pm_power_off and arm_pm_restart from the Xen specific
> intialization code.
> 
> 
> 
> Stefano Stabellini (7):
>       xen/arm: actually pass a non-NULL percpu pointer to request_percpu_irq
>       xen/arm: implement HYPERVISOR_vcpu_op
>       xen/arm: SMP support
>       xen: move the xenvm machine to mach-virt
>       xen/arm: XEN selects ARM_PSCI
>       xenvm: add a simple PSCI node and a second cpu
>       xen/arm: use sched_op hypercalls for machine reboot and power off
> 
>  arch/arm/Kconfig                     |    1 +
>  arch/arm/boot/dts/Makefile           |    4 +-
>  arch/arm/boot/dts/xenvm-4.2.dts      |   13 ++++++
>  arch/arm/include/asm/xen/hypercall.h |    1 +
>  arch/arm/mach-vexpress/v2m.c         |    1 -
>  arch/arm/mach-virt/virt.c            |    1 +
>  arch/arm/xen/enlighten.c             |   75 ++++++++++++++++++++++++++++++---
>  arch/arm/xen/hypercall.S             |    1 +
>  8 files changed, 87 insertions(+), 10 deletions(-)
> 
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/sstabellini/xen.git 3.9-rc3-smp-5-tag
> 
> Cheers,
> 
> Stefano
> 

Re: [PATCH v5 0/7] xen/arm: move to mach-virt and support SMP

[Konrad Rzeszutek Wilk]

On Fri, Apr 26, 2013 at 11:41:32AM +0100, Stefano Stabellini wrote:
> Arnd, Olof,
> during the last few merge windows Konrad has always sent the Xen ARM
> patches to Linus via his tree, but this time (3.10 merge window), after
> consulting with Konrad I am thinking of sending to Linus the pull
> request myself.

<nods> They look good to me. If there are cross-platform patches
(say in drivers/xen) they would still need my Ack (or Jeremy's).

> 
> Arnd, are you OK with this patch series?
> 
> It's all very Xen specific and it is already in linux-next.
> The only non-Xen files that are touched are arch/arm/Kconfig, to select
> ARM_PSCI on XEN, and arch/arm/boot/dts/Makefile, to build the xenvm DTB
> if CONFIG_ARCH_VIRT.

I don't have much knowledge in the 'dts' Makefile but the rest looks
OK to me.
> 
> Cheers,
> 
> Stefano
> 
> 
> On Thu, 25 Apr 2013, Stefano Stabellini wrote:
> > Hi all,
> > this patch series, based on 3.9-rc3, moves xenvm to mach-virt,
> > introduces SMP support in Xen on ARM and implements machine reboot and
> > power off via Xen sched_op hypercalls.
> > 
> > Each patch comes with a detailed changelog.
> > 
> > 
> > The merge window is approaching, this patch series only contains Xen
> > specific changes for ARM, if anybody has any objections speak now
> > please.
> > 
> > I would like an ack from an ARM maintainer on patch #4 and #5 since they
> > touch generic ARM Makefiles.
> > 
> > 
> > Changes in v5:
> > - allocate xen_vcpu_info dynamically, aligning it to the size of the
> > struct;
> > - use VCPUOP_register_vcpu_info on cpu0 too;
> > - set pm_power_off and arm_pm_restart from the Xen specific
> > intialization code.
> > 
> > 
> > 
> > Stefano Stabellini (7):
> >       xen/arm: actually pass a non-NULL percpu pointer to request_percpu_irq
> >       xen/arm: implement HYPERVISOR_vcpu_op
> >       xen/arm: SMP support
> >       xen: move the xenvm machine to mach-virt
> >       xen/arm: XEN selects ARM_PSCI
> >       xenvm: add a simple PSCI node and a second cpu
> >       xen/arm: use sched_op hypercalls for machine reboot and power off
> > 
> >  arch/arm/Kconfig                     |    1 +
> >  arch/arm/boot/dts/Makefile           |    4 +-
> >  arch/arm/boot/dts/xenvm-4.2.dts      |   13 ++++++
> >  arch/arm/include/asm/xen/hypercall.h |    1 +
> >  arch/arm/mach-vexpress/v2m.c         |    1 -
> >  arch/arm/mach-virt/virt.c            |    1 +
> >  arch/arm/xen/enlighten.c             |   75 ++++++++++++++++++++++++++++++---
> >  arch/arm/xen/hypercall.S             |    1 +
> >  8 files changed, 87 insertions(+), 10 deletions(-)
> > 
> > 
> > git://git.kernel.org/pub/scm/linux/kernel/git/sstabellini/xen.git 3.9-rc3-smp-5-tag
> > 
> > Cheers,
> > 
> > Stefano
> >