[U-Boot] [RFC PATCH 0/6] ARMv7: Add HYP mode switching support

[U-Boot] [RFC PATCH 0/6] ARMv7: Add HYP mode switching support

[Andre Przywara]

ARM CPUs with the virtualization extension have a new mode called
HYP mode, which allows hypervisors to safely control and monitor
guests. The current hypervisor (KVM and Xen) implementations
require the kernel to be entered in that HYP mode.

This patch series introduces a new U-Boot command called "hypmode"
which can be used at any time at the U-Boot command prompt to
switch the CPU into HYP mode - ideally this would be done before
starting the kernel. Since U-Boot does not use the MMU, it runs fine
in HYP mode, so you could as well enter HYP mode earlier.

The process of switching into HYP mode requires the CPU to be in
non-secure state, which requires the GIC to be programmed properly
first. Explanations about the details are in the commit messages
of the respective patches.

The code aims to be as generic as possible, though currently it has
only been tested on the Versatile Express TC-2 board. The last patch
thus enables the feature for that board and relies on the Versatile
Express updates patches sent out lately[1]. 

I would like to get some feedback on the patches, especially about:
1) Is the code in the right places? I used arch/arm/lib/ and
arch/arm/cpu/armv7/ as directories for the ARM code and
common/ for the actual command.
2) Is a command to switch to HYP actually the right thing? Should
HYP be entered automatically instead? Or provide an option to skip
HYP mode if requested by the user?
3) Does it make sense to provide a "nonsec" command also? I cannot
name any usecases by now, but the implementation would be rather
easy with basically all the functionality already there.

In general I appreciate any comments about coding style, patch
layout or the actual implementation.

Thanks for watching!

Andre.

[1] http://lists.denx.de/pipermail/u-boot/2013-April/151366.html

Andre Przywara (6):
  ARM: add secure monitor handler to switch to non-secure state
  ARM: add assembly routine to switch to non-secure state
  ARM: add U-Boot command "hypmode" to switch to non-secure state
  ARM: add SMP support for non-secure switch
  ARM: extend non-secure switch to also go into HYP mode
  ARM: VExpress: enable ARMv7 virt support for VExpress A15

 arch/arm/cpu/armv7/start.S          | 116 ++++++++++++++++++++++++++++---
 arch/arm/include/asm/armv7.h        |   2 +
 arch/arm/lib/Makefile               |   2 +
 arch/arm/lib/virt-v7.c              | 132 ++++++++++++++++++++++++++++++++++++
 common/Makefile                     |   1 +
 common/cmd_virt.c                   |  65 ++++++++++++++++++
 include/configs/vexpress_ca15_tc2.h |   3 +
 7 files changed, 311 insertions(+), 10 deletions(-)
 create mode 100644 arch/arm/lib/virt-v7.c
 create mode 100644 common/cmd_virt.c

-- 
1.7.12.1

[U-Boot] [RFC PATCH 1/6] ARM: add secure monitor handler to switch to non-secure state

[Andre Przywara]

A prerequisite for using virtualization is to be in HYP mode, which
requires the CPU to be in non-secure state.
According to the ARM ARM this should not be done in SVC mode, so we
have to setup a SMC handler for this. We reuse the current vector
table for this and make sure that we only access the MVBAR register
if the CPU supports the virtualization extensions.

Introduce a monitor handler routine which switches the CPU to
non-secure state by setting the NS bit (and associated bits).

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 arch/arm/cpu/armv7/start.S | 20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)

diff --git a/arch/arm/cpu/armv7/start.S b/arch/arm/cpu/armv7/start.S
index e9e57e6..7bfb19d 100644
--- a/arch/arm/cpu/armv7/start.S
+++ b/arch/arm/cpu/armv7/start.S
@@ -155,6 +155,10 @@ reset:
 	/* Set vector address in CP15 VBAR register */
 	ldr	r0, =_start
 	mcr	p15, 0, r0, c12, c0, 0	@Set VBAR
+
+	mrc	p15, 0, r1, c0, c1, 1	@ check for security extension
+	ands	r1, r1, #0x30
+	mcrne	p15, 0, r0, c12, c0, 1	@ Set secure monitor MVBAR
 #endif
 
 	/* the mask ROM code should have PLL and others stable */
@@ -256,6 +260,9 @@ ENTRY(c_runtime_cpu_setup)
 	/* Set vector address in CP15 VBAR register */
 	ldr     r0, =_start
 	mcr     p15, 0, r0, c12, c0, 0  @Set VBAR
+	mrc	p15, 0, r1, c0, c1, 1	@ check for security extension
+	ands	r1, r1, #0x30
+	mcrne	p15, 0, r0, c12, c0, 1  @ Set secure monitor MVBAR
 
 	bx	lr
 
@@ -492,9 +499,16 @@ undefined_instruction:
 
 	.align	5
 software_interrupt:
-	get_bad_stack_swi
-	bad_save_user_regs
-	bl	do_software_interrupt
+	mrc	p15, 0, r1, c1, c1, 0		@ read SCR
+	bic	r1, r1, #0x07f
+	orr	r1, r1, #0x31			@ enable NS, AW, FW
+
+	mrc	p15, 0, r0, c12, c0, 0		@ save secure copy of VBAR
+	mcr	p15, 0, r1, c1, c1, 0		@ write SCR, switch to non-sec
+	isb
+	mcr	p15, 0, r0, c12, c0, 0		@ write non-secure copy of VBAR
+
+	movs	pc, lr
 
 	.align	5
 prefetch_abort:
-- 
1.7.12.1

[U-Boot] [RFC PATCH 2/6] ARM: add assembly routine to switch to non-secure state

[Andre Przywara]

While actually switching to non-secure state is one thing, the
more important part of this process is to make sure that we still
have full access to the interrupt controller (GIC).
The GIC is fully aware of secure vs. non-secure state, some
registers are banked, others may be configured to be accessible from
secure state only.
To be as generic as possible, we get the GIC memory mapped address
based on the PERIPHBASE register. We check explicitly for
ARM Cortex-A7 and A15 cores, assuming an A9 otherwise, as for those
cores we know the offsets for the GIC CPU interface from the
PERIPHBASE content. Other cores could be added as needed.

With the GIC accessible, we:
a) allow private interrupts to be delivered to the core
   (GICD_IGROUPR0 = 0xFFFFFFFF)
b) enable the CPU interface (GICC_CTLR[0] = 1)
c) set the priority filter to allow non-secure interrupts
   (GICC_PMR = 0x80)

After having switched to non-secure state, we also enable the
non-secure GIC CPU interface, since this register is banked.

Also we allow access to all coprocessor interfaces from non-secure
state by writing the appropriate bits in the NSACR register.

For reasons obvious later we only use caller saved registers r0-r3.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 arch/arm/cpu/armv7/start.S | 45 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)

diff --git a/arch/arm/cpu/armv7/start.S b/arch/arm/cpu/armv7/start.S
index 7bfb19d..401b0eb 100644
--- a/arch/arm/cpu/armv7/start.S
+++ b/arch/arm/cpu/armv7/start.S
@@ -561,3 +561,48 @@ fiq:
 
 #endif /* CONFIG_USE_IRQ */
 #endif /* CONFIG_SPL_BUILD */
+
+/* Routine to initialize GIC CPU interface and switch to nonsecure state.
+ */
+.globl _nonsec_gic_switch
+_nonsec_gic_switch:
+	mrc	p15, 4, r2, c15, c0, 0		@ r2 = PERIPHBASE
+	add	r3, r2, #0x1000			@ GIC dist i/f offset
+	mvn	r1, #0
+	str	r1, [r3, #0x80]			@ allow private interrupts
+
+	mrc	p15, 0, r0, c0, c0, 0		@ MIDR
+	bfc	r0, #16, #8			@ mask out variant, arch
+	bfc	r0, #0, #4			@ and revision
+	movw	r1, #0xc070
+	movt	r1, #0x4100
+	cmp	r0, r1				@ check for Cortex-A7
+	orr	r1, #0xf0
+	cmpne	r0, r1				@ check for Cortex-A15
+	movne	r1, #0x100			@ GIC CPU offset for A9
+	moveq	r1, #0x2000			@ GIC CPU offset for A15/A7
+	add	r3, r2, r1			@ r3 = GIC CPU i/f addr
+
+	mov	r1, #1
+	str	r1, [r3, #0]			@ set GICC_CTLR[enable]
+	mov	r1, #0x80
+	str	r1, [r3, #4]			@ set GICC_PIMR[7]
+
+	movw	r1, #0x3fff
+	movt	r1, #0x0006
+	mcr	p15, 0, r1, c1, c1, 2		@ NSACR = all copros to non-sec
+
+	ldr	r1, =_start
+	mcr	p15, 0, r1, c12, c0, 0		@ set VBAR
+	mcr	p15, 0, r1, c12, c0, 1		@ set MVBAR
+
+	isb
+	smc	#0				@ call into MONITOR mode
+	isb					@ clobbers r0 and r1
+
+	mov	r1, #1
+	str	r1, [r3, #0]			@ set GICC_CTLR[enable]
+	add	r2, r2, #0x1000			@ GIC dist i/f offset
+	str	r1, [r2]			@ allow private interrupts
+
+	mov	pc, lr
-- 
1.7.12.1

[U-Boot] [RFC PATCH 3/6] ARM: add U-Boot command "hypmode" to switch to non-secure state

[Andre Przywara]

In preparation for the actual HYP mode switch, we introduce a new
U-Boot command called "hypmode". For now we only do the non-secure
switch here.

Some part of the work is done in the assembly routine in start.S,
introduced with the previous patch, but for the full glory we need
to setup the GIC distributor interface once for the whole system,
which is done in C here.
The routine is placed in arch/arm/lib to allow easy access from
different boards or CPUs.

First we check for the availability of the security extensions.

The generic timer base frequency register is only accessible from
secure state, so we have to program it now. Actually this should be
done from primary firmware before, but some boards seems to omit
this, so if needed we do this here with a board specific value.

Since we need a safe way to access the GIC, we use the PERIPHBASE
registers on Cortex-A15 and A7 CPUs and do some sanity checks.

Then we actually do the GIC enablement:
a) enable the GIC distributor, both for non-secure and secure state
   (GICD_CTLR[1:0] = 11b)
b) allow all interrupts to be handled from non-secure state
   (GICD_IGROUPRn = 0xFFFFFFFF)
The core specific GIC setup is then done in the assembly routine.

The actual U-Boot command is pretty small: calling the routine and
doing some error reporting. A return value of 1 will be added later.

To enable the whole code we introduce the CONFIG_CMD_VIRT variable.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 arch/arm/include/asm/armv7.h |   2 +
 arch/arm/lib/Makefile        |   2 +
 arch/arm/lib/virt-v7.c       | 116 +++++++++++++++++++++++++++++++++++++++++++
 common/Makefile              |   1 +
 common/cmd_virt.c            |  59 ++++++++++++++++++++++
 5 files changed, 180 insertions(+)
 create mode 100644 arch/arm/lib/virt-v7.c
 create mode 100644 common/cmd_virt.c

diff --git a/arch/arm/include/asm/armv7.h b/arch/arm/include/asm/armv7.h
index a73630b..3567692 100644
--- a/arch/arm/include/asm/armv7.h
+++ b/arch/arm/include/asm/armv7.h
@@ -74,4 +74,6 @@ void v7_outer_cache_inval_all(void);
 void v7_outer_cache_flush_range(u32 start, u32 end);
 void v7_outer_cache_inval_range(u32 start, u32 end);
 
+int armv7_switch_nonsec(void);
+
 #endif
diff --git a/arch/arm/lib/Makefile b/arch/arm/lib/Makefile
index 6ae161a..c5b6b69 100644
--- a/arch/arm/lib/Makefile
+++ b/arch/arm/lib/Makefile
@@ -58,6 +58,8 @@ COBJS-y	+= reset.o
 COBJS-y	+= cache.o
 COBJS-y	+= cache-cp15.o
 
+COBJS-y	+= virt-v7.o
+
 SRCS	:= $(GLSOBJS:.o=.S) $(GLCOBJS:.o=.c) \
 	   $(SOBJS-y:.o=.S) $(COBJS-y:.o=.c)
 OBJS	:= $(addprefix $(obj),$(SOBJS-y) $(COBJS-y))
diff --git a/arch/arm/lib/virt-v7.c b/arch/arm/lib/virt-v7.c
new file mode 100644
index 0000000..416ca29
--- /dev/null
+++ b/arch/arm/lib/virt-v7.c
@@ -0,0 +1,116 @@
+/*
+ * (C) Copyright 2013
+ * Andre Przywara, Linaro
+ *
+ * routines to push ARMv7 processors from secure into non-secure state
+ * needed to enable ARMv7 virtualization for current hypervisors
+ *
+ * See file CREDITS for list of people who contributed to this
+ * project.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation; either version 2 of
+ * the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+ * MA 02111-1307 USA
+ */
+
+#include <common.h>
+#include <asm/armv7.h>
+
+/* the assembly routine doing the actual work in start.S */
+void _nonsec_gic_switch(void);
+
+#define GICD_CTLR	0x000
+#define GICD_TYPER	0x004
+#define GICD_IGROUPR0	0x080
+#define GICD_SGIR	0xf00
+
+#define CPU_ARM_CORTEX_A15	0x4100c0f0
+#define CPU_ARM_CORTEX_A7	0x4100c070
+
+static inline unsigned int read_cpsr(void)
+{
+	unsigned int reg;
+
+	asm volatile ("mrs %0, cpsr\n" : "=r" (reg));
+	return reg;
+}
+
+int armv7_switch_nonsec(void)
+{
+	unsigned int reg;
+	volatile unsigned int *gicdptr;
+	unsigned itlinesnr, i;
+
+	/* check whether the CPU supports the security extensions */
+	asm("mrc p15, 0, %0, c0, c1, 1\n" : "=r"(reg));
+	if ((reg & 0xF0) == 0)
+		return 2;
+
+	/* the timer frequency for the generic timer needs to be
+	 * programmed still in secure state, should be done by firmware.
+	 * check whether we have the generic timer first
+	 */
+#ifdef CONFIG_SYS_CLK_FREQ
+	asm("mrc p15, 0, %0, c0, c1, 1\n" : "=r"(reg));
+	if ((reg & 0xF0000) == 0x10000)
+		asm("mcr p15, 0, %0, c14, c0, 0\n"
+		: : "r"(CONFIG_SYS_CLK_FREQ));
+#endif
+
+	/* the SCR register will be set directly in the monitor mode handler,
+	 * according to the spec one should not tinker with it in secure state
+	 * in SVC mode. Do not try to read it once in non-secure state,
+	 * any access to it will trap.
+	 */
+
+	/* check whether we are an Cortex-A15 or A7.
+	 * The actual non-secure switch should work with all CPUs supporting
+	 * the security extension, but we need the GIC address,
+	 * which we know only for sure for those two CPUs.
+	 */
+	asm("mrc p15, 0, %0, c0, c0, 0\n" : "=r"(reg));
+	if (((reg & 0xFF00FFF0) != 0x4100C0F0) &&
+	    ((reg & 0xFF00FFF0) != 0x4100C070))
+		return 3;
+
+	/* get the GIC base address from the A15 PERIPHBASE register */
+	asm("mrc p15, 4, %0, c15, c0, 0\n" : "=r" (reg));
+
+	/* the PERIPHBASE can be mapped above 4 GB (lower 8 bits used to
+	 * encode this). Bail out here since we cannot access this without
+	 * enabling paging.
+	 */
+	if ((reg & 0xff) != 0)
+		return 4;
+
+	/* GIC distributor registers start at offset 0x1000 */
+	gicdptr = (unsigned *)(reg + 0x1000);
+
+	/* enable the GIC distributor */
+	gicdptr[GICD_CTLR / 4] |= 0x03;
+
+	/* TYPER[4:0] contains an encoded number of all interrupts */
+	itlinesnr = gicdptr[GICD_TYPER / 4] & 0x1f;
+
+	/* set all bits in the GIC group registers to one to allow access
+	 * from non-secure state
+	 */
+	for (i = 0; i <= itlinesnr; i++)
+		gicdptr[GICD_IGROUPR0 / 4 + i] = (unsigned)-1;
+
+	/* call the non-sec switching code on this CPU */
+	_nonsec_gic_switch();
+
+	return 0;
+}
diff --git a/common/Makefile b/common/Makefile
index 0e0fff1..d9ba5ab 100644
--- a/common/Makefile
+++ b/common/Makefile
@@ -129,6 +129,7 @@ COBJS-y += cmd_load.o
 COBJS-$(CONFIG_LOGBUFFER) += cmd_log.o
 COBJS-$(CONFIG_ID_EEPROM) += cmd_mac.o
 COBJS-$(CONFIG_CMD_MD5SUM) += cmd_md5sum.o
+COBJS-$(CONFIG_CMD_VIRT) += cmd_virt.o
 COBJS-$(CONFIG_CMD_MEMORY) += cmd_mem.o
 COBJS-$(CONFIG_CMD_IO) += cmd_io.o
 COBJS-$(CONFIG_CMD_MFSL) += cmd_mfsl.o
diff --git a/common/cmd_virt.c b/common/cmd_virt.c
new file mode 100644
index 0000000..132b6b1
--- /dev/null
+++ b/common/cmd_virt.c
@@ -0,0 +1,59 @@
+/*
+ * (C) Copyright 2013
+ * Andre Przywara, Linaro
+ *
+ * command to switch an ARMv7 CPU with security extensions into
+ * non-secure state
+ *
+ * See file CREDITS for list of people who contributed to this
+ * project.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation; either version 2 of
+ * the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+ * MA 02111-1307 USA
+ */
+
+#include <common.h>
+#include <command.h>
+
+#include <asm/armv7.h>
+
+static int do_nonsec(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])
+{
+	unsigned int ret;
+
+	ret = armv7_switch_nonsec();
+
+	switch (ret) {
+	case 0:
+		break;
+	case 2:
+		printf("Security extensions not implemented.\n");
+		break;
+	case 3:
+		printf("CPU not supported, must be either Cortex-A15 or A7.\n");
+		break;
+	case 4:
+		printf("PERIPHBASE is above 4 GB, cannot access this.\n");
+		break;
+	}
+
+	return ret;
+}
+
+U_BOOT_CMD(
+	hypmode, 1, 0, do_nonsec,
+	"switch ARM CPUs into non-secure state",
+	""
+);
-- 
1.7.12.1

[U-Boot] [RFC PATCH 4/6] ARM: add SMP support for non-secure switch

[Andre Przywara]

Currently the non-secure switch is only done for the boot processor.
To later allow full SMP support, we have to switch all secondary
cores into non-secure state also.

So we add an entry point for secondary CPUs coming out of low-power
state and make sure we put them into WFI again after having switched
to non-secure state.
For this we acknowledge and EOI the wake-up IPI, then go into WFI.
Once being kicked out of it later, we sanity check that the start
address has actually been changed (since another attempt to switch
to non-secure would block the core) and jump to the new address.

The actual CPU kick is done by sending an inter-processor interrupt
via the GIC to all CPU interfaces except the requesting processor.
The secondary cores will then setup their respective GIC CPU
interface.

The address secondary cores jump to is board specific, we provide
the value here for the Versatile Express board.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 arch/arm/cpu/armv7/start.S          | 27 ++++++++++++++++++++++++++-
 arch/arm/lib/virt-v7.c              | 10 +++++++++-
 include/configs/vexpress_ca15_tc2.h |  1 +
 3 files changed, 36 insertions(+), 2 deletions(-)

diff --git a/arch/arm/cpu/armv7/start.S b/arch/arm/cpu/armv7/start.S
index 401b0eb..2b47881 100644
--- a/arch/arm/cpu/armv7/start.S
+++ b/arch/arm/cpu/armv7/start.S
@@ -563,8 +563,19 @@ fiq:
 #endif /* CONFIG_SPL_BUILD */
 
 /* Routine to initialize GIC CPU interface and switch to nonsecure state.
+ * Will be executed directly by secondary CPUs after coming out of
+ * WFI, or can be called directly by C code for CPU 0.
+ * Those two paths mandate to not use any stack and to only use registers
+ * r0-r3 to comply with both the C ABI and the requirement of SMP startup
+ * code.
  */
 .globl _nonsec_gic_switch
+.globl _smp_pen
+_smp_pen:
+	mrs	r0, cpsr
+	orr	r0, r0, #0xc0
+	msr	cpsr, r0			@ disable interrupts
+	mov	lr, #0				@ clear LR to mark secondary
 _nonsec_gic_switch:
 	mrc	p15, 4, r2, c15, c0, 0		@ r2 = PERIPHBASE
 	add	r3, r2, #0x1000			@ GIC dist i/f offset
@@ -605,4 +616,18 @@ _nonsec_gic_switch:
 	add	r2, r2, #0x1000			@ GIC dist i/f offset
 	str	r1, [r2]			@ allow private interrupts
 
-	mov	pc, lr
+	cmp	lr, #0
+	movne	pc, lr				@ CPU 0 to return
+						@ all others: go to sleep
+_ack_int:
+	ldr	r1, [r3, #0x0c]			@ read GICD acknowledge
+	str	r1, [r3, #0x10]			@ write GICD EOI
+
+	adr	r1, _smp_pen
+waitloop:
+	wfi
+	ldr	r0, =CONFIG_SYSFLAGS_ADDR	@ load start address
+	ldr	r0, [r0]
+	cmp	r0, r1			@ make sure we dont execute this code
+	beq	waitloop		@ again (due to a spurious wakeup)
+	mov	pc, r0
diff --git a/arch/arm/lib/virt-v7.c b/arch/arm/lib/virt-v7.c
index 416ca29..5ca093a 100644
--- a/arch/arm/lib/virt-v7.c
+++ b/arch/arm/lib/virt-v7.c
@@ -29,6 +29,7 @@
 
 /* the assembly routine doing the actual work in start.S */
 void _nonsec_gic_switch(void);
+void _smp_pen(void);
 
 #define GICD_CTLR	0x000
 #define GICD_TYPER	0x004
@@ -51,6 +52,7 @@ int armv7_switch_nonsec(void)
 	unsigned int reg;
 	volatile unsigned int *gicdptr;
 	unsigned itlinesnr, i;
+	unsigned int *sysflags;
 
 	/* check whether the CPU supports the security extensions */
 	asm("mrc p15, 0, %0, c0, c1, 1\n" : "=r"(reg));
@@ -109,7 +111,13 @@ int armv7_switch_nonsec(void)
 	for (i = 0; i <= itlinesnr; i++)
 		gicdptr[GICD_IGROUPR0 / 4 + i] = (unsigned)-1;
 
-	/* call the non-sec switching code on this CPU */
+	/* now kick all CPUs (expect this one) by writing to GICD_SIGR */
+	sysflags = (void *)CONFIG_SYSFLAGS_ADDR;
+	sysflags[1] = (unsigned)-1;
+	sysflags[0] = (uintptr_t)_smp_pen;
+	gicdptr[GICD_SGIR / 4] = 1U << 24;
+
+	/* call the non-sec switching code on this CPU also */
 	_nonsec_gic_switch();
 
 	return 0;
diff --git a/include/configs/vexpress_ca15_tc2.h b/include/configs/vexpress_ca15_tc2.h
index 9e230ad..210a27c 100644
--- a/include/configs/vexpress_ca15_tc2.h
+++ b/include/configs/vexpress_ca15_tc2.h
@@ -32,5 +32,6 @@
 #define CONFIG_BOOTP_VCI_STRING     "U-boot.armv7.vexpress_ca15x2_tc2"
 
 #define CONFIG_SYS_CLK_FREQ 24000000
+#define CONFIG_SYSFLAGS_ADDR 0x1c010030
 
 #endif
-- 
1.7.12.1

[U-Boot] [RFC PATCH 5/6] ARM: extend non-secure switch to also go into HYP mode

[Andre Przywara]

For the KVM and XEN hypervisors to be usable, we need to enter the
kernel in HYP mode. Now that we already are in non-secure state,
HYP mode switching is within short reach.

While doing the non-secure switch, we have to enable the HVC
instruction and setup the HYP mode HVBAR (while still secure).

The actual switch is done by dropping back from a HYP mode handler
without actually leaving HYP mode, so we introduce a new handler
routine in the exception vector table.

In the assembly switching routine - which we rename to hyp_gic_switch
on the way - we save and restore the banked LR and SP registers
around the hypercall to do the actual HYP mode switch.

The C routine first checks whether we are in HYP mode already and
also whether the virtualization extensions are available. It also
checks whether the HYP mode switch was finally successful.
The U-Boot command only adds and adjusts some error reporting.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 arch/arm/cpu/armv7/start.S   | 34 +++++++++++++++++++++++-----------
 arch/arm/include/asm/armv7.h |  2 +-
 arch/arm/lib/virt-v7.c       | 24 ++++++++++++++++--------
 common/cmd_virt.c            | 20 +++++++++++++-------
 4 files changed, 53 insertions(+), 27 deletions(-)

diff --git a/arch/arm/cpu/armv7/start.S b/arch/arm/cpu/armv7/start.S
index 2b47881..00890a3 100644
--- a/arch/arm/cpu/armv7/start.S
+++ b/arch/arm/cpu/armv7/start.S
@@ -41,7 +41,7 @@ _start: b	reset
 	ldr	pc, _software_interrupt
 	ldr	pc, _prefetch_abort
 	ldr	pc, _data_abort
-	ldr	pc, _not_used
+	ldr	pc, _hyp_trap
 	ldr	pc, _irq
 	ldr	pc, _fiq
 #ifdef CONFIG_SPL_BUILD
@@ -49,7 +49,7 @@ _undefined_instruction: .word _undefined_instruction
 _software_interrupt:	.word _software_interrupt
 _prefetch_abort:	.word _prefetch_abort
 _data_abort:		.word _data_abort
-_not_used:		.word _not_used
+_hyp_trap:		.word _hyp_trap
 _irq:			.word _irq
 _fiq:			.word _fiq
 _pad:			.word 0x12345678 /* now 16*4=64 */
@@ -58,7 +58,7 @@ _undefined_instruction: .word undefined_instruction
 _software_interrupt:	.word software_interrupt
 _prefetch_abort:	.word prefetch_abort
 _data_abort:		.word data_abort
-_not_used:		.word not_used
+_hyp_trap:		.word hyp_trap
 _irq:			.word irq
 _fiq:			.word fiq
 _pad:			.word 0x12345678 /* now 16*4=64 */
@@ -502,12 +502,18 @@ software_interrupt:
 	mrc	p15, 0, r1, c1, c1, 0		@ read SCR
 	bic	r1, r1, #0x07f
 	orr	r1, r1, #0x31			@ enable NS, AW, FW
+	mrc	p15, 0, r0, c0, c1, 1		@ check for Virt ext
+	and	r0, r0, #0xf000
+	cmp	r0, #0x1000
+	orreq	r1, r1, #0x100			@ allow HVC instruction
 
 	mrc	p15, 0, r0, c12, c0, 0		@ save secure copy of VBAR
 	mcr	p15, 0, r1, c1, c1, 0		@ write SCR, switch to non-sec
 	isb
 	mcr	p15, 0, r0, c12, c0, 0		@ write non-secure copy of VBAR
 
+	mcreq	p15, 4, r0, c12, c0, 0		@ write HYP mode HVBAR
+
 	movs	pc, lr
 
 	.align	5
@@ -523,10 +529,9 @@ data_abort:
 	bl	do_data_abort
 
 	.align	5
-not_used:
-	get_bad_stack
-	bad_save_user_regs
-	bl	do_not_used
+hyp_trap:
+	.byte 0x00, 0xe3, 0x0e, 0xe1		@ mrs lr, elr_hyp
+	mov pc, lr
 
 #ifdef CONFIG_USE_IRQ
 
@@ -562,21 +567,21 @@ fiq:
 #endif /* CONFIG_USE_IRQ */
 #endif /* CONFIG_SPL_BUILD */
 
-/* Routine to initialize GIC CPU interface and switch to nonsecure state.
- * Will be executed directly by secondary CPUs after coming out of
+/* Routine to initialize GIC CPU interface, switch to nonsecure and to HYP
+ * mode. Will be executed directly by secondary CPUs after coming out of
  * WFI, or can be called directly by C code for CPU 0.
  * Those two paths mandate to not use any stack and to only use registers
  * r0-r3 to comply with both the C ABI and the requirement of SMP startup
  * code.
  */
-.globl _nonsec_gic_switch
+.globl _hyp_gic_switch
 .globl _smp_pen
 _smp_pen:
 	mrs	r0, cpsr
 	orr	r0, r0, #0xc0
 	msr	cpsr, r0			@ disable interrupts
 	mov	lr, #0				@ clear LR to mark secondary
-_nonsec_gic_switch:
+_hyp_gic_switch:
 	mrc	p15, 4, r2, c15, c0, 0		@ r2 = PERIPHBASE
 	add	r3, r2, #0x1000			@ GIC dist i/f offset
 	mvn	r1, #0
@@ -616,6 +621,13 @@ _nonsec_gic_switch:
 	add	r2, r2, #0x1000			@ GIC dist i/f offset
 	str	r1, [r2]			@ allow private interrupts
 
+	mov	r2, lr
+	mov	r1, sp
+	.byte 0x70, 0x00, 0x40, 0xe1		@ hvc #0
+	isb
+	mov	sp, r1
+	mov	lr, r2
+
 	cmp	lr, #0
 	movne	pc, lr				@ CPU 0 to return
 						@ all others: go to sleep
diff --git a/arch/arm/include/asm/armv7.h b/arch/arm/include/asm/armv7.h
index 3567692..6c6955b 100644
--- a/arch/arm/include/asm/armv7.h
+++ b/arch/arm/include/asm/armv7.h
@@ -74,6 +74,6 @@ void v7_outer_cache_inval_all(void);
 void v7_outer_cache_flush_range(u32 start, u32 end);
 void v7_outer_cache_inval_range(u32 start, u32 end);
 
-int armv7_switch_nonsec(void);
+int armv7_switch_hyp(void);
 
 #endif
diff --git a/arch/arm/lib/virt-v7.c b/arch/arm/lib/virt-v7.c
index 5ca093a..72481c6 100644
--- a/arch/arm/lib/virt-v7.c
+++ b/arch/arm/lib/virt-v7.c
@@ -3,6 +3,7 @@
  * Andre Przywara, Linaro
  *
  * routines to push ARMv7 processors from secure into non-secure state
+ * and from non-secure SVC into HYP mode
  * needed to enable ARMv7 virtualization for current hypervisors
  *
  * See file CREDITS for list of people who contributed to this
@@ -28,7 +29,7 @@
 #include <asm/armv7.h>
 
 /* the assembly routine doing the actual work in start.S */
-void _nonsec_gic_switch(void);
+void _hyp_gic_switch(void);
 void _smp_pen(void);
 
 #define GICD_CTLR	0x000
@@ -47,16 +48,20 @@ static inline unsigned int read_cpsr(void)
 	return reg;
 }
 
-int armv7_switch_nonsec(void)
+int armv7_switch_hyp(void)
 {
 	unsigned int reg;
 	volatile unsigned int *gicdptr;
 	unsigned itlinesnr, i;
 	unsigned int *sysflags;
 
-	/* check whether the CPU supports the security extensions */
+	/* check whether we are in HYP mode already */
+	if ((read_cpsr() & 0x1F) == 0x1a)
+		return 1;
+
+	/* check whether the CPU supports the virtualization extensions */
 	asm("mrc p15, 0, %0, c0, c1, 1\n" : "=r"(reg));
-	if ((reg & 0xF0) == 0)
+	if ((reg & 0xF000) != 0x1000)
 		return 2;
 
 	/* the timer frequency for the generic timer needs to be
@@ -77,8 +82,8 @@ int armv7_switch_nonsec(void)
 	 */
 
 	/* check whether we are an Cortex-A15 or A7.
-	 * The actual non-secure switch should work with all CPUs supporting
-	 * the security extension, but we need the GIC address,
+	 * The actual HYP switch should work with all CPUs supporting
+	 * the virtualization extension, but we need the GIC address,
 	 * which we know only for sure for those two CPUs.
 	 */
 	asm("mrc p15, 0, %0, c0, c0, 0\n" : "=r"(reg));
@@ -117,8 +122,11 @@ int armv7_switch_nonsec(void)
 	sysflags[0] = (uintptr_t)_smp_pen;
 	gicdptr[GICD_SGIR / 4] = 1U << 24;
 
-	/* call the non-sec switching code on this CPU also */
-	_nonsec_gic_switch();
+	/* call the HYP switching code on this CPU also */
+	_hyp_gic_switch();
+
+	if ((read_cpsr() & 0x1F) != 0x1a)
+		return 5;
 
 	return 0;
 }
diff --git a/common/cmd_virt.c b/common/cmd_virt.c
index 132b6b1..78b1aa1 100644
--- a/common/cmd_virt.c
+++ b/common/cmd_virt.c
@@ -2,8 +2,8 @@
  * (C) Copyright 2013
  * Andre Przywara, Linaro
  *
- * command to switch an ARMv7 CPU with security extensions into
- * non-secure state
+ * command to switch an ARMv7 CPU with virtualization extensions into
+ * HYP mode to allow hypervisors to install themselves
  *
  * See file CREDITS for list of people who contributed to this
  * project.
@@ -29,17 +29,20 @@
 
 #include <asm/armv7.h>
 
-static int do_nonsec(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])
+static int do_hyp(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])
 {
 	unsigned int ret;
 
-	ret = armv7_switch_nonsec();
+	ret = armv7_switch_hyp();
 
 	switch (ret) {
 	case 0:
 		break;
+	case 1:
+		printf("Already in HYP mode\n");
+		break;
 	case 2:
-		printf("Security extensions not implemented.\n");
+		printf("Virtualization extensions not implemented.\n");
 		break;
 	case 3:
 		printf("CPU not supported, must be either Cortex-A15 or A7.\n");
@@ -47,13 +50,16 @@ static int do_nonsec(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])
 	case 4:
 		printf("PERIPHBASE is above 4 GB, cannot access this.\n");
 		break;
+	case 5:
+		printf("HYP mode switch not successful.\n");
+		break;
 	}
 
 	return ret;
 }
 
 U_BOOT_CMD(
-	hypmode, 1, 0, do_nonsec,
-	"switch ARM CPUs into non-secure state",
+	hypmode, 1, 0, do_hyp,
+	"switch ARM CPUs into HYP mode",
 	""
 );
-- 
1.7.12.1

[U-Boot] [RFC PATCH 6/6] ARM: VExpress: enable ARMv7 virt support for VExpress A15

[Andre Przywara]

Enable the "hypmode" command for the Versatile Express TC2 board
with the virtualization capable Cortex-A15 CPU to allow booting
Xen or a KVM-enabled Linux kernel in HYP mode.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 include/configs/vexpress_ca15_tc2.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/include/configs/vexpress_ca15_tc2.h b/include/configs/vexpress_ca15_tc2.h
index 210a27c..9df4400 100644
--- a/include/configs/vexpress_ca15_tc2.h
+++ b/include/configs/vexpress_ca15_tc2.h
@@ -31,6 +31,8 @@
 #include "vexpress_common.h"
 #define CONFIG_BOOTP_VCI_STRING     "U-boot.armv7.vexpress_ca15x2_tc2"
 
+#define CONFIG_CMD_VIRT
+
 #define CONFIG_SYS_CLK_FREQ 24000000
 #define CONFIG_SYSFLAGS_ADDR 0x1c010030
 
-- 
1.7.12.1

[U-Boot] [RFC PATCH 0/6] ARMv7: Add HYP mode switching support

[Peter Maydell]

On 26 April 2013 14:14, Andre Przywara <andre.przywara@linaro.org> wrote:
> ARM CPUs with the virtualization extension have a new mode called
> HYP mode, which allows hypervisors to safely control and monitor
> guests. The current hypervisor (KVM and Xen) implementations
> require the kernel to be entered in that HYP mode.
>
> This patch series introduces a new U-Boot command called "hypmode"
> which can be used at any time at the U-Boot command prompt to
> switch the CPU into HYP mode - ideally this would be done before
> starting the kernel.

The obvious question here is "why do we need a new command?".
The kernel booting specification says "boot the kernel in
Hyp mode" so we should just always do that for booting Linux,
surely?

thanks
-- PMM

[U-Boot] [RFC PATCH 0/6] ARMv7: Add HYP mode switching support

[Andre Przywara]

On 04/26/2013 03:18 PM, Peter Maydell wrote:
> On 26 April 2013 14:14, Andre Przywara <andre.przywara@linaro.org> wrote:
>> ARM CPUs with the virtualization extension have a new mode called
>> HYP mode, which allows hypervisors to safely control and monitor
>> guests. The current hypervisor (KVM and Xen) implementations
>> require the kernel to be entered in that HYP mode.
>>
>> This patch series introduces a new U-Boot command called "hypmode"
>> which can be used at any time at the U-Boot command prompt to
>> switch the CPU into HYP mode - ideally this would be done before
>> starting the kernel.
>
> The obvious question here is "why do we need a new command?".
> The kernel booting specification says "boot the kernel in
> Hyp mode" so we should just always do that for booting Linux,
> surely?

Because it avoids regressions. I kind of feel uneasy to do a lot of 
tinkering with secure state and the GIC unconditionally, especially if 
enabled on many boards with virt-capable CPUs.

As written in question 2) later in that mail, I can also live with a 
command to _dis_able the HYP mode switching in case it causes problems.

Regards,
Andre.

[U-Boot] [RFC PATCH 0/6] ARMv7: Add HYP mode switching support

[Peter Maydell]

On 26 April 2013 14:24, Andre Przywara <andre.przywara@linaro.org> wrote:
> On 04/26/2013 03:18 PM, Peter Maydell wrote:
>> The obvious question here is "why do we need a new command?".
>> The kernel booting specification says "boot the kernel in
>> Hyp mode" so we should just always do that for booting Linux,
>> surely?

> Because it avoids regressions. I kind of feel uneasy to do a lot of
> tinkering with secure state and the GIC unconditionally, especially if
> enabled on many boards with virt-capable CPUs.

There aren't exactly very many of those out there, so if we
default to "boot in HYP mode" then (a) KVM will just work
out of the box and (b) people doing u-boot ports to their
board will find any issues and be able to submit fixes.
If we don't turn it on by default then we'll end up stuck
in a world where KVM doesn't work on half the virt capable
boards out there.

I have no objection to a "turn this off" escape hatch if
you think it's useful.

-- PMM

[U-Boot] [RFC PATCH 0/6] ARMv7: Add HYP mode switching support

[Andre Przywara]

On 04/26/2013 03:42 PM, Peter Maydell wrote:
> On 26 April 2013 14:24, Andre Przywara <andre.przywara@linaro.org> wrote:
>> On 04/26/2013 03:18 PM, Peter Maydell wrote:
>>> The obvious question here is "why do we need a new command?".
>>> The kernel booting specification says "boot the kernel in
>>> Hyp mode" so we should just always do that for booting Linux,
>>> surely?
>
>> Because it avoids regressions. I kind of feel uneasy to do a lot of
>> tinkering with secure state and the GIC unconditionally, especially if
>> enabled on many boards with virt-capable CPUs.
>
> There aren't exactly very many of those out there, so if we
> default to "boot in HYP mode" then (a) KVM will just work
> out of the box and (b) people doing u-boot ports to their
> board will find any issues and be able to submit fixes.
> If we don't turn it on by default then we'll end up stuck
> in a world where KVM doesn't work on half the virt capable
> boards out there.

OK, that's a point. I changed the code already and will include it in 
the next revision.

Thanks,
Andre.

>
> I have no objection to a "turn this off" escape hatch if
> you think it's useful.
>
> -- PMM
>

[U-Boot] [RFC PATCH 0/6] ARMv7: Add HYP mode switching support

[Christoffer Dall]

On Fri, Apr 26, 2013 at 7:14 AM, Andre Przywara
<andre.przywara@linaro.org> wrote:
> On 04/26/2013 03:42 PM, Peter Maydell wrote:
>> On 26 April 2013 14:24, Andre Przywara <andre.przywara@linaro.org> wrote:
>>> On 04/26/2013 03:18 PM, Peter Maydell wrote:
>>>> The obvious question here is "why do we need a new command?".
>>>> The kernel booting specification says "boot the kernel in
>>>> Hyp mode" so we should just always do that for booting Linux,
>>>> surely?
>>
>>> Because it avoids regressions. I kind of feel uneasy to do a lot of
>>> tinkering with secure state and the GIC unconditionally, especially if
>>> enabled on many boards with virt-capable CPUs.
>>
>> There aren't exactly very many of those out there, so if we
>> default to "boot in HYP mode" then (a) KVM will just work
>> out of the box and (b) people doing u-boot ports to their
>> board will find any issues and be able to submit fixes.
>> If we don't turn it on by default then we'll end up stuck
>> in a world where KVM doesn't work on half the virt capable
>> boards out there.
>
> OK, that's a point. I changed the code already and will include it in
> the next revision.
>
I strongly agree with Peter, and in fact I think it should only be a
compile time option, not a run-time option, only for those debugging
u-boot and a kernel on some new platform.

-Christoffer

[U-Boot] [RFC PATCH 1/6] ARM: add secure monitor handler to switch to non-secure state

[Christoffer Dall]

On Fri, Apr 26, 2013 at 03:14:54PM +0200, Andre Przywara wrote:
> A prerequisite for using virtualization is to be in HYP mode, which
> requires the CPU to be in non-secure state.
> According to the ARM ARM this should not be done in SVC mode, so we
> have to setup a SMC handler for this. We reuse the current vector
> table for this and make sure that we only access the MVBAR register
> if the CPU supports the virtualization extensions.
>
> Introduce a monitor handler routine which switches the CPU to
> non-secure state by setting the NS bit (and associated bits).
>
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>  arch/arm/cpu/armv7/start.S | 20 +++++++++++++++++---
>  1 file changed, 17 insertions(+), 3 deletions(-)
>
> diff --git a/arch/arm/cpu/armv7/start.S b/arch/arm/cpu/armv7/start.S
> index e9e57e6..7bfb19d 100644
> --- a/arch/arm/cpu/armv7/start.S
> +++ b/arch/arm/cpu/armv7/start.S
> @@ -155,6 +155,10 @@ reset:
>   /* Set vector address in CP15 VBAR register */
>   ldr r0, =_start
>   mcr p15, 0, r0, c12, c0, 0 @Set VBAR
> +
> + mrc p15, 0, r1, c0, c1, 1 @ check for security extension
> + ands r1, r1, #0x30
> + mcrne p15, 0, r0, c12, c0, 1 @ Set secure monitor MVBAR
>  #endif
>
>   /* the mask ROM code should have PLL and others stable */
> @@ -256,6 +260,9 @@ ENTRY(c_runtime_cpu_setup)
>   /* Set vector address in CP15 VBAR register */
>   ldr     r0, =_start
>   mcr     p15, 0, r0, c12, c0, 0  @Set VBAR
> + mrc p15, 0, r1, c0, c1, 1 @ check for security extension
> + ands r1, r1, #0x30
> + mcrne p15, 0, r0, c12, c0, 1  @ Set secure monitor MVBAR
>
>   bx lr
>
> @@ -492,9 +499,16 @@ undefined_instruction:
>
>   .align 5
>  software_interrupt:
> - get_bad_stack_swi
> - bad_save_user_regs
> - bl do_software_interrupt

I think you should add a comment block here saying that this is handling
an smc #0 call and the intent is to change to non-secure world.

> + mrc p15, 0, r1, c1, c1, 0 @ read SCR
> + bic r1, r1, #0x07f
> + orr r1, r1, #0x31 @ enable NS, AW, FW
> +
> + mrc p15, 0, r0, c12, c0, 0 @ save secure copy of VBAR
> + mcr p15, 0, r1, c1, c1, 0 @ write SCR, switch to non-sec
> + isb
> + mcr p15, 0, r0, c12, c0, 0 @ write non-secure copy of VBAR
> +
> + movs pc, lr
>
>   .align 5
>  prefetch_abort:
> --
> 1.7.12.1
>

[U-Boot] [RFC PATCH 4/6] ARM: add SMP support for non-secure switch

[Christoffer Dall]

On Fri, Apr 26, 2013 at 6:14 AM, Andre Przywara
<andre.przywara@linaro.org> wrote:
> Currently the non-secure switch is only done for the boot processor.
> To later allow full SMP support, we have to switch all secondary
> cores into non-secure state also.
>
> So we add an entry point for secondary CPUs coming out of low-power
> state and make sure we put them into WFI again after having switched
> to non-secure state.
> For this we acknowledge and EOI the wake-up IPI, then go into WFI.
> Once being kicked out of it later, we sanity check that the start
> address has actually been changed (since another attempt to switch
> to non-secure would block the core) and jump to the new address.
>
> The actual CPU kick is done by sending an inter-processor interrupt
> via the GIC to all CPU interfaces except the requesting processor.
> The secondary cores will then setup their respective GIC CPU
> interface.
>
> The address secondary cores jump to is board specific, we provide
> the value here for the Versatile Express board.
>
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>  arch/arm/cpu/armv7/start.S          | 27 ++++++++++++++++++++++++++-
>  arch/arm/lib/virt-v7.c              | 10 +++++++++-
>  include/configs/vexpress_ca15_tc2.h |  1 +
>  3 files changed, 36 insertions(+), 2 deletions(-)
>
> diff --git a/arch/arm/cpu/armv7/start.S b/arch/arm/cpu/armv7/start.S
> index 401b0eb..2b47881 100644
> --- a/arch/arm/cpu/armv7/start.S
> +++ b/arch/arm/cpu/armv7/start.S
> @@ -563,8 +563,19 @@ fiq:
>  #endif /* CONFIG_SPL_BUILD */
>
>  /* Routine to initialize GIC CPU interface and switch to nonsecure state.
> + * Will be executed directly by secondary CPUs after coming out of
> + * WFI, or can be called directly by C code for CPU 0.
> + * Those two paths mandate to not use any stack and to only use registers
> + * r0-r3 to comply with both the C ABI and the requirement of SMP startup
> + * code.
>   */
>  .globl _nonsec_gic_switch
> +.globl _smp_pen
> +_smp_pen:
> +       mrs     r0, cpsr
> +       orr     r0, r0, #0xc0
> +       msr     cpsr, r0                        @ disable interrupts
> +       mov     lr, #0                          @ clear LR to mark secondary
>  _nonsec_gic_switch:
>         mrc     p15, 4, r2, c15, c0, 0          @ r2 = PERIPHBASE
>         add     r3, r2, #0x1000                 @ GIC dist i/f offset
> @@ -605,4 +616,18 @@ _nonsec_gic_switch:
>         add     r2, r2, #0x1000                 @ GIC dist i/f offset
>         str     r1, [r2]                        @ allow private interrupts
>
> -       mov     pc, lr
> +       cmp     lr, #0
> +       movne   pc, lr                          @ CPU 0 to return
> +                                               @ all others: go to sleep
> +_ack_int:
> +       ldr     r1, [r3, #0x0c]                 @ read GICD acknowledge
> +       str     r1, [r3, #0x10]                 @ write GICD EOI
> +
> +       adr     r1, _smp_pen
> +waitloop:
> +       wfi
> +       ldr     r0, =CONFIG_SYSFLAGS_ADDR       @ load start address
> +       ldr     r0, [r0]
> +       cmp     r0, r1                  @ make sure we dont execute this code
> +       beq     waitloop                @ again (due to a spurious wakeup)
> +       mov     pc, r0
> diff --git a/arch/arm/lib/virt-v7.c b/arch/arm/lib/virt-v7.c
> index 416ca29..5ca093a 100644
> --- a/arch/arm/lib/virt-v7.c
> +++ b/arch/arm/lib/virt-v7.c
> @@ -29,6 +29,7 @@
>
>  /* the assembly routine doing the actual work in start.S */
>  void _nonsec_gic_switch(void);
> +void _smp_pen(void);
>
>  #define GICD_CTLR      0x000
>  #define GICD_TYPER     0x004
> @@ -51,6 +52,7 @@ int armv7_switch_nonsec(void)
>         unsigned int reg;
>         volatile unsigned int *gicdptr;
>         unsigned itlinesnr, i;
> +       unsigned int *sysflags;
>
>         /* check whether the CPU supports the security extensions */
>         asm("mrc p15, 0, %0, c0, c1, 1\n" : "=r"(reg));
> @@ -109,7 +111,13 @@ int armv7_switch_nonsec(void)
>         for (i = 0; i <= itlinesnr; i++)
>                 gicdptr[GICD_IGROUPR0 / 4 + i] = (unsigned)-1;
>
> -       /* call the non-sec switching code on this CPU */
> +       /* now kick all CPUs (expect this one) by writing to GICD_SIGR */
> +       sysflags = (void *)CONFIG_SYSFLAGS_ADDR;
> +       sysflags[1] = (unsigned)-1;
> +       sysflags[0] = (uintptr_t)_smp_pen;
> +       gicdptr[GICD_SGIR / 4] = 1U << 24;
> +
> +       /* call the non-sec switching code on this CPU also */
>         _nonsec_gic_switch();
>
>         return 0;
> diff --git a/include/configs/vexpress_ca15_tc2.h b/include/configs/vexpress_ca15_tc2.h
> index 9e230ad..210a27c 100644
> --- a/include/configs/vexpress_ca15_tc2.h
> +++ b/include/configs/vexpress_ca15_tc2.h
> @@ -32,5 +32,6 @@
>  #define CONFIG_BOOTP_VCI_STRING     "U-boot.armv7.vexpress_ca15x2_tc2"
>
>  #define CONFIG_SYS_CLK_FREQ 24000000
> +#define CONFIG_SYSFLAGS_ADDR 0x1c010030
>
>  #endif
> --


hmm, is all this likely to work across all armv7 cores?  I imagined
that the SMP boot protocol could be potentially vastly different on
different implementations and this stuff would have to call into board
specific code...?

[U-Boot] [RFC PATCH 4/6] ARM: add SMP support for non-secure switch

[Andre Przywara]

On 04/27/2013 12:13 AM, Christoffer Dall wrote:
> On Fri, Apr 26, 2013 at 6:14 AM, Andre Przywara
> <andre.przywara@linaro.org> wrote:
>> Currently the non-secure switch is only done for the boot processor.
>> To later allow full SMP support, we have to switch all secondary
>> cores into non-secure state also.
>>
>> So we add an entry point for secondary CPUs coming out of low-power
>> state and make sure we put them into WFI again after having switched
>> to non-secure state.
>> For this we acknowledge and EOI the wake-up IPI, then go into WFI.
>> Once being kicked out of it later, we sanity check that the start
>> address has actually been changed (since another attempt to switch
>> to non-secure would block the core) and jump to the new address.
>>
>> The actual CPU kick is done by sending an inter-processor interrupt
>> via the GIC to all CPU interfaces except the requesting processor.
>> The secondary cores will then setup their respective GIC CPU
>> interface.
>>
>> The address secondary cores jump to is board specific, we provide
>> the value here for the Versatile Express board.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   arch/arm/cpu/armv7/start.S          | 27 ++++++++++++++++++++++++++-
>>   arch/arm/lib/virt-v7.c              | 10 +++++++++-
>>   include/configs/vexpress_ca15_tc2.h |  1 +
>>   3 files changed, 36 insertions(+), 2 deletions(-)
>>
>> diff --git a/arch/arm/cpu/armv7/start.S b/arch/arm/cpu/armv7/start.S
>> index 401b0eb..2b47881 100644
>> --- a/arch/arm/cpu/armv7/start.S
>> +++ b/arch/arm/cpu/armv7/start.S
>> @@ -563,8 +563,19 @@ fiq:
>>   #endif /* CONFIG_SPL_BUILD */
>>
>>   /* Routine to initialize GIC CPU interface and switch to nonsecure state.
>> + * Will be executed directly by secondary CPUs after coming out of
>> + * WFI, or can be called directly by C code for CPU 0.
>> + * Those two paths mandate to not use any stack and to only use registers
>> + * r0-r3 to comply with both the C ABI and the requirement of SMP startup
>> + * code.
>>    */
>>   .globl _nonsec_gic_switch
>> +.globl _smp_pen
>> +_smp_pen:
>> +       mrs     r0, cpsr
>> +       orr     r0, r0, #0xc0
>> +       msr     cpsr, r0                        @ disable interrupts
>> +       mov     lr, #0                          @ clear LR to mark secondary
>>   _nonsec_gic_switch:
>>          mrc     p15, 4, r2, c15, c0, 0          @ r2 = PERIPHBASE
>>          add     r3, r2, #0x1000                 @ GIC dist i/f offset
>> @@ -605,4 +616,18 @@ _nonsec_gic_switch:
>>          add     r2, r2, #0x1000                 @ GIC dist i/f offset
>>          str     r1, [r2]                        @ allow private interrupts
>>
>> -       mov     pc, lr
>> +       cmp     lr, #0
>> +       movne   pc, lr                          @ CPU 0 to return
>> +                                               @ all others: go to sleep
>> +_ack_int:
>> +       ldr     r1, [r3, #0x0c]                 @ read GICD acknowledge
>> +       str     r1, [r3, #0x10]                 @ write GICD EOI
>> +
>> +       adr     r1, _smp_pen
>> +waitloop:
>> +       wfi
>> +       ldr     r0, =CONFIG_SYSFLAGS_ADDR       @ load start address
>> +       ldr     r0, [r0]
>> +       cmp     r0, r1                  @ make sure we dont execute this code
>> +       beq     waitloop                @ again (due to a spurious wakeup)
>> +       mov     pc, r0
>> diff --git a/arch/arm/lib/virt-v7.c b/arch/arm/lib/virt-v7.c
>> index 416ca29..5ca093a 100644
>> --- a/arch/arm/lib/virt-v7.c
>> +++ b/arch/arm/lib/virt-v7.c
>> @@ -29,6 +29,7 @@
>>
>>   /* the assembly routine doing the actual work in start.S */
>>   void _nonsec_gic_switch(void);
>> +void _smp_pen(void);
>>
>>   #define GICD_CTLR      0x000
>>   #define GICD_TYPER     0x004
>> @@ -51,6 +52,7 @@ int armv7_switch_nonsec(void)
>>          unsigned int reg;
>>          volatile unsigned int *gicdptr;
>>          unsigned itlinesnr, i;
>> +       unsigned int *sysflags;
>>
>>          /* check whether the CPU supports the security extensions */
>>          asm("mrc p15, 0, %0, c0, c1, 1\n" : "=r"(reg));
>> @@ -109,7 +111,13 @@ int armv7_switch_nonsec(void)
>>          for (i = 0; i <= itlinesnr; i++)
>>                  gicdptr[GICD_IGROUPR0 / 4 + i] = (unsigned)-1;
>>
>> -       /* call the non-sec switching code on this CPU */
>> +       /* now kick all CPUs (expect this one) by writing to GICD_SIGR */
>> +       sysflags = (void *)CONFIG_SYSFLAGS_ADDR;
>> +       sysflags[1] = (unsigned)-1;
>> +       sysflags[0] = (uintptr_t)_smp_pen;
>> +       gicdptr[GICD_SGIR / 4] = 1U << 24;
>> +
>> +       /* call the non-sec switching code on this CPU also */
>>          _nonsec_gic_switch();
>>
>>          return 0;
>> diff --git a/include/configs/vexpress_ca15_tc2.h b/include/configs/vexpress_ca15_tc2.h
>> index 9e230ad..210a27c 100644
>> --- a/include/configs/vexpress_ca15_tc2.h
>> +++ b/include/configs/vexpress_ca15_tc2.h
>> @@ -32,5 +32,6 @@
>>   #define CONFIG_BOOTP_VCI_STRING     "U-boot.armv7.vexpress_ca15x2_tc2"
>>
>>   #define CONFIG_SYS_CLK_FREQ 24000000
>> +#define CONFIG_SYSFLAGS_ADDR 0x1c010030
>>
>>   #endif
>> --
>
>
> hmm, is all this likely to work across all armv7 cores?

Probably not.

> I imagined that the SMP boot protocol could be potentially vastly different on
> different implementations and this stuff would have to call into board
> specific code...?

Right. For now I am trying to get this code upstream. Support for other 
boards can then be merged in on demand. Looks like at least the Arndale 
board can be kicked very similarly, only the SYSFLAGS address is 
different. I am willing to re-architecture the SMP code as needed.

Regards,
Andre.

[U-Boot] [RFC PATCH 4/6] ARM: add SMP support for non-secure switch

[Christoffer Dall]

On Mon, May 06, 2013 at 03:19:40PM +0200, Andre Przywara wrote:
> On 04/27/2013 12:13 AM, Christoffer Dall wrote:
> >On Fri, Apr 26, 2013 at 6:14 AM, Andre Przywara
> ><andre.przywara@linaro.org> wrote:
> >>Currently the non-secure switch is only done for the boot processor.
> >>To later allow full SMP support, we have to switch all secondary
> >>cores into non-secure state also.
> >>
> >>So we add an entry point for secondary CPUs coming out of low-power
> >>state and make sure we put them into WFI again after having switched
> >>to non-secure state.
> >>For this we acknowledge and EOI the wake-up IPI, then go into WFI.
> >>Once being kicked out of it later, we sanity check that the start
> >>address has actually been changed (since another attempt to switch
> >>to non-secure would block the core) and jump to the new address.
> >>
> >>The actual CPU kick is done by sending an inter-processor interrupt
> >>via the GIC to all CPU interfaces except the requesting processor.
> >>The secondary cores will then setup their respective GIC CPU
> >>interface.
> >>
> >>The address secondary cores jump to is board specific, we provide
> >>the value here for the Versatile Express board.
> >>
> >>Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> >>---
> >>  arch/arm/cpu/armv7/start.S          | 27 ++++++++++++++++++++++++++-
> >>  arch/arm/lib/virt-v7.c              | 10 +++++++++-
> >>  include/configs/vexpress_ca15_tc2.h |  1 +
> >>  3 files changed, 36 insertions(+), 2 deletions(-)
> >>
> >>diff --git a/arch/arm/cpu/armv7/start.S b/arch/arm/cpu/armv7/start.S
> >>index 401b0eb..2b47881 100644
> >>--- a/arch/arm/cpu/armv7/start.S
> >>+++ b/arch/arm/cpu/armv7/start.S
> >>@@ -563,8 +563,19 @@ fiq:
> >>  #endif /* CONFIG_SPL_BUILD */
> >>
> >>  /* Routine to initialize GIC CPU interface and switch to nonsecure state.
> >>+ * Will be executed directly by secondary CPUs after coming out of
> >>+ * WFI, or can be called directly by C code for CPU 0.
> >>+ * Those two paths mandate to not use any stack and to only use registers
> >>+ * r0-r3 to comply with both the C ABI and the requirement of SMP startup
> >>+ * code.
> >>   */
> >>  .globl _nonsec_gic_switch
> >>+.globl _smp_pen
> >>+_smp_pen:
> >>+       mrs     r0, cpsr
> >>+       orr     r0, r0, #0xc0
> >>+       msr     cpsr, r0                        @ disable interrupts
> >>+       mov     lr, #0                          @ clear LR to mark secondary
> >>  _nonsec_gic_switch:
> >>         mrc     p15, 4, r2, c15, c0, 0          @ r2 = PERIPHBASE
> >>         add     r3, r2, #0x1000                 @ GIC dist i/f offset
> >>@@ -605,4 +616,18 @@ _nonsec_gic_switch:
> >>         add     r2, r2, #0x1000                 @ GIC dist i/f offset
> >>         str     r1, [r2]                        @ allow private interrupts
> >>
> >>-       mov     pc, lr
> >>+       cmp     lr, #0
> >>+       movne   pc, lr                          @ CPU 0 to return
> >>+                                               @ all others: go to sleep
> >>+_ack_int:
> >>+       ldr     r1, [r3, #0x0c]                 @ read GICD acknowledge
> >>+       str     r1, [r3, #0x10]                 @ write GICD EOI
> >>+
> >>+       adr     r1, _smp_pen
> >>+waitloop:
> >>+       wfi
> >>+       ldr     r0, =CONFIG_SYSFLAGS_ADDR       @ load start address
> >>+       ldr     r0, [r0]
> >>+       cmp     r0, r1                  @ make sure we dont execute this code
> >>+       beq     waitloop                @ again (due to a spurious wakeup)
> >>+       mov     pc, r0
> >>diff --git a/arch/arm/lib/virt-v7.c b/arch/arm/lib/virt-v7.c
> >>index 416ca29..5ca093a 100644
> >>--- a/arch/arm/lib/virt-v7.c
> >>+++ b/arch/arm/lib/virt-v7.c
> >>@@ -29,6 +29,7 @@
> >>
> >>  /* the assembly routine doing the actual work in start.S */
> >>  void _nonsec_gic_switch(void);
> >>+void _smp_pen(void);
> >>
> >>  #define GICD_CTLR      0x000
> >>  #define GICD_TYPER     0x004
> >>@@ -51,6 +52,7 @@ int armv7_switch_nonsec(void)
> >>         unsigned int reg;
> >>         volatile unsigned int *gicdptr;
> >>         unsigned itlinesnr, i;
> >>+       unsigned int *sysflags;
> >>
> >>         /* check whether the CPU supports the security extensions */
> >>         asm("mrc p15, 0, %0, c0, c1, 1\n" : "=r"(reg));
> >>@@ -109,7 +111,13 @@ int armv7_switch_nonsec(void)
> >>         for (i = 0; i <= itlinesnr; i++)
> >>                 gicdptr[GICD_IGROUPR0 / 4 + i] = (unsigned)-1;
> >>
> >>-       /* call the non-sec switching code on this CPU */
> >>+       /* now kick all CPUs (expect this one) by writing to GICD_SIGR */
> >>+       sysflags = (void *)CONFIG_SYSFLAGS_ADDR;
> >>+       sysflags[1] = (unsigned)-1;
> >>+       sysflags[0] = (uintptr_t)_smp_pen;
> >>+       gicdptr[GICD_SGIR / 4] = 1U << 24;
> >>+
> >>+       /* call the non-sec switching code on this CPU also */
> >>         _nonsec_gic_switch();
> >>
> >>         return 0;
> >>diff --git a/include/configs/vexpress_ca15_tc2.h b/include/configs/vexpress_ca15_tc2.h
> >>index 9e230ad..210a27c 100644
> >>--- a/include/configs/vexpress_ca15_tc2.h
> >>+++ b/include/configs/vexpress_ca15_tc2.h
> >>@@ -32,5 +32,6 @@
> >>  #define CONFIG_BOOTP_VCI_STRING     "U-boot.armv7.vexpress_ca15x2_tc2"
> >>
> >>  #define CONFIG_SYS_CLK_FREQ 24000000
> >>+#define CONFIG_SYSFLAGS_ADDR 0x1c010030
> >>
> >>  #endif
> >>--
> >
> >
> >hmm, is all this likely to work across all armv7 cores?
> 
> Probably not.
> 
> >I imagined that the SMP boot protocol could be potentially vastly different on
> >different implementations and this stuff would have to call into board
> >specific code...?
> 
> Right. For now I am trying to get this code upstream. Support for
> other boards can then be merged in on demand. Looks like at least
> the Arndale board can be kicked very similarly, only the SYSFLAGS
> address is different. I am willing to re-architecture the SMP code
> as needed.
> 
Sorry for the late response, I've been out sick.

So, I'm not very familiar with conventions and practices for the U-boot
project, but it seems quite clear to me that since these changes go in
arch/arm/lib/... paths those changes must be generic.

Therefore, I'm pretty sure that you need to factor out anything that's
specific to vexpress into a part of the code that's specific to vexpress
and thereby establish a board interface that generic code can call into
for other boards too, for example Arndale; in fact it would be cool if
you could show this working for both platforms at the same time, since
the working code is already available.

-Christoffer