[PATCH v2 0/7] Allow the kernel to be booted in HYP mode

[PATCH v2 0/7] Allow the kernel to be booted in HYP mode

[Marc Zyngier]

This patch series allows the kernel (and the decompressor) to be
entered in HYP mode, which is a requirement for supporting hypervisors
such as KVM.

The main idea is to have a stub hypervisor to go back and forth
between HYP and SVC modes, and let the kernel install a full blown
hypervisor if necessary. Also, we ensure that a kernel without virt
support can still be entered in HYP mode and safely go back to SVC.

Most of the heavy lifting has been done by Dave Martin, but all bugs
have been introduced by me while reworking it. This series also relies
on Dave's earlier opcode injection series :
http://comments.gmane.org/gmane.linux.ports.arm.kernel/180901

It's been tested on a variety of platforms (VE A9/A15, Fast Model)
with all combinaisons of entering in HYP or SVC, ARM or Thumb2
kernels.

Patches against 3.6-rc4, and available at:
git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git hyp-mode-boot-next

* From v1:
  - Fix undefined behaviour when initializing HDCR

Dave Martin (5):
  ARM: virt: allow the kernel to be entered in HYP mode
  ARM: zImage/virt: hyp mode entry support for the zImage loader
  ARM: virt: Update documentation for hyp mode entry support
  ARM: virt: Add boot-time diagnostics
  ARM: virt: Add CONFIG_ARM_VIRT_EXT option

Marc Zyngier (2):
  ARM: opcodes: add __ERET/__MSR_ELR_HYP instruction encoding
  ARM: virt: arch_timers: enable access to physical timers

 Documentation/arm/Booting           |  22 +++-
 arch/arm/boot/compressed/.gitignore |   1 +
 arch/arm/boot/compressed/Makefile   |   9 +-
 arch/arm/boot/compressed/head.S     |  71 ++++++++++--
 arch/arm/include/asm/assembler.h    |  28 +++++
 arch/arm/include/asm/opcodes-virt.h |  10 ++
 arch/arm/include/asm/ptrace.h       |   1 +
 arch/arm/include/asm/virt.h         |  69 +++++++++++
 arch/arm/kernel/Makefile            |   2 +
 arch/arm/kernel/head.S              |  14 ++-
 arch/arm/kernel/hyp-stub.S          | 223 ++++++++++++++++++++++++++++++++++++
 arch/arm/kernel/setup.c             |  20 ++++
 arch/arm/kernel/smp.c               |   3 +
 arch/arm/mm/Kconfig                 |  17 +++
 14 files changed, 478 insertions(+), 12 deletions(-)
 create mode 100644 arch/arm/include/asm/virt.h
 create mode 100644 arch/arm/kernel/hyp-stub.S

-- 
1.7.12

[PATCH v2 1/7] ARM: opcodes: add __ERET/__MSR_ELR_HYP instruction encoding

[Marc Zyngier]

Enabling boot from HYP mode requires the use of some more
virt-specific instructions ("eret" and "msr elr_hyp, reg").

Add the necessary encoding to asm/opcode-virt.h.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm/include/asm/opcodes-virt.h | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/arm/include/asm/opcodes-virt.h b/arch/arm/include/asm/opcodes-virt.h
index b85665a..efcfdf9 100644
--- a/arch/arm/include/asm/opcodes-virt.h
+++ b/arch/arm/include/asm/opcodes-virt.h
@@ -26,4 +26,14 @@
 	0xF7E08000 | (((imm16) & 0xF000) << 4) | ((imm16) & 0x0FFF)	\
 )
 
+#define __ERET	__inst_arm_thumb32(					\
+	0xE160006E,							\
+	0xF3DE8F00							\
+)
+
+#define __MSR_ELR_HYP(regnum)	__inst_arm_thumb32(			\
+	0xE12EF300 | regnum,						\
+	0xF3808E30 | (regnum << 16)					\
+)
+
 #endif /* ! __ASM_ARM_OPCODES_VIRT_H */
-- 
1.7.12

[PATCH v2 2/7] ARM: virt: allow the kernel to be entered in HYP mode

[Marc Zyngier]

From: Dave Martin <dave.martin@linaro.org>

This patch does two things:

  * Ensure that asynchronous aborts are masked at kernel entry.
    The bootloader should be masking these anyway, but this reduces
    the damage window just in case it doesn't.

  * Enter svc mode via exception return to ensure that CPU state is
    properly serialised.  This does not matter when switching from
    an ordinary privileged mode ("PL1" modes in ARMv7-AR rev C
    parlance), but it potentially does matter when switching from a
    another privileged mode such as hyp mode.

This should allow the kernel to boot safely either from svc mode or
hyp mode, even if no support for use of the ARM Virtualization
Extensions is built into the kernel.

Signed-off-by: Dave Martin <dave.martin@linaro.org>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm/include/asm/assembler.h |  28 ++++++
 arch/arm/include/asm/ptrace.h    |   1 +
 arch/arm/include/asm/virt.h      |  52 +++++++++++
 arch/arm/kernel/Makefile         |   2 +
 arch/arm/kernel/head.S           |  14 ++-
 arch/arm/kernel/hyp-stub.S       | 192 +++++++++++++++++++++++++++++++++++++++
 6 files changed, 286 insertions(+), 3 deletions(-)
 create mode 100644 arch/arm/include/asm/virt.h
 create mode 100644 arch/arm/kernel/hyp-stub.S

diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h
index 03fb936..658a15d 100644
--- a/arch/arm/include/asm/assembler.h
+++ b/arch/arm/include/asm/assembler.h
@@ -22,6 +22,7 @@
 
 #include <asm/ptrace.h>
 #include <asm/domain.h>
+#include <asm/opcodes-virt.h>
 
 #define IOMEM(x)	(x)
 
@@ -240,6 +241,33 @@
 #endif
 
 /*
+ * Helper macro to enter SVC mode cleanly and mask interrupts. reg is
+ * a scratch register for the macro to overwrite.
+ *
+ * This macro is intended for forcing the CPU into SVC mode at boot time.
+ * you cannot return to the original mode.
+ *
+ * Beware, it also clobers LR.
+ */
+.macro safe_svcmode_maskall reg:req
+	mrs	\reg , cpsr
+	mov	lr , \reg
+	and	lr , lr , #MODE_MASK
+	cmp	lr , #HYP_MODE
+	orr	\reg , \reg , #PSR_A_BIT | PSR_I_BIT | PSR_F_BIT
+	bic	\reg , \reg , #MODE_MASK
+	orr	\reg , \reg , #SVC_MODE
+THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
+	msr	spsr_cxsf, \reg
+	adr	lr, BSYM(2f)
+	bne	1f
+	__MSR_ELR_HYP(14)
+	__ERET
+1:	movs	pc, lr
+2:
+.endm
+
+/*
  * STRT/LDRT access macros with ARM and Thumb-2 variants
  */
 #ifdef CONFIG_THUMB2_KERNEL
diff --git a/arch/arm/include/asm/ptrace.h b/arch/arm/include/asm/ptrace.h
index 355ece5..91ef6c2 100644
--- a/arch/arm/include/asm/ptrace.h
+++ b/arch/arm/include/asm/ptrace.h
@@ -44,6 +44,7 @@
 #define IRQ_MODE	0x00000012
 #define SVC_MODE	0x00000013
 #define ABT_MODE	0x00000017
+#define HYP_MODE	0x0000001a
 #define UND_MODE	0x0000001b
 #define SYSTEM_MODE	0x0000001f
 #define MODE32_BIT	0x00000010
diff --git a/arch/arm/include/asm/virt.h b/arch/arm/include/asm/virt.h
new file mode 100644
index 0000000..0a99723
--- /dev/null
+++ b/arch/arm/include/asm/virt.h
@@ -0,0 +1,52 @@
+/*
+ * Copyright (c) 2012 Linaro Limited.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+ */
+
+#ifndef VIRT_H
+#define VIRT_H
+
+#include <asm/ptrace.h>
+
+/*
+ * Flag indicating that the kernel was not entered in the same mode on every
+ * CPU.  The zImage loader stashes this value in an SPSR, so we need an
+ * architecturally defined flag bit here (the N flag, as it happens)
+ */
+#define BOOT_CPU_MODE_MISMATCH (1<<31)
+
+#ifndef __ASSEMBLY__
+
+#ifdef CONFIG_ARM_VIRT_EXT
+/*
+ * __boot_cpu_mode records what mode the primary CPU was booted in.
+ * A correctly-implemented bootloader must start all CPUs in the same mode:
+ * if it fails to do this, the flag BOOT_CPU_MODE_MISMATCH is set to indicate
+ * that some CPU(s) were booted in a different mode.
+ *
+ * This allows the kernel to flag an error when the secondaries have come up.
+ */
+extern int __boot_cpu_mode;
+
+void __hyp_set_vectors(unsigned long phys_vector_base);
+unsigned long __hyp_get_vectors(void);
+#else
+#define __boot_cpu_mode	(SVC_MODE)
+#endif
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* ! VIRT_H */
diff --git a/arch/arm/kernel/Makefile b/arch/arm/kernel/Makefile
index 7ad2d5c..49b61a3 100644
--- a/arch/arm/kernel/Makefile
+++ b/arch/arm/kernel/Makefile
@@ -82,4 +82,6 @@ head-y			:= head$(MMUEXT).o
 obj-$(CONFIG_DEBUG_LL)	+= debug.o
 obj-$(CONFIG_EARLY_PRINTK)	+= early_printk.o
 
+obj-$(CONFIG_ARM_VIRT_EXT)	+= hyp-stub.o
+
 extra-y := $(head-y) vmlinux.lds
diff --git a/arch/arm/kernel/head.S b/arch/arm/kernel/head.S
index 3db960e..27093e4 100644
--- a/arch/arm/kernel/head.S
+++ b/arch/arm/kernel/head.S
@@ -83,8 +83,12 @@ ENTRY(stext)
  THUMB(	.thumb			)	@ switch to Thumb now.
  THUMB(1:			)
 
-	setmode	PSR_F_BIT | PSR_I_BIT | SVC_MODE, r9 @ ensure svc mode
-						@ and irqs disabled
+#ifdef CONFIG_ARM_VIRT_EXT
+	bl	__hyp_stub_install
+#endif
+	@ ensure svc mode and all interrupts masked
+	safe_svcmode_maskall r9
+
 	mrc	p15, 0, r9, c0, c0		@ get processor id
 	bl	__lookup_processor_type		@ r5=procinfo r9=cpuid
 	movs	r10, r5				@ invalid processor (r5=0)?
@@ -326,7 +330,11 @@ ENTRY(secondary_startup)
 	 * the processor type - there is no need to check the machine type
 	 * as it has already been validated by the primary processor.
 	 */
-	setmode	PSR_F_BIT | PSR_I_BIT | SVC_MODE, r9
+#ifdef CONFIG_ARM_VIRT_EXT
+	bl	__hyp_stub_install
+#endif
+	safe_svcmode_maskall r9
+
 	mrc	p15, 0, r9, c0, c0		@ get processor id
 	bl	__lookup_processor_type
 	movs	r10, r5				@ invalid processor?
diff --git a/arch/arm/kernel/hyp-stub.S b/arch/arm/kernel/hyp-stub.S
new file mode 100644
index 0000000..b03e924
--- /dev/null
+++ b/arch/arm/kernel/hyp-stub.S
@@ -0,0 +1,192 @@
+/*
+ * Copyright (c) 2012 Linaro Limited.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+ */
+
+#include <linux/init.h>
+#include <linux/linkage.h>
+#include <asm/assembler.h>
+#include <asm/virt.h>
+
+/*
+ * For the kernel proper, we need to find out the CPU boot mode long after
+ * boot, so we need to store it in a writable variable.
+ *
+ * This is not in .bss, because we set it sufficiently early that the boot-time
+ * zeroing of .bss would clobber it.
+ */
+.data
+ENTRY(__boot_cpu_mode)
+	.long	0
+.text
+
+	/*
+	 * Save the primary CPU boot mode. Requires 3 scratch registers.
+	 */
+	.macro	store_primary_cpu_mode	reg1, reg2, reg3
+	mrs	\reg1, cpsr
+	and	\reg1, \reg1, #MODE_MASK
+	adr	\reg2, .L__boot_cpu_mode_offset
+	ldr	\reg3, [\reg2]
+	str	\reg1, [\reg2, \reg3]
+	.endm
+
+	/*
+	 * Compare the current mode with the one saved on the primary CPU.
+	 * If they don't match, record that fact. The Z bit indicates
+	 * if there's a match or not.
+	 * Requires 3 additionnal scratch registers.
+	 */
+	.macro	compare_cpu_mode_with_primary mode, reg1, reg2, reg3
+	adr	\reg2, .L__boot_cpu_mode_offset
+	ldr	\reg3, [\reg2]
+	ldr	\reg1, [\reg2, \reg3]
+	cmp	\mode, \reg1		@ matches primary CPU boot mode?
+	orrne	r7, r7, #BOOT_CPU_MODE_MISMATCH
+	strne	r7, [r5, r6]		@ record what happened and give up
+	.endm
+
+/*
+ * Hypervisor stub installation functions.
+ *
+ * These must be called with the MMU and D-cache off.
+ * They are not ABI compliant and are only intended to be called from the kernel
+ * entry points in head.S.
+ */
+@ Call this from the primary CPU
+ENTRY(__hyp_stub_install)
+	store_primary_cpu_mode	r4, r5, r6
+ENDPROC(__hyp_stub_install)
+
+	@ fall through...
+
+@ Secondary CPUs should call here
+ENTRY(__hyp_stub_install_secondary)
+	mrs	r4, cpsr
+	and	r4, r4, #MODE_MASK
+
+	/*
+	 * If the secondary has booted with a different mode, give up
+	 * immediately.
+	 */
+	compare_cpu_mode_with_primary	r4, r5, r6, r7
+	bxne	lr
+
+	/*
+	 * Once we have given up on one CPU, we do not try to install the
+	 * stub hypervisor on the remaining ones: because the saved boot mode
+	 * is modified, it can't compare equal to the CPSR mode field any
+	 * more.
+	 *
+	 * Otherwise...
+	 */
+
+	cmp	r4, #HYP_MODE
+	bxne	lr			@ give up if the CPU is not in HYP mode
+
+/*
+ * Configure HSCTLR to set correct exception endianness/instruction set
+ * state etc.
+ * Turn off all traps
+ * Eventually, CPU-specific code might be needed -- assume not for now
+ *
+ * This code relies on the "eret" instruction to synchronize the
+ * various coprocessor accesses.
+ */
+	@ Now install the hypervisor stub:
+	adr	r7, __hyp_stub_vectors
+	mcr	p15, 4, r7, c12, c0, 0	@ set hypervisor vector base (HVBAR)
+
+	@ Disable all traps, so we don't get any nasty surprise
+	mov	r7, #0
+	mcr	p15, 4, r7, c1, c1, 0	@ HCR
+	mcr	p15, 4, r7, c1, c1, 2	@ HCPTR
+	mcr	p15, 4, r7, c1, c1, 3	@ HSTR
+
+THUMB(	orr	r7, #(1 << 30)	)	@ HSCTLR.TE
+#ifdef CONFIG_CPU_BIG_ENDIAN
+	orr	r7, #(1 << 9)		@ HSCTLR.EE
+#endif
+	mcr	p15, 4, r7, c1, c0, 0	@ HSCTLR
+
+	mrc	p15, 4, r7, c1, c1, 1	@ HDCR
+	and	r7, #0x1f		@ Preserve HPMN
+	mcr	p15, 4, r7, c1, c1, 1	@ HDCR
+
+	bic	r7, r4, #MODE_MASK
+	orr	r7, r7, #SVC_MODE
+THUMB(	orr	r7, r7, #PSR_T_BIT	)
+	msr	spsr_cxsf, r7		@ This is SPSR_hyp.
+
+	__MSR_ELR_HYP(14)		@ msr elr_hyp, lr
+	__ERET				@ return, switching to SVC mode
+					@ The boot CPU mode is left in r4.
+ENDPROC(__hyp_stub_install_secondary)
+
+__hyp_stub_do_trap:
+	cmp	r0, #-1
+	mrceq	p15, 4, r0, c12, c0, 0	@ get HVBAR
+	mcrne	p15, 4, r0, c12, c0, 0	@ set HVBAR
+	__ERET
+ENDPROC(__hyp_stub_do_trap)
+
+/*
+ * __hyp_set_vectors: Call this after boot to set the initial hypervisor
+ * vectors as part of hypervisor installation.  On an SMP system, this should
+ * be called on each CPU.
+ *
+ * r0 must be the physical address of the new vector table (which must lie in
+ * the bottom 4GB of physical address space.
+ *
+ * r0 must be 32-byte aligned.
+ *
+ * Before calling this, you must check that the stub hypervisor is installed
+ * everywhere, by waiting for any secondary CPUs to be brought up and then
+ * checking that BOOT_CPU_MODE_HAVE_HYP(__boot_cpu_mode) is true.
+ *
+ * If not, there is a pre-existing hypervisor, some CPUs failed to boot, or
+ * something else went wrong... in such cases, trying to install a new
+ * hypervisor is unlikely to work as desired.
+ *
+ * When you call into your shiny new hypervisor, sp_hyp will contain junk,
+ * so you will need to set that to something sensible@the new hypervisor's
+ * initialisation entry point.
+ */
+ENTRY(__hyp_get_vectors)
+	mov	r0, #-1
+ENDPROC(__hyp_get_vectors)
+	@ fall through
+ENTRY(__hyp_set_vectors)
+	__HVC(0)
+	bx	lr
+ENDPROC(__hyp_set_vectors)
+
+.align 2
+.L__boot_cpu_mode_offset:
+	.long	__boot_cpu_mode - .
+
+.align 5
+__hyp_stub_vectors:
+__hyp_stub_reset:	W(b)	.
+__hyp_stub_und:		W(b)	.
+__hyp_stub_svc:		W(b)	.
+__hyp_stub_pabort:	W(b)	.
+__hyp_stub_dabort:	W(b)	.
+__hyp_stub_trap:	W(b)	__hyp_stub_do_trap
+__hyp_stub_irq:		W(b)	.
+__hyp_stub_fiq:		W(b)	.
+ENDPROC(__hyp_stub_vectors)
+
-- 
1.7.12

[PATCH v2 2/7] ARM: virt: allow the kernel to be entered in HYP mode

[Tony Lindgren]

Hi,

* Marc Zyngier <marc.zyngier@arm.com> [120907 10:04]:
> From: Dave Martin <dave.martin@linaro.org>
> 
> This patch does two things:
> 
>   * Ensure that asynchronous aborts are masked at kernel entry.
>     The bootloader should be masking these anyway, but this reduces
>     the damage window just in case it doesn't.
> 
>   * Enter svc mode via exception return to ensure that CPU state is
>     properly serialised.  This does not matter when switching from
>     an ordinary privileged mode ("PL1" modes in ARMv7-AR rev C
>     parlance), but it potentially does matter when switching from a
>     another privileged mode such as hyp mode.
> 
> This should allow the kernel to boot safely either from svc mode or
> hyp mode, even if no support for use of the ARM Virtualization
> Extensions is built into the kernel.
> 
> Signed-off-by: Dave Martin <dave.martin@linaro.org>
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>

Just bisected this down in linux-next for breaking booting of
my omap2420 ARMv6 based n8x0..

> --- a/arch/arm/kernel/head.S
> +++ b/arch/arm/kernel/head.S
> @@ -83,8 +83,12 @@ ENTRY(stext)
>   THUMB(	.thumb			)	@ switch to Thumb now.
>   THUMB(1:			)
>  
> -	setmode	PSR_F_BIT | PSR_I_BIT | SVC_MODE, r9 @ ensure svc mode
> -						@ and irqs disabled
> +#ifdef CONFIG_ARM_VIRT_EXT
> +	bl	__hyp_stub_install
> +#endif
> +	@ ensure svc mode and all interrupts masked
> +	safe_svcmode_maskall r9
> +
>  	mrc	p15, 0, r9, c0, c0		@ get processor id
>  	bl	__lookup_processor_type		@ r5=procinfo r9=cpuid
>  	movs	r10, r5				@ invalid processor (r5=0)?

..and looks like undoing this part fixes it. Any ideas?

I quickly tried disabling ARCH_OMAP3 and ARCH_OMAP4 so it's
ARMv6 but that does not help.

Regards,

Tony

[PATCH v2 2/7] ARM: virt: allow the kernel to be entered in HYP mode

[Russell King - ARM Linux]

On Fri, Oct 05, 2012 at 01:08:22PM -0700, Tony Lindgren wrote:
> Just bisected this down in linux-next for breaking booting of
> my omap2420 ARMv6 based n8x0..
> 
> > --- a/arch/arm/kernel/head.S
> > +++ b/arch/arm/kernel/head.S
> > @@ -83,8 +83,12 @@ ENTRY(stext)
> >   THUMB(	.thumb			)	@ switch to Thumb now.
> >   THUMB(1:			)
> >  
> > -	setmode	PSR_F_BIT | PSR_I_BIT | SVC_MODE, r9 @ ensure svc mode
> > -						@ and irqs disabled
> > +#ifdef CONFIG_ARM_VIRT_EXT
> > +	bl	__hyp_stub_install
> > +#endif
> > +	@ ensure svc mode and all interrupts masked
> > +	safe_svcmode_maskall r9
> > +
> >  	mrc	p15, 0, r9, c0, c0		@ get processor id
> >  	bl	__lookup_processor_type		@ r5=procinfo r9=cpuid
> >  	movs	r10, r5				@ invalid processor (r5=0)?
> 
> ..and looks like undoing this part fixes it. Any ideas?
> 
> I quickly tried disabling ARCH_OMAP3 and ARCH_OMAP4 so it's
> ARMv6 but that does not help.

You really should Cc me when you hit something like this.  I was
thinking about sending my tree (which contains these changes) this
evening but if they're breaking stuff, I'd prefer to delay that
stuff at least for a while.

[PATCH v2 2/7] ARM: virt: allow the kernel to be entered in HYP mode

[Tony Lindgren]

* Russell King - ARM Linux <linux@arm.linux.org.uk> [121005 16:10]:
> On Fri, Oct 05, 2012 at 01:08:22PM -0700, Tony Lindgren wrote:
> > Just bisected this down in linux-next for breaking booting of
> > my omap2420 ARMv6 based n8x0..
> > 
> > > --- a/arch/arm/kernel/head.S
> > > +++ b/arch/arm/kernel/head.S
> > > @@ -83,8 +83,12 @@ ENTRY(stext)
> > >   THUMB(	.thumb			)	@ switch to Thumb now.
> > >   THUMB(1:			)
> > >  
> > > -	setmode	PSR_F_BIT | PSR_I_BIT | SVC_MODE, r9 @ ensure svc mode
> > > -						@ and irqs disabled
> > > +#ifdef CONFIG_ARM_VIRT_EXT
> > > +	bl	__hyp_stub_install
> > > +#endif
> > > +	@ ensure svc mode and all interrupts masked
> > > +	safe_svcmode_maskall r9
> > > +
> > >  	mrc	p15, 0, r9, c0, c0		@ get processor id
> > >  	bl	__lookup_processor_type		@ r5=procinfo r9=cpuid
> > >  	movs	r10, r5				@ invalid processor (r5=0)?
> > 
> > ..and looks like undoing this part fixes it. Any ideas?
> > 
> > I quickly tried disabling ARCH_OMAP3 and ARCH_OMAP4 so it's
> > ARMv6 but that does not help.
> 
> You really should Cc me when you hit something like this.  I was
> thinking about sending my tree (which contains these changes) this
> evening but if they're breaking stuff, I'd prefer to delay that
> stuff at least for a while.

Sorry was planning to cc you for sure but forgot. Got any ideas
what to try to fix this?

Tony

[PATCH v2 2/7] ARM: virt: allow the kernel to be entered in HYP mode

[Tony Lindgren]

* Tony Lindgren <tony@atomide.com> [121005 16:27]:
> * Russell King - ARM Linux <linux@arm.linux.org.uk> [121005 16:10]:
> > On Fri, Oct 05, 2012 at 01:08:22PM -0700, Tony Lindgren wrote:
> > > Just bisected this down in linux-next for breaking booting of
> > > my omap2420 ARMv6 based n8x0..
> > > 
> > > > --- a/arch/arm/kernel/head.S
> > > > +++ b/arch/arm/kernel/head.S
> > > > @@ -83,8 +83,12 @@ ENTRY(stext)
> > > >   THUMB(	.thumb			)	@ switch to Thumb now.
> > > >   THUMB(1:			)
> > > >  
> > > > -	setmode	PSR_F_BIT | PSR_I_BIT | SVC_MODE, r9 @ ensure svc mode
> > > > -						@ and irqs disabled
> > > > +#ifdef CONFIG_ARM_VIRT_EXT
> > > > +	bl	__hyp_stub_install
> > > > +#endif
> > > > +	@ ensure svc mode and all interrupts masked
> > > > +	safe_svcmode_maskall r9
> > > > +
> > > >  	mrc	p15, 0, r9, c0, c0		@ get processor id
> > > >  	bl	__lookup_processor_type		@ r5=procinfo r9=cpuid
> > > >  	movs	r10, r5				@ invalid processor (r5=0)?
> > > 
> > > ..and looks like undoing this part fixes it. Any ideas?
> > > 
> > > I quickly tried disabling ARCH_OMAP3 and ARCH_OMAP4 so it's
> > > ARMv6 but that does not help.

The same kernel boots on 2430sdp, which is the same ARMv6 core
as 2430 if I remember correctly. So this hints that it has something
to do with the bits set differently by the bootloader?

Regards,

Tony

[PATCH v2 2/7] ARM: virt: allow the kernel to be entered in HYP mode

[Nicolas Pitre]

On Fri, 5 Oct 2012, Tony Lindgren wrote:

> * Tony Lindgren <tony@atomide.com> [121005 16:27]:
> > * Russell King - ARM Linux <linux@arm.linux.org.uk> [121005 16:10]:
> > > On Fri, Oct 05, 2012 at 01:08:22PM -0700, Tony Lindgren wrote:
> > > > Just bisected this down in linux-next for breaking booting of
> > > > my omap2420 ARMv6 based n8x0..
> > > > 
> > > > > --- a/arch/arm/kernel/head.S
> > > > > +++ b/arch/arm/kernel/head.S
> > > > > @@ -83,8 +83,12 @@ ENTRY(stext)
> > > > >   THUMB(	.thumb			)	@ switch to Thumb now.
> > > > >   THUMB(1:			)
> > > > >  
> > > > > -	setmode	PSR_F_BIT | PSR_I_BIT | SVC_MODE, r9 @ ensure svc mode
> > > > > -						@ and irqs disabled
> > > > > +#ifdef CONFIG_ARM_VIRT_EXT
> > > > > +	bl	__hyp_stub_install
> > > > > +#endif
> > > > > +	@ ensure svc mode and all interrupts masked
> > > > > +	safe_svcmode_maskall r9
> > > > > +
> > > > >  	mrc	p15, 0, r9, c0, c0		@ get processor id
> > > > >  	bl	__lookup_processor_type		@ r5=procinfo r9=cpuid
> > > > >  	movs	r10, r5				@ invalid processor (r5=0)?
> > > > 
> > > > ..and looks like undoing this part fixes it. Any ideas?
> > > > 
> > > > I quickly tried disabling ARCH_OMAP3 and ARCH_OMAP4 so it's
> > > > ARMv6 but that does not help.
> 
> The same kernel boots on 2430sdp, which is the same ARMv6 core
> as 2430 if I remember correctly. So this hints that it has something
> to do with the bits set differently by the bootloader?

Possibly.

What if you apply this on top:

diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h
index 683a1e6b60..b276c26e19 100644
--- a/arch/arm/include/asm/assembler.h
+++ b/arch/arm/include/asm/assembler.h
@@ -254,8 +254,7 @@
 	mov	lr , \reg
 	and	lr , lr , #MODE_MASK
 	cmp	lr , #HYP_MODE
-	orr	\reg , \reg , #PSR_A_BIT | PSR_I_BIT | PSR_F_BIT
-	bic	\reg , \reg , #MODE_MASK
+	mov	\reg , #PSR_A_BIT | PSR_I_BIT | PSR_F_BIT
 	orr	\reg , \reg , #SVC_MODE
 THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
 	msr	spsr_cxsf, \reg



Nicolas

[PATCH v2 2/7] ARM: virt: allow the kernel to be entered in HYP mode

[Tony Lindgren]

* Nicolas Pitre <nicolas.pitre@linaro.org> [121005 18:33]:
> On Fri, 5 Oct 2012, Tony Lindgren wrote:
> 
> > * Tony Lindgren <tony@atomide.com> [121005 16:27]:
> > > * Russell King - ARM Linux <linux@arm.linux.org.uk> [121005 16:10]:
> > > > On Fri, Oct 05, 2012 at 01:08:22PM -0700, Tony Lindgren wrote:
> > > > > Just bisected this down in linux-next for breaking booting of
> > > > > my omap2420 ARMv6 based n8x0..
> > > > > 
> > > > > > --- a/arch/arm/kernel/head.S
> > > > > > +++ b/arch/arm/kernel/head.S
> > > > > > @@ -83,8 +83,12 @@ ENTRY(stext)
> > > > > >   THUMB(	.thumb			)	@ switch to Thumb now.
> > > > > >   THUMB(1:			)
> > > > > >  
> > > > > > -	setmode	PSR_F_BIT | PSR_I_BIT | SVC_MODE, r9 @ ensure svc mode
> > > > > > -						@ and irqs disabled
> > > > > > +#ifdef CONFIG_ARM_VIRT_EXT
> > > > > > +	bl	__hyp_stub_install
> > > > > > +#endif
> > > > > > +	@ ensure svc mode and all interrupts masked
> > > > > > +	safe_svcmode_maskall r9
> > > > > > +
> > > > > >  	mrc	p15, 0, r9, c0, c0		@ get processor id
> > > > > >  	bl	__lookup_processor_type		@ r5=procinfo r9=cpuid
> > > > > >  	movs	r10, r5				@ invalid processor (r5=0)?
> > > > > 
> > > > > ..and looks like undoing this part fixes it. Any ideas?
> > > > > 
> > > > > I quickly tried disabling ARCH_OMAP3 and ARCH_OMAP4 so it's
> > > > > ARMv6 but that does not help.
> > 
> > The same kernel boots on 2430sdp, which is the same ARMv6 core
> > as 2430 if I remember correctly. So this hints that it has something
> > to do with the bits set differently by the bootloader?
> 
> Possibly.
> 
> What if you apply this on top:
> 
> diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h
> index 683a1e6b60..b276c26e19 100644
> --- a/arch/arm/include/asm/assembler.h
> +++ b/arch/arm/include/asm/assembler.h
> @@ -254,8 +254,7 @@
>  	mov	lr , \reg
>  	and	lr , lr , #MODE_MASK
>  	cmp	lr , #HYP_MODE
> -	orr	\reg , \reg , #PSR_A_BIT | PSR_I_BIT | PSR_F_BIT
> -	bic	\reg , \reg , #MODE_MASK
> +	mov	\reg , #PSR_A_BIT | PSR_I_BIT | PSR_F_BIT
>  	orr	\reg , \reg , #SVC_MODE
>  THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
>  	msr	spsr_cxsf, \reg
> 
> 
> 

Thanks but that does not seem to help.

Regards,

Tony

[PATCH v2 2/7] ARM: virt: allow the kernel to be entered in HYP mode

[Marc Zyngier]

Hi Tony,

On Fri, 5 Oct 2012 13:08:22 -0700, Tony Lindgren <tony@atomide.com> wrote:
> Hi,
> 
> * Marc Zyngier <marc.zyngier@arm.com> [120907 10:04]:
>> From: Dave Martin <dave.martin@linaro.org>
>> 
>> This patch does two things:
>> 
>>   * Ensure that asynchronous aborts are masked at kernel entry.
>>     The bootloader should be masking these anyway, but this reduces
>>     the damage window just in case it doesn't.
>> 
>>   * Enter svc mode via exception return to ensure that CPU state is
>>     properly serialised.  This does not matter when switching from
>>     an ordinary privileged mode ("PL1" modes in ARMv7-AR rev C
>>     parlance), but it potentially does matter when switching from a
>>     another privileged mode such as hyp mode.
>> 
>> This should allow the kernel to boot safely either from svc mode or
>> hyp mode, even if no support for use of the ARM Virtualization
>> Extensions is built into the kernel.
>> 
>> Signed-off-by: Dave Martin <dave.martin@linaro.org>
>> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> 
> Just bisected this down in linux-next for breaking booting of
> my omap2420 ARMv6 based n8x0..
> 
>> --- a/arch/arm/kernel/head.S
>> +++ b/arch/arm/kernel/head.S
>> @@ -83,8 +83,12 @@ ENTRY(stext)
>>   THUMB(	.thumb			)	@ switch to Thumb now.
>>   THUMB(1:			)
>>  
>> -	setmode	PSR_F_BIT | PSR_I_BIT | SVC_MODE, r9 @ ensure svc mode
>> -						@ and irqs disabled
>> +#ifdef CONFIG_ARM_VIRT_EXT
>> +	bl	__hyp_stub_install
>> +#endif
>> +	@ ensure svc mode and all interrupts masked
>> +	safe_svcmode_maskall r9
>> +
>>  	mrc	p15, 0, r9, c0, c0		@ get processor id
>>  	bl	__lookup_processor_type		@ r5=procinfo r9=cpuid
>>  	movs	r10, r5				@ invalid processor (r5=0)?
> 
> ..and looks like undoing this part fixes it. Any ideas?
> 
> I quickly tried disabling ARCH_OMAP3 and ARCH_OMAP4 so it's
> ARMv6 but that does not help.

If you compiled for v6 only, we can safely exclude __hyp_stub_install, and
I assume that you get past the decompressor.

If so, that indicates some side effect of the safe_svcmode_maskall macro,
and I suspect the "movs pc, lr" bit.

Can you try the attached patch? It basically falls back to the previous
behaviour if not entered in HYP mode.

Thanks,

        M.
-- 
Fast, cheap, reliable. Pick two.
-------------- next part --------------
diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h
index 658a15d..b21b97f 100644
--- a/arch/arm/include/asm/assembler.h
+++ b/arch/arm/include/asm/assembler.h
@@ -254,16 +254,17 @@
 	mov	lr , \reg
 	and	lr , lr , #MODE_MASK
 	cmp	lr , #HYP_MODE
-	orr	\reg , \reg , #PSR_A_BIT | PSR_I_BIT | PSR_F_BIT
+	orr	\reg , \reg , #PSR_I_BIT | PSR_F_BIT
 	bic	\reg , \reg , #MODE_MASK
 	orr	\reg , \reg , #SVC_MODE
 THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
-	msr	spsr_cxsf, \reg
-	adr	lr, BSYM(2f)
 	bne	1f
+	orr	\reg, \reg, #PSR_A_BIT
+	adr	lr, BSYM(2f)
+	msr	spsr_cxsf, \reg
 	__MSR_ELR_HYP(14)
 	__ERET
-1:	movs	pc, lr
+1:	msr	cpsr_c, \reg
 2:
 .endm
 

[PATCH v2 2/7] ARM: virt: allow the kernel to be entered in HYP mode

[Nicolas Pitre]

On Sat, 6 Oct 2012, Marc Zyngier wrote:

> Hi Tony,
> 
> On Fri, 5 Oct 2012 13:08:22 -0700, Tony Lindgren <tony@atomide.com> wrote:
> > Hi,
> > 
> > * Marc Zyngier <marc.zyngier@arm.com> [120907 10:04]:
> >> From: Dave Martin <dave.martin@linaro.org>
> >> 
> >> This patch does two things:
> >> 
> >>   * Ensure that asynchronous aborts are masked at kernel entry.
> >>     The bootloader should be masking these anyway, but this reduces
> >>     the damage window just in case it doesn't.
> >> 
> >>   * Enter svc mode via exception return to ensure that CPU state is
> >>     properly serialised.  This does not matter when switching from
> >>     an ordinary privileged mode ("PL1" modes in ARMv7-AR rev C
> >>     parlance), but it potentially does matter when switching from a
> >>     another privileged mode such as hyp mode.
> >> 
> >> This should allow the kernel to boot safely either from svc mode or
> >> hyp mode, even if no support for use of the ARM Virtualization
> >> Extensions is built into the kernel.
> >> 
> >> Signed-off-by: Dave Martin <dave.martin@linaro.org>
> >> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> > 
> > Just bisected this down in linux-next for breaking booting of
> > my omap2420 ARMv6 based n8x0..
> > 
> >> --- a/arch/arm/kernel/head.S
> >> +++ b/arch/arm/kernel/head.S
> >> @@ -83,8 +83,12 @@ ENTRY(stext)
> >>   THUMB(	.thumb			)	@ switch to Thumb now.
> >>   THUMB(1:			)
> >>  
> >> -	setmode	PSR_F_BIT | PSR_I_BIT | SVC_MODE, r9 @ ensure svc mode
> >> -						@ and irqs disabled
> >> +#ifdef CONFIG_ARM_VIRT_EXT
> >> +	bl	__hyp_stub_install
> >> +#endif
> >> +	@ ensure svc mode and all interrupts masked
> >> +	safe_svcmode_maskall r9
> >> +
> >>  	mrc	p15, 0, r9, c0, c0		@ get processor id
> >>  	bl	__lookup_processor_type		@ r5=procinfo r9=cpuid
> >>  	movs	r10, r5				@ invalid processor (r5=0)?
> > 
> > ..and looks like undoing this part fixes it. Any ideas?
> > 
> > I quickly tried disabling ARCH_OMAP3 and ARCH_OMAP4 so it's
> > ARMv6 but that does not help.
> 
> If you compiled for v6 only, we can safely exclude __hyp_stub_install, and
> I assume that you get past the decompressor.
> 
> If so, that indicates some side effect of the safe_svcmode_maskall macro,
> and I suspect the "movs pc, lr" bit.

That would be surprizing if the "movs pc, lr" was to blame.  This should 
work on all architectures.

However the A bit might be to blame.

> Can you try the attached patch? It basically falls back to the previous
> behaviour if not entered in HYP mode.

This is likely to work of course.  However I think we should try to 
pinpoint the exact problem i.e. whether it is the A bit or the "movs pc, 
lr" which makes a difference (it is unlikely to be both).

So I was about to suggest to test this patch as well:

diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h
index 683a1e6b60..118e22ee46 100644
--- a/arch/arm/include/asm/assembler.h
+++ b/arch/arm/include/asm/assembler.h
@@ -254,8 +254,7 @@
 	mov	lr , \reg
 	and	lr , lr , #MODE_MASK
 	cmp	lr , #HYP_MODE
-	orr	\reg , \reg , #PSR_A_BIT | PSR_I_BIT | PSR_F_BIT
-	bic	\reg , \reg , #MODE_MASK
+	mov	\reg , #PSR_I_BIT | PSR_F_BIT
 	orr	\reg , \reg , #SVC_MODE
 THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
 	msr	spsr_cxsf, \reg


Nicolas

[PATCH v2 2/7] ARM: virt: allow the kernel to be entered in HYP mode

[Tony Lindgren]

* Nicolas Pitre <nicolas.pitre@linaro.org> [121006 07:07]:
> On Sat, 6 Oct 2012, Marc Zyngier wrote:
> > 
> > If so, that indicates some side effect of the safe_svcmode_maskall macro,
> > and I suspect the "movs pc, lr" bit.
> 
> That would be surprizing if the "movs pc, lr" was to blame.  This should 
> work on all architectures.
> 
> However the A bit might be to blame.
> 
> > Can you try the attached patch? It basically falls back to the previous
> > behaviour if not entered in HYP mode.
> 
> This is likely to work of course.  However I think we should try to 
> pinpoint the exact problem i.e. whether it is the A bit or the "movs pc, 
> lr" which makes a difference (it is unlikely to be both).
> 
> So I was about to suggest to test this patch as well:

No luck with this patch alone.

Regards,

Tony
 
> diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h
> index 683a1e6b60..118e22ee46 100644
> --- a/arch/arm/include/asm/assembler.h
> +++ b/arch/arm/include/asm/assembler.h
> @@ -254,8 +254,7 @@
>  	mov	lr , \reg
>  	and	lr , lr , #MODE_MASK
>  	cmp	lr , #HYP_MODE
> -	orr	\reg , \reg , #PSR_A_BIT | PSR_I_BIT | PSR_F_BIT
> -	bic	\reg , \reg , #MODE_MASK
> +	mov	\reg , #PSR_I_BIT | PSR_F_BIT
>  	orr	\reg , \reg , #SVC_MODE
>  THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
>  	msr	spsr_cxsf, \reg
> 
> 
> Nicolas

[PATCH v2 2/7] ARM: virt: allow the kernel to be entered in HYP mode

[Marc Zyngier]

On Sat, 6 Oct 2012 10:06:00 -0400 (EDT), Nicolas Pitre
<nicolas.pitre@linaro.org> wrote:
> On Sat, 6 Oct 2012, Marc Zyngier wrote:
> 
>> Hi Tony,
>> 
>> On Fri, 5 Oct 2012 13:08:22 -0700, Tony Lindgren <tony@atomide.com>
>> wrote:
>> > Hi,
>> > 
>> > * Marc Zyngier <marc.zyngier@arm.com> [120907 10:04]:
>> >> From: Dave Martin <dave.martin@linaro.org>
>> >> 
>> >> This patch does two things:
>> >> 
>> >>   * Ensure that asynchronous aborts are masked at kernel entry.
>> >>     The bootloader should be masking these anyway, but this reduces
>> >>     the damage window just in case it doesn't.
>> >> 
>> >>   * Enter svc mode via exception return to ensure that CPU state is
>> >>     properly serialised.  This does not matter when switching from
>> >>     an ordinary privileged mode ("PL1" modes in ARMv7-AR rev C
>> >>     parlance), but it potentially does matter when switching from a
>> >>     another privileged mode such as hyp mode.
>> >> 
>> >> This should allow the kernel to boot safely either from svc mode or
>> >> hyp mode, even if no support for use of the ARM Virtualization
>> >> Extensions is built into the kernel.
>> >> 
>> >> Signed-off-by: Dave Martin <dave.martin@linaro.org>
>> >> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
>> > 
>> > Just bisected this down in linux-next for breaking booting of
>> > my omap2420 ARMv6 based n8x0..
>> > 
>> >> --- a/arch/arm/kernel/head.S
>> >> +++ b/arch/arm/kernel/head.S
>> >> @@ -83,8 +83,12 @@ ENTRY(stext)
>> >>   THUMB(	.thumb			)	@ switch to Thumb now.
>> >>   THUMB(1:			)
>> >>  
>> >> -	setmode	PSR_F_BIT | PSR_I_BIT | SVC_MODE, r9 @ ensure svc mode
>> >> -						@ and irqs disabled
>> >> +#ifdef CONFIG_ARM_VIRT_EXT
>> >> +	bl	__hyp_stub_install
>> >> +#endif
>> >> +	@ ensure svc mode and all interrupts masked
>> >> +	safe_svcmode_maskall r9
>> >> +
>> >>  	mrc	p15, 0, r9, c0, c0		@ get processor id
>> >>  	bl	__lookup_processor_type		@ r5=procinfo r9=cpuid
>> >>  	movs	r10, r5				@ invalid processor (r5=0)?
>> > 
>> > ..and looks like undoing this part fixes it. Any ideas?
>> > 
>> > I quickly tried disabling ARCH_OMAP3 and ARCH_OMAP4 so it's
>> > ARMv6 but that does not help.
>> 
>> If you compiled for v6 only, we can safely exclude __hyp_stub_install,
>> and
>> I assume that you get past the decompressor.
>> 
>> If so, that indicates some side effect of the safe_svcmode_maskall
macro,
>> and I suspect the "movs pc, lr" bit.
> 
> That would be surprizing if the "movs pc, lr" was to blame.  This should

> work on all architectures.
> 
> However the A bit might be to blame.
> 
>> Can you try the attached patch? It basically falls back to the previous
>> behaviour if not entered in HYP mode.
> 
> This is likely to work of course.  However I think we should try to 
> pinpoint the exact problem i.e. whether it is the A bit or the "movs pc,

> lr" which makes a difference (it is unlikely to be both).

Agreed. We need to get to the bottom of this. Also, finding out which
revisions of the 1136 are present on both of Tony's platforms could help
understanding why this works on OMAP 2430 and not 2420.

> So I was about to suggest to test this patch as well:
> 
> diff --git a/arch/arm/include/asm/assembler.h
> b/arch/arm/include/asm/assembler.h
> index 683a1e6b60..118e22ee46 100644
> --- a/arch/arm/include/asm/assembler.h
> +++ b/arch/arm/include/asm/assembler.h
> @@ -254,8 +254,7 @@
>  	mov	lr , \reg
>  	and	lr , lr , #MODE_MASK
>  	cmp	lr , #HYP_MODE
> -	orr	\reg , \reg , #PSR_A_BIT | PSR_I_BIT | PSR_F_BIT
> -	bic	\reg , \reg , #MODE_MASK
> +	mov	\reg , #PSR_I_BIT | PSR_F_BIT
>  	orr	\reg , \reg , #SVC_MODE
>  THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
>  	msr	spsr_cxsf, \reg

Yup, that should give us all the information we need.

Thanks Nico.

        M.
-- 
Fast, cheap, reliable. Pick two.

[PATCH v2 2/7] ARM: virt: allow the kernel to be entered in HYP mode

[Tony Lindgren]

Hi,

* Marc Zyngier <marc.zyngier@arm.com> [121006 03:19]:
> 
> If you compiled for v6 only, we can safely exclude __hyp_stub_install, and
> I assume that you get past the decompressor.

Yes, by default it's v6 + v7, but making it v6 only did not help.
 
> If so, that indicates some side effect of the safe_svcmode_maskall macro,
> and I suspect the "movs pc, lr" bit.
> 
> Can you try the attached patch? It basically falls back to the previous
> behaviour if not entered in HYP mode.

Yes, with this it boots OK.

Regards,

Tony

> diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h
> index 658a15d..b21b97f 100644
> --- a/arch/arm/include/asm/assembler.h
> +++ b/arch/arm/include/asm/assembler.h
> @@ -254,16 +254,17 @@
>  	mov	lr , \reg
>  	and	lr , lr , #MODE_MASK
>  	cmp	lr , #HYP_MODE
> -	orr	\reg , \reg , #PSR_A_BIT | PSR_I_BIT | PSR_F_BIT
> +	orr	\reg , \reg , #PSR_I_BIT | PSR_F_BIT
>  	bic	\reg , \reg , #MODE_MASK
>  	orr	\reg , \reg , #SVC_MODE
>  THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
> -	msr	spsr_cxsf, \reg
> -	adr	lr, BSYM(2f)
>  	bne	1f
> +	orr	\reg, \reg, #PSR_A_BIT
> +	adr	lr, BSYM(2f)
> +	msr	spsr_cxsf, \reg
>  	__MSR_ELR_HYP(14)
>  	__ERET
> -1:	movs	pc, lr
> +1:	msr	cpsr_c, \reg
>  2:
>  .endm
>  

[PATCH v2 2/7] ARM: virt: allow the kernel to be entered in HYP mode

[Nicolas Pitre]

On Sat, 6 Oct 2012, Tony Lindgren wrote:

> * Marc Zyngier <marc.zyngier@arm.com> [121006 03:19]:
> 
> > If so, that indicates some side effect of the safe_svcmode_maskall macro,
> > and I suspect the "movs pc, lr" bit.
> > 
> > Can you try the attached patch? It basically falls back to the previous
> > behaviour if not entered in HYP mode.
> 
> Yes, with this it boots OK.

OK. In that case, I suggest this patch be sent to Russell to fix this 
issue so he could push the ARM stuff to Linus ASAP.

Acked-by: Nicolas Pitre <nico@linaro.org>


Nicolas

[PATCH v2 2/7] ARM: virt: allow the kernel to be entered in HYP mode

[Tony Lindgren]

* Nicolas Pitre <nicolas.pitre@linaro.org> [121006 08:33]:
> On Sat, 6 Oct 2012, Tony Lindgren wrote:
> 
> > * Marc Zyngier <marc.zyngier@arm.com> [121006 03:19]:
> > 
> > > If so, that indicates some side effect of the safe_svcmode_maskall macro,
> > > and I suspect the "movs pc, lr" bit.
> > > 
> > > Can you try the attached patch? It basically falls back to the previous
> > > behaviour if not entered in HYP mode.
> > 
> > Yes, with this it boots OK.
> 
> OK. In that case, I suggest this patch be sent to Russell to fix this 
> issue so he could push the ARM stuff to Linus ASAP.
> 
> Acked-by: Nicolas Pitre <nico@linaro.org>

Yes we can come back to this:

Tested-by: Tony Lindgren <tony@atomide.com>

[PATCH v2 2/7] ARM: virt: allow the kernel to be entered in HYP mode

[Marc Zyngier]

On Sat, 6 Oct 2012 08:40:52 -0700, Tony Lindgren <tony@atomide.com> wrote:
> * Nicolas Pitre <nicolas.pitre@linaro.org> [121006 08:33]:
>> On Sat, 6 Oct 2012, Tony Lindgren wrote:
>> 
>> > * Marc Zyngier <marc.zyngier@arm.com> [121006 03:19]:
>> > 
>> > > If so, that indicates some side effect of the safe_svcmode_maskall
>> > > macro,
>> > > and I suspect the "movs pc, lr" bit.
>> > > 
>> > > Can you try the attached patch? It basically falls back to the
>> > > previous
>> > > behaviour if not entered in HYP mode.
>> > 
>> > Yes, with this it boots OK.
>> 
>> OK. In that case, I suggest this patch be sent to Russell to fix this 
>> issue so he could push the ARM stuff to Linus ASAP.
>> 
>> Acked-by: Nicolas Pitre <nico@linaro.org>
> 
> Yes we can come back to this:
> 
> Tested-by: Tony Lindgren <tony@atomide.com>

Now in Russell's patch system as #7549/1.

Thanks a lot for your help on this both of you.

        M.
-- 
Fast, cheap, reliable. Pick two.

[PATCH v2 2/7] ARM: virt: allow the kernel to be entered in HYP mode

[Russell King - ARM Linux]

On Sat, Oct 06, 2012 at 11:32:16AM -0400, Nicolas Pitre wrote:
> On Sat, 6 Oct 2012, Tony Lindgren wrote:
> > * Marc Zyngier <marc.zyngier@arm.com> [121006 03:19]:
> > 
> > > If so, that indicates some side effect of the safe_svcmode_maskall macro,
> > > and I suspect the "movs pc, lr" bit.
> > > 
> > > Can you try the attached patch? It basically falls back to the previous
> > > behaviour if not entered in HYP mode.
> > 
> > Yes, with this it boots OK.
> 
> OK. In that case, I suggest this patch be sent to Russell to fix this 
> issue so he could push the ARM stuff to Linus ASAP.
> 
> Acked-by: Nicolas Pitre <nico@linaro.org>

I've already sent the pull request (it's late enough already for the
first bunch of stuff... Linus' truely brilliant timing for the N'th
time sees the merge window open when I'm away - why does he keep doing
that to me?) for everything _except_ stuff in my 'devel-stable' branch,
which includes the opcodes, virt, and new cache maintanence stuff.

I won't be sending Linus another pull request until at least three
days after he merges the current request, so we'll just have to hope
that this doesn't turn out to be a short merge window...

[PATCH v2 2/7] ARM: virt: allow the kernel to be entered in HYP mode

[Tony Lindgren]

* Marc Zyngier <marc.zyngier@arm.com> [121006 03:19]:
> 
> If so, that indicates some side effect of the safe_svcmode_maskall macro,
> and I suspect the "movs pc, lr" bit.
> 
> Can you try the attached patch? It basically falls back to the previous
> behaviour if not entered in HYP mode.
...

> diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h
> index 658a15d..b21b97f 100644
> --- a/arch/arm/include/asm/assembler.h
> +++ b/arch/arm/include/asm/assembler.h
> @@ -254,16 +254,17 @@
>  	mov	lr , \reg
>  	and	lr , lr , #MODE_MASK
>  	cmp	lr , #HYP_MODE
> -	orr	\reg , \reg , #PSR_A_BIT | PSR_I_BIT | PSR_F_BIT
> +	orr	\reg , \reg , #PSR_I_BIT | PSR_F_BIT
>  	bic	\reg , \reg , #MODE_MASK
>  	orr	\reg , \reg , #SVC_MODE
>  THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
> -	msr	spsr_cxsf, \reg
> -	adr	lr, BSYM(2f)
>  	bne	1f
> +	orr	\reg, \reg, #PSR_A_BIT
> +	adr	lr, BSYM(2f)
> +	msr	spsr_cxsf, \reg
>  	__MSR_ELR_HYP(14)
>  	__ERET
> -1:	movs	pc, lr
> +1:	msr	cpsr_c, \reg
>  2:
>  .endm
>  

The minimal version of this that still boots on my n800 is just
the last change of the above patch:

--- a/arch/arm/include/asm/assembler.h
+++ b/arch/arm/include/asm/assembler.h
@@ -263,7 +263,7 @@ THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
 	bne	1f
 	__MSR_ELR_HYP(14)
 	__ERET
-1:	movs	pc, lr
+1:	msr	cpsr_c, \reg
 2:
 .endm
 

[PATCH v2 2/7] ARM: virt: allow the kernel to be entered in HYP mode

[Dave Martin]

On Sat, Oct 06, 2012 at 09:00:32AM -0700, Tony Lindgren wrote:
> * Marc Zyngier <marc.zyngier@arm.com> [121006 03:19]:
> > 
> > If so, that indicates some side effect of the safe_svcmode_maskall macro,
> > and I suspect the "movs pc, lr" bit.
> > 
> > Can you try the attached patch? It basically falls back to the previous
> > behaviour if not entered in HYP mode.
> ...
> 
> > diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h
> > index 658a15d..b21b97f 100644
> > --- a/arch/arm/include/asm/assembler.h
> > +++ b/arch/arm/include/asm/assembler.h
> > @@ -254,16 +254,17 @@
> >  	mov	lr , \reg
> >  	and	lr , lr , #MODE_MASK
> >  	cmp	lr , #HYP_MODE
> > -	orr	\reg , \reg , #PSR_A_BIT | PSR_I_BIT | PSR_F_BIT
> > +	orr	\reg , \reg , #PSR_I_BIT | PSR_F_BIT
> >  	bic	\reg , \reg , #MODE_MASK
> >  	orr	\reg , \reg , #SVC_MODE
> >  THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
> > -	msr	spsr_cxsf, \reg
> > -	adr	lr, BSYM(2f)
> >  	bne	1f
> > +	orr	\reg, \reg, #PSR_A_BIT
> > +	adr	lr, BSYM(2f)
> > +	msr	spsr_cxsf, \reg
> >  	__MSR_ELR_HYP(14)
> >  	__ERET
> > -1:	movs	pc, lr
> > +1:	msr	cpsr_c, \reg
> >  2:
> >  .endm
> >  
> 
> The minimal version of this that still boots on my n800 is just
> the last change of the above patch:
> 
> --- a/arch/arm/include/asm/assembler.h
> +++ b/arch/arm/include/asm/assembler.h
> @@ -263,7 +263,7 @@ THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
>  	bne	1f
>  	__MSR_ELR_HYP(14)
>  	__ERET
> -1:	movs	pc, lr
> +1:	msr	cpsr_c, \reg
>  2:
>  .endm
>  

In an attempt to narrow this down...

Can you follow this (i.e., _after_ a known successful switch to SVC mode)

(a)
	mrs	\reg, cpsr
	msr	spsr_cxsf, \reg
	adr	\reg, 3f
	movs	pc, lr
3:

and (b)

	mrs	\reg, cpsr
	orr	\reg, \reg, #CPSR_A_BIT
	msr	cpsr_cxsf, \reg

and (c)

	mrs	\reg, cpsr
	orr	\reg, \reg, #CPSR_A_BIT
	msr	spsr_cxsf, \reg
	adr	\reg, 3f
	movs	pc, lr
3:




If only (a) works, this would suggest that the attempt to set the A bit
is causing the problem.

If only (b) works, this suggests that the A bit is OK but that some
invalid hardware state, or something else we don't understand, is causing
exception returns to fail in general.

If (a) and (b) work but (c) fails, this suggests that specifically
trying to set the A bit via an exception return is problematic.

If all of them work then this suggests some invalid hardware state or
something else we don't understand, but which is cleared by the initial
msr cpsr_c which clobbers the processor mode.


Cheers
---Dave

[PATCH v2 2/7] ARM: virt: allow the kernel to be entered in HYP mode

[Marc Zyngier]

On 08/10/12 12:01, Dave Martin wrote:
> On Sat, Oct 06, 2012 at 09:00:32AM -0700, Tony Lindgren wrote:
>> * Marc Zyngier <marc.zyngier@arm.com> [121006 03:19]:
>>>
>>> If so, that indicates some side effect of the safe_svcmode_maskall macro,
>>> and I suspect the "movs pc, lr" bit.
>>>
>>> Can you try the attached patch? It basically falls back to the previous
>>> behaviour if not entered in HYP mode.
>> ...
>>
>>> diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h
>>> index 658a15d..b21b97f 100644
>>> --- a/arch/arm/include/asm/assembler.h
>>> +++ b/arch/arm/include/asm/assembler.h
>>> @@ -254,16 +254,17 @@
>>>  	mov	lr , \reg
>>>  	and	lr , lr , #MODE_MASK
>>>  	cmp	lr , #HYP_MODE
>>> -	orr	\reg , \reg , #PSR_A_BIT | PSR_I_BIT | PSR_F_BIT
>>> +	orr	\reg , \reg , #PSR_I_BIT | PSR_F_BIT
>>>  	bic	\reg , \reg , #MODE_MASK
>>>  	orr	\reg , \reg , #SVC_MODE
>>>  THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
>>> -	msr	spsr_cxsf, \reg
>>> -	adr	lr, BSYM(2f)
>>>  	bne	1f
>>> +	orr	\reg, \reg, #PSR_A_BIT
>>> +	adr	lr, BSYM(2f)
>>> +	msr	spsr_cxsf, \reg
>>>  	__MSR_ELR_HYP(14)
>>>  	__ERET
>>> -1:	movs	pc, lr
>>> +1:	msr	cpsr_c, \reg
>>>  2:
>>>  .endm
>>>  
>>
>> The minimal version of this that still boots on my n800 is just
>> the last change of the above patch:
>>
>> --- a/arch/arm/include/asm/assembler.h
>> +++ b/arch/arm/include/asm/assembler.h
>> @@ -263,7 +263,7 @@ THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
>>  	bne	1f
>>  	__MSR_ELR_HYP(14)
>>  	__ERET
>> -1:	movs	pc, lr
>> +1:	msr	cpsr_c, \reg
>>  2:
>>  .endm
>>  
> 
> In an attempt to narrow this down...
> 
> Can you follow this (i.e., _after_ a known successful switch to SVC mode)
> 
> (a)
> 	mrs	\reg, cpsr
> 	msr	spsr_cxsf, \reg
> 	adr	\reg, 3f
> 	movs	pc, lr
> 3:
> 
> and (b)
> 
> 	mrs	\reg, cpsr
> 	orr	\reg, \reg, #CPSR_A_BIT
> 	msr	cpsr_cxsf, \reg
> 
> and (c)
> 
> 	mrs	\reg, cpsr
> 	orr	\reg, \reg, #CPSR_A_BIT
> 	msr	spsr_cxsf, \reg
> 	adr	\reg, 3f
> 	movs	pc, lr
> 3:
> 
> 
> 
> 
> If only (a) works, this would suggest that the attempt to set the A bit
> is causing the problem.
> 
> If only (b) works, this suggests that the A bit is OK but that some
> invalid hardware state, or something else we don't understand, is causing
> exception returns to fail in general.
> 
> If (a) and (b) work but (c) fails, this suggests that specifically
> trying to set the A bit via an exception return is problematic.
> 
> If all of them work then this suggests some invalid hardware state or
> something else we don't understand, but which is cleared by the initial
> msr cpsr_c which clobbers the processor mode.

Playing with an Integrator-CP with a 1136 tile, the only way I could
cause the thing to fail was to boot the thing in SYSTEM mode.

Tony, can you possibly also try the attached patch?

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sysmode.patch
Type: text/x-diff
Size: 593 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20121008/d1635b08/attachment.bin>

[PATCH v2 2/7] ARM: virt: allow the kernel to be entered in HYP mode

[Tony Lindgren]

* Marc Zyngier <marc.zyngier@arm.com> [121008 04:34]:
> On 08/10/12 12:01, Dave Martin wrote:
> > 
> > In an attempt to narrow this down...
> > 
> > Can you follow this (i.e., _after_ a known successful switch to SVC mode)
> > 
> > (a)
> > 	mrs	\reg, cpsr
> > 	msr	spsr_cxsf, \reg
> > 	adr	\reg, 3f
> > 	movs	pc, lr
> > 3:
> > 
> > and (b)
> > 
> > 	mrs	\reg, cpsr
> > 	orr	\reg, \reg, #CPSR_A_BIT
> > 	msr	cpsr_cxsf, \reg
> > 
> > and (c)
> > 
> > 	mrs	\reg, cpsr
> > 	orr	\reg, \reg, #CPSR_A_BIT
> > 	msr	spsr_cxsf, \reg
> > 	adr	\reg, 3f
> > 	movs	pc, lr
> > 3:
> > 
> > 
> > 
> > 
> > If only (a) works, this would suggest that the attempt to set the A bit
> > is causing the problem.
> > 
> > If only (b) works, this suggests that the A bit is OK but that some
> > invalid hardware state, or something else we don't understand, is causing
> > exception returns to fail in general.
> > 
> > If (a) and (b) work but (c) fails, this suggests that specifically
> > trying to set the A bit via an exception return is problematic.
> > 
> > If all of them work then this suggests some invalid hardware state or
> > something else we don't understand, but which is cleared by the initial
> > msr cpsr_c which clobbers the processor mode.

Thanks it does not seem to be the A bit..
 
> Playing with an Integrator-CP with a 1136 tile, the only way I could
> cause the thing to fail was to boot the thing in SYSTEM mode.
> 
> Tony, can you possibly also try the attached patch?

Yup you figured it out n800 is in system mode while 2430sdp is not..
I verified it with printhex2.

Your patch fails to compile with:

arch/arm/kernel/head.S: Assembler messages:
arch/arm/kernel/head.S:336: Error: symbol `omgsys' is already defined

Probably because it's included twice. Also, shouldn't it
be bne omgsys instead of beq?

Regards,

Tony

> diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h
> index 683a1e6..7714ec7 100644
> --- a/arch/arm/include/asm/assembler.h
> +++ b/arch/arm/include/asm/assembler.h
> @@ -253,6 +253,8 @@
>  	mrs	\reg , cpsr
>  	mov	lr , \reg
>  	and	lr , lr , #MODE_MASK
> +	cmp	lr , #SYSTEM_MODE	/* Yet another braindead platform? */
> +	beq	omgsys
>  	cmp	lr , #HYP_MODE
>  	orr	\reg , \reg , #PSR_A_BIT | PSR_I_BIT | PSR_F_BIT
>  	bic	\reg , \reg , #MODE_MASK
> @@ -264,6 +266,7 @@ THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
>  	__MSR_ELR_HYP(14)
>  	__ERET
>  1:	movs	pc, lr
> +omgsys: msr	cpsr_c, \reg
>  2:
>  .endm
>  

[PATCH v2 2/7] ARM: virt: allow the kernel to be entered in HYP mode

[Dave Martin]

On Mon, Oct 08, 2012 at 12:01:09PM +0100, Dave Martin wrote:
> On Sat, Oct 06, 2012 at 09:00:32AM -0700, Tony Lindgren wrote:
> > * Marc Zyngier <marc.zyngier@arm.com> [121006 03:19]:
> > > 
> > > If so, that indicates some side effect of the safe_svcmode_maskall macro,
> > > and I suspect the "movs pc, lr" bit.
> > > 
> > > Can you try the attached patch? It basically falls back to the previous
> > > behaviour if not entered in HYP mode.
> > ...
> > 
> > > diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h
> > > index 658a15d..b21b97f 100644
> > > --- a/arch/arm/include/asm/assembler.h
> > > +++ b/arch/arm/include/asm/assembler.h
> > > @@ -254,16 +254,17 @@
> > >  	mov	lr , \reg
> > >  	and	lr , lr , #MODE_MASK
> > >  	cmp	lr , #HYP_MODE
> > > -	orr	\reg , \reg , #PSR_A_BIT | PSR_I_BIT | PSR_F_BIT
> > > +	orr	\reg , \reg , #PSR_I_BIT | PSR_F_BIT
> > >  	bic	\reg , \reg , #MODE_MASK
> > >  	orr	\reg , \reg , #SVC_MODE
> > >  THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
> > > -	msr	spsr_cxsf, \reg
> > > -	adr	lr, BSYM(2f)
> > >  	bne	1f
> > > +	orr	\reg, \reg, #PSR_A_BIT
> > > +	adr	lr, BSYM(2f)
> > > +	msr	spsr_cxsf, \reg
> > >  	__MSR_ELR_HYP(14)
> > >  	__ERET
> > > -1:	movs	pc, lr
> > > +1:	msr	cpsr_c, \reg
> > >  2:
> > >  .endm
> > >  
> > 
> > The minimal version of this that still boots on my n800 is just
> > the last change of the above patch:
> > 
> > --- a/arch/arm/include/asm/assembler.h
> > +++ b/arch/arm/include/asm/assembler.h
> > @@ -263,7 +263,7 @@ THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
> >  	bne	1f
> >  	__MSR_ELR_HYP(14)
> >  	__ERET
> > -1:	movs	pc, lr
> > +1:	msr	cpsr_c, \reg
> >  2:
> >  .endm
> >  
> 
> In an attempt to narrow this down...
> 
> Can you follow this (i.e., _after_ a known successful switch to SVC mode)
> 
> (a)
> 	mrs	\reg, cpsr
> 	msr	spsr_cxsf, \reg
> 	adr	\reg, 3f
> 	movs	pc, lr
> 3:
> 
> and (b)
> 
> 	mrs	\reg, cpsr
> 	orr	\reg, \reg, #CPSR_A_BIT
> 	msr	cpsr_cxsf, \reg
> 
> and (c)
> 
> 	mrs	\reg, cpsr
> 	orr	\reg, \reg, #CPSR_A_BIT
> 	msr	spsr_cxsf, \reg
> 	adr	\reg, 3f
> 	movs	pc, lr
> 3:
> 
> 
> 
> 
> If only (a) works, this would suggest that the attempt to set the A bit
> is causing the problem.
> 
> If only (b) works, this suggests that the A bit is OK but that some
> invalid hardware state, or something else we don't understand, is causing
> exception returns to fail in general.
> 
> If (a) and (b) work but (c) fails, this suggests that specifically
> trying to set the A bit via an exception return is problematic.
> 
> If all of them work then this suggests some invalid hardware state or
> something else we don't understand, but which is cleared by the initial
> msr cpsr_c which clobbers the processor mode.


Marc Z also just pointed out to me that there is one architecturally
valid explanation for why the movs route could fail: if the kernel is
entered in System mode for some reason -- this causes msr spsr and movs
pc to become UNPREDICTABLE.  If this is the explanation, then
(a), (b) and (c) should all work, provided the CPU has already been forced
out of System mode.

Of course, we're not supposed to be entered in System mode ... but since
the whole purpose of this code is to force us into a sane state, we should
work around it anyway.  I think Marc is busy rolling a patch for that.

Cheers
---Dave

[PATCH v2 3/7] ARM: zImage/virt: hyp mode entry support for the zImage loader

[Marc Zyngier]

From: Dave Martin <dave.martin@linaro.org>

The zImage loader needs to turn on the MMU in order to take
advantage of caching while decompressing the zImage.  Running this
in hyp mode would require the LPAE pagetable format to be
supported; to avoid this complexity, this patch switches out of hyp
mode, and returns back to hyp mode just before booting the kernel.

This implementation assumes that the Hyp mode view of memory and the
PL1 view of memory are coherent, providing that the MMU and caches
are off in both, as required by the boot protocol.  The zImage
decompression code must drain the write buffer on completion anyway, and
entry into Hyp mode should flush any prefetch buffer, avoiding hazards
associated with local write buffers and the pipeline.

Signed-off-by: Dave Martin <dave.martin@linaro.org>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm/boot/compressed/.gitignore |  1 +
 arch/arm/boot/compressed/Makefile   |  9 ++++-
 arch/arm/boot/compressed/head.S     | 71 +++++++++++++++++++++++++++++++++----
 arch/arm/kernel/hyp-stub.S          | 18 ++++++++++
 4 files changed, 91 insertions(+), 8 deletions(-)

diff --git a/arch/arm/boot/compressed/.gitignore b/arch/arm/boot/compressed/.gitignore
index d0d441c..f79a08e 100644
--- a/arch/arm/boot/compressed/.gitignore
+++ b/arch/arm/boot/compressed/.gitignore
@@ -1,6 +1,7 @@
 ashldi3.S
 font.c
 lib1funcs.S
+hyp-stub.S
 piggy.gzip
 piggy.lzo
 piggy.lzma
diff --git a/arch/arm/boot/compressed/Makefile b/arch/arm/boot/compressed/Makefile
index bb26756..a517153 100644
--- a/arch/arm/boot/compressed/Makefile
+++ b/arch/arm/boot/compressed/Makefile
@@ -30,6 +30,10 @@ FONTC	= $(srctree)/drivers/video/console/font_acorn_8x8.c
 OBJS		+= string.o
 CFLAGS_string.o	:= -Os
 
+ifeq ($(CONFIG_ARM_VIRT_EXT),y)
+OBJS		+= hyp-stub.o
+endif
+
 #
 # Architecture dependencies
 #
@@ -126,7 +130,7 @@ KBUILD_CFLAGS = $(subst -pg, , $(ORIG_CFLAGS))
 endif
 
 ccflags-y := -fpic -fno-builtin -I$(obj)
-asflags-y := -Wa,-march=all
+asflags-y := -Wa,-march=all -DZIMAGE
 
 # Supply kernel BSS size to the decompressor via a linker symbol.
 KBSS_SZ = $(shell $(CROSS_COMPILE)size $(obj)/../../../../vmlinux | \
@@ -198,3 +202,6 @@ $(obj)/font.c: $(FONTC)
 
 $(obj)/vmlinux.lds: $(obj)/vmlinux.lds.in arch/arm/boot/Makefile $(KCONFIG_CONFIG)
 	@sed "$(SEDFLAGS)" < $< > $@
+
+$(obj)/hyp-stub.S: $(srctree)/arch/$(SRCARCH)/kernel/hyp-stub.S
+	$(call cmd,shipped)
diff --git a/arch/arm/boot/compressed/head.S b/arch/arm/boot/compressed/head.S
index b8c64b8..0749a61 100644
--- a/arch/arm/boot/compressed/head.S
+++ b/arch/arm/boot/compressed/head.S
@@ -9,6 +9,7 @@
  * published by the Free Software Foundation.
  */
 #include <linux/linkage.h>
+#include <asm/assembler.h>
 
 /*
  * Debugging stuff
@@ -132,7 +133,12 @@ start:
 		.word	start			@ absolute load/run zImage address
 		.word	_edata			@ zImage end address
  THUMB(		.thumb			)
-1:		mov	r7, r1			@ save architecture ID
+1:
+		mrs	r9, cpsr
+#ifdef CONFIG_ARM_VIRT_EXT
+		bl	__hyp_stub_install	@ get into SVC mode, reversibly
+#endif
+		mov	r7, r1			@ save architecture ID
 		mov	r8, r2			@ save atags pointer
 
 #ifndef __ARM_ARCH_2__
@@ -148,9 +154,9 @@ start:
  ARM(		swi	0x123456	)	@ angel_SWI_ARM
  THUMB(		svc	0xab		)	@ angel_SWI_THUMB
 not_angel:
-		mrs	r2, cpsr		@ turn off interrupts to
-		orr	r2, r2, #0xc0		@ prevent angel from running
-		msr	cpsr_c, r2
+		safe_svcmode_maskall r0
+		msr	spsr_cxsf, r9		@ Save the CPU boot mode in
+						@ SPSR
 #else
 		teqp	pc, #0x0c000003		@ turn off interrupts
 #endif
@@ -350,6 +356,20 @@ dtb_check_done:
 		adr	r5, restart
 		bic	r5, r5, #31
 
+/* Relocate the hyp vector base if necessary */
+#ifdef CONFIG_ARM_VIRT_EXT
+		mrs	r0, spsr
+		and	r0, r0, #MODE_MASK
+		cmp	r0, #HYP_MODE
+		bne	1f
+
+		bl	__hyp_get_vectors
+		sub	r0, r0, r5
+		add	r0, r0, r10
+		bl	__hyp_set_vectors
+1:
+#endif
+
 		sub	r9, r6, r5		@ size to copy
 		add	r9, r9, #31		@ rounded up to a multiple
 		bic	r9, r9, #31		@ ... of 32 bytes
@@ -458,11 +478,29 @@ not_relocated:	mov	r0, #0
 		bl	decompress_kernel
 		bl	cache_clean_flush
 		bl	cache_off
-		mov	r0, #0			@ must be zero
 		mov	r1, r7			@ restore architecture number
 		mov	r2, r8			@ restore atags pointer
- ARM(		mov	pc, r4	)		@ call kernel
- THUMB(		bx	r4	)		@ entry point is always ARM
+
+#ifdef CONFIG_ARM_VIRT_EXT
+		mrs	r0, spsr		@ Get saved CPU boot mode
+		and	r0, r0, #MODE_MASK
+		cmp	r0, #HYP_MODE		@ if not booted in HYP mode...
+		bne	__enter_kernel		@ boot kernel directly
+
+		adr	r12, .L__hyp_reentry_vectors_offset
+		ldr	r0, [r12]
+		add	r0, r0, r12
+
+		bl	__hyp_set_vectors
+		__HVC(0)			@ otherwise bounce to hyp mode
+
+		b	.			@ should never be reached
+
+		.align	2
+.L__hyp_reentry_vectors_offset:	.long	__hyp_reentry_vectors - .
+#else
+		b	__enter_kernel
+#endif
 
 		.align	2
 		.type	LC0, #object
@@ -1191,6 +1229,25 @@ memdump:	mov	r12, r0
 #endif
 
 		.ltorg
+
+#ifdef CONFIG_ARM_VIRT_EXT
+.align 5
+__hyp_reentry_vectors:
+		W(b)	.			@ reset
+		W(b)	.			@ undef
+		W(b)	.			@ svc
+		W(b)	.			@ pabort
+		W(b)	.			@ dabort
+		W(b)	__enter_kernel		@ hyp
+		W(b)	.			@ irq
+		W(b)	.			@ fiq
+#endif /* CONFIG_ARM_VIRT_EXT */
+
+__enter_kernel:
+		mov	r0, #0			@ must be 0
+ ARM(		mov	pc, r4	)		@ call kernel
+ THUMB(		bx	r4	)		@ entry point is always ARM
+
 reloc_code_end:
 
 		.align
diff --git a/arch/arm/kernel/hyp-stub.S b/arch/arm/kernel/hyp-stub.S
index b03e924..7060941 100644
--- a/arch/arm/kernel/hyp-stub.S
+++ b/arch/arm/kernel/hyp-stub.S
@@ -21,6 +21,7 @@
 #include <asm/assembler.h>
 #include <asm/virt.h>
 
+#ifndef ZIMAGE
 /*
  * For the kernel proper, we need to find out the CPU boot mode long after
  * boot, so we need to store it in a writable variable.
@@ -59,6 +60,21 @@ ENTRY(__boot_cpu_mode)
 	strne	r7, [r5, r6]		@ record what happened and give up
 	.endm
 
+#else	/* ZIMAGE */
+
+	.macro	store_primary_cpu_mode	reg1:req, reg2:req, reg3:req
+	.endm
+
+/*
+ * The zImage loader only runs on one CPU, so we don't bother with mult-CPU
+ * consistency checking:
+ */
+	.macro	compare_cpu_mode_with_primary mode, reg1, reg2, reg3
+	cmp	\mode, \mode
+	.endm
+
+#endif /* ZIMAGE */
+
 /*
  * Hypervisor stub installation functions.
  *
@@ -174,9 +190,11 @@ ENTRY(__hyp_set_vectors)
 	bx	lr
 ENDPROC(__hyp_set_vectors)
 
+#ifndef ZIMAGE
 .align 2
 .L__boot_cpu_mode_offset:
 	.long	__boot_cpu_mode - .
+#endif
 
 .align 5
 __hyp_stub_vectors:
-- 
1.7.12

[PATCH v2 4/7] ARM: virt: Update documentation for hyp mode entry support

[Marc Zyngier]

From: Dave Martin <dave.martin@linaro.org>

Document the possibility of the kernel being entered in HYP mode.

Signed-off-by: Dave Martin <dave.martin@linaro.org>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 Documentation/arm/Booting | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

diff --git a/Documentation/arm/Booting b/Documentation/arm/Booting
index a341d87..0c1f475 100644
--- a/Documentation/arm/Booting
+++ b/Documentation/arm/Booting
@@ -154,13 +154,33 @@ In either case, the following conditions must be met:
 
 - CPU mode
   All forms of interrupts must be disabled (IRQs and FIQs)
-  The CPU must be in SVC mode.  (A special exception exists for Angel)
+
+  For CPUs which do not include the ARM virtualization extensions, the
+  CPU must be in SVC mode.  (A special exception exists for Angel)
+
+  CPUs which include support for the virtualization extensions can be
+  entered in HYP mode in order to enable the kernel to make full use of
+  these extensions.  This is the recommended boot method for such CPUs,
+  unless the virtualisations are already in use by a pre-installed
+  hypervisor.
+
+  If the kernel is not entered in HYP mode for any reason, it must be
+  entered in SVC mode.
 
 - Caches, MMUs
   The MMU must be off.
   Instruction cache may be on or off.
   Data cache must be off.
 
+  If the kernel is entered in HYP mode, the above requirements apply to
+  the HYP mode configuration in addition to the ordinary PL1 (privileged
+  kernel modes) configuration.  In addition, all traps into the
+  hypervisor must be disabled, and PL1 access must be granted for all
+  peripherals and CPU resources for which this is architecturally
+  possible.  Except for entering in HYP mode, the system configuration
+  should be such that a kernel which does not include support for the
+  virtualization extensions can boot correctly without extra help.
+
 - The boot loader is expected to call the kernel image by jumping
   directly to the first instruction of the kernel image.
 
-- 
1.7.12

[PATCH v2 5/7] ARM: virt: Add boot-time diagnostics

[Marc Zyngier]

From: Dave Martin <dave.martin@linaro.org>

In order to easily detect pathological cases, print some diagnostics
when the kernel boots.

This also provides helpers to detect that HYP mode is actually available,
which can be used by other subsystems to enable HYP specific features.

Signed-off-by: Dave Martin <dave.martin@linaro.org>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm/include/asm/virt.h | 17 +++++++++++++++++
 arch/arm/kernel/setup.c     | 20 ++++++++++++++++++++
 arch/arm/kernel/smp.c       |  3 +++
 3 files changed, 40 insertions(+)

diff --git a/arch/arm/include/asm/virt.h b/arch/arm/include/asm/virt.h
index 0a99723..86164df 100644
--- a/arch/arm/include/asm/virt.h
+++ b/arch/arm/include/asm/virt.h
@@ -47,6 +47,23 @@ unsigned long __hyp_get_vectors(void);
 #define __boot_cpu_mode	(SVC_MODE)
 #endif
 
+#ifndef ZIMAGE
+void hyp_mode_check(void);
+
+/* Reports the availability of HYP mode */
+static inline bool is_hyp_mode_available(void)
+{
+	return ((__boot_cpu_mode & MODE_MASK) == HYP_MODE &&
+		!(__boot_cpu_mode & BOOT_CPU_MODE_MISMATCH));
+}
+
+/* Check if the bootloader has booted CPUs in different modes */
+static inline bool is_hyp_mode_mismatched(void)
+{
+	return !!(__boot_cpu_mode & BOOT_CPU_MODE_MISMATCH);
+}
+#endif
+
 #endif /* __ASSEMBLY__ */
 
 #endif /* ! VIRT_H */
diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c
index a81dcec..04fd01f 100644
--- a/arch/arm/kernel/setup.c
+++ b/arch/arm/kernel/setup.c
@@ -55,6 +55,7 @@
 #include <asm/traps.h>
 #include <asm/unwind.h>
 #include <asm/memblock.h>
+#include <asm/virt.h>
 
 #if defined(CONFIG_DEPRECATED_PARAM_STRUCT)
 #include "compat.h"
@@ -937,6 +938,21 @@ static int __init meminfo_cmp(const void *_a, const void *_b)
 	return cmp < 0 ? -1 : cmp > 0 ? 1 : 0;
 }
 
+void __init hyp_mode_check(void)
+{
+#ifdef CONFIG_ARM_VIRT_EXT
+	if (is_hyp_mode_available()) {
+		pr_info("CPU: All CPU(s) started in HYP mode.\n");
+		pr_info("CPU: Virtualization extensions available.\n");
+	} else if (is_hyp_mode_mismatched()) {
+		pr_warn("CPU: WARNING: CPU(s) started in wrong/inconsistent modes (primary CPU mode 0x%x)\n",
+			__boot_cpu_mode & MODE_MASK);
+		pr_warn("CPU: This may indicate a broken bootloader or firmware.\n");
+	} else
+		pr_info("CPU: All CPU(s) started in SVC mode.\n");
+#endif
+}
+
 void __init setup_arch(char **cmdline_p)
 {
 	struct machine_desc *mdesc;
@@ -980,6 +996,10 @@ void __init setup_arch(char **cmdline_p)
 	if (is_smp())
 		smp_init_cpus();
 #endif
+
+	if (!is_smp())
+		hyp_mode_check();
+
 	reserve_crashkernel();
 
 	tcm_init();
diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
index ebd8ad2..adf226e 100644
--- a/arch/arm/kernel/smp.c
+++ b/arch/arm/kernel/smp.c
@@ -42,6 +42,7 @@
 #include <asm/ptrace.h>
 #include <asm/localtimer.h>
 #include <asm/smp_plat.h>
+#include <asm/virt.h>
 
 /*
  * as from 2.5, kernels no longer have an init_tasks structure
@@ -287,6 +288,8 @@ void __init smp_cpus_done(unsigned int max_cpus)
 	       num_online_cpus(),
 	       bogosum / (500000/HZ),
 	       (bogosum / (5000/HZ)) % 100);
+
+	hyp_mode_check();
 }
 
 void __init smp_prepare_boot_cpu(void)
-- 
1.7.12

[PATCH v2 6/7] ARM: virt: Add CONFIG_ARM_VIRT_EXT option

[Marc Zyngier]

From: Dave Martin <dave.martin@linaro.org>

It is now possible to enable the virtualization extention support.

Signed-off-by: Dave Martin <dave.martin@linaro.org>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm/mm/Kconfig | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig
index 101b968..c9a4963 100644
--- a/arch/arm/mm/Kconfig
+++ b/arch/arm/mm/Kconfig
@@ -624,6 +624,23 @@ config ARM_THUMBEE
 	  Say Y here if you have a CPU with the ThumbEE extension and code to
 	  make use of it. Say N for code that can run on CPUs without ThumbEE.
 
+config ARM_VIRT_EXT
+	bool "Native support for the ARM Virtualization Extensions"
+	depends on MMU && CPU_V7
+	help
+	  Enable the kernel to make use of the ARM Virtualization
+	  Extensions to install hypervisors without run-time firmware
+	  assistance.
+
+	  A compliant bootloader is required in order to make maximum
+	  use of this feature.  Refer to Documentation/arm/Booting for
+	  details.
+
+	  It is safe to enable this option even if the kernel may not be
+	  booted in HYP mode, may not have support for the
+	  virtualization extensions, or may be booted with a
+	  non-compliant bootloader.
+
 config SWP_EMULATE
 	bool "Emulate SWP/SWPB instructions"
 	depends on !CPU_USE_DOMAINS && CPU_V7
-- 
1.7.12

[PATCH v2 7/7] ARM: virt: arch_timers: enable access to physical timers

[Marc Zyngier]

If booting in HYP mode, it makes sense to enable the use of the
physical timers, so the kernel can use them directly.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm/kernel/hyp-stub.S | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/arch/arm/kernel/hyp-stub.S b/arch/arm/kernel/hyp-stub.S
index 7060941..65b2417 100644
--- a/arch/arm/kernel/hyp-stub.S
+++ b/arch/arm/kernel/hyp-stub.S
@@ -142,6 +142,19 @@ THUMB(	orr	r7, #(1 << 30)	)	@ HSCTLR.TE
 	and	r7, #0x1f		@ Preserve HPMN
 	mcr	p15, 4, r7, c1, c1, 1	@ HDCR
 
+#if !defined(ZIMAGE) && defined(CONFIG_ARM_ARCH_TIMER)
+	@ make CNTP_* and CNTPCT accessible from PL1
+	mrc	p15, 0, r7, c0, c1, 1	@ ID_PFR1
+	lsr	r7, #16
+	and	r7, #0xf
+	cmp	r7, #1
+	bne	1f
+	mrc	p15, 4, r7, c14, c1, 0	@ CNTHCTL
+	orr	r7, r7, #3		@ PL1PCEN | PL1PCTEN
+	mcr	p15, 4, r7, c14, c1, 0	@ CNTHCTL
+1:
+#endif
+
 	bic	r7, r4, #MODE_MASK
 	orr	r7, r7, #SVC_MODE
 THUMB(	orr	r7, r7, #PSR_T_BIT	)
-- 
1.7.12