bootwrapper can't be compiled for cubieboard2

bootwrapper can't be compiled for cubieboard2

[Josh Zhao]

Hi Ian,
I try compile the bootwrapper for cubieboard2 like this:
 joshzhao@joshzhao-ThinkCentre-M58p:~/project/Xen/A20/boot-wrapper$
make cubieboard2_defconfig  CROSS_COMPILE=arm-linux-gnueabihf-
#
# configuration written to .config
#
joshzhao@joshzhao-ThinkCentre-M58p:~/project/Xen/A20/boot-wrapper$
make CROSS_COMPILE=arm-linux-gnueabihf-
make -C scripts/kconfig -f Makefile.bootwrapper silentoldconfig
cd ../../ && scripts/kconfig/conf --silentoldconfig
/home/joshzhao/project/Xen/A20/boot-wrapper/Kconfig
make: Nothing to be done for `_all'.
joshzhao@joshzhao-ThinkCentre-M58p:~/project/Xen/A20/boot-wrapper$

Re: bootwrapper can't be compiled for cubieboard2

[Ian Campbell]

On Fri, 2013-08-30 at 15:12 +0800, Josh Zhao wrote:
> Hi Ian,
> I try compile the bootwrapper for cubieboard2 like this:

Hi Josh,

cubieboards2 support is not really in place yet, so I'm afraid there
will currently be rough edges like this.

That said I think this worked for me, but based on the discussions I
think the better approach is to merge Andre's u-boot patches to support
entry in HYP mode into the cubieboard branch. I'll only work on
bootwrapper any further if some interesting platform comes along for
which there is no possibility of updating the bootloader, so for now you
should consider it dead/dormant.

Anyhow, I've tried inserting Andre's patches into the cubieboard2 uboot
and stumbled across a problem -- it turns out that u-boot has enabled
the MMU (SCTLR.M), apparently because it is required for the caches to
be on (see arch/arm/lib/cache-cp15.c:cache_enable). unfortunately SCTLR
is banked so when nonsec-virt.S:_secure_monitor enabled SCR.NS
everything falls apart. Naïvely enabling SCTLR.M while in monitor mode
didn't seem to help.

Calling cleanup_before_linux() early in armv7_switch_nonsec() seemed to
do the job -- Andre, do you think that is the right fix?

Not sure why this didn't affect vexpress etc -- perhaps caches aren't
enabled for that platform?

Even with this fixed things don't quite work for me, I don't quite
remember how/why (I last looked at this 7 days ago). I've been trying to
set aside Friday as "cubieboard hacking day", but I've ended up with a
bunch of meetings today, hopefully I'll still manage to investigate a
bit.

Ian.

Re: bootwrapper can't be compiled for cubieboard2

[Andre Przywara]

On 08/30/2013 10:06 AM, Ian Campbell wrote:
> On Fri, 2013-08-30 at 15:12 +0800, Josh Zhao wrote:
>> Hi Ian,
>> I try compile the bootwrapper for cubieboard2 like this:
>
> Hi Josh,
>
> cubieboards2 support is not really in place yet, so I'm afraid there
> will currently be rough edges like this.
>
> That said I think this worked for me, but based on the discussions I
> think the better approach is to merge Andre's u-boot patches to support
> entry in HYP mode into the cubieboard branch. I'll only work on
> bootwrapper any further if some interesting platform comes along for
> which there is no possibility of updating the bootloader, so for now you
> should consider it dead/dormant.
>
> Anyhow, I've tried inserting Andre's patches into the cubieboard2 uboot
> and stumbled across a problem -- it turns out that u-boot has enabled
> the MMU (SCTLR.M), apparently because it is required for the caches to
> be on (see arch/arm/lib/cache-cp15.c:cache_enable). unfortunately SCTLR
> is banked so when nonsec-virt.S:_secure_monitor enabled SCR.NS
> everything falls apart. Naïvely enabling SCTLR.M while in monitor mode
> didn't seem to help.

Ah, good point. My whole code approach was relying on the fact that 
u-boot runs with MMU _off_. With the MMU differences in HYP mode we may 
run into troubles here, it may work if we switch late enough, though.

> Calling cleanup_before_linux() early in armv7_switch_nonsec() seemed to
> do the job -- Andre, do you think that is the right fix?

Do you have a pointer to a u-boot repository with the cubieboard 
patches? Then I'd take a look.

> Not sure why this didn't affect vexpress etc -- perhaps caches aren't
> enabled for that platform?

AFAIK it's rather normal for u-boot to run with at least some caches 
off. These only get turned on if really necessary.
So I'd refrain from enabling the MMU just to support some cache levels 
on a bootloader.

Thanks for looking at this!
Andre.

> Even with this fixed things don't quite work for me, I don't quite
> remember how/why (I last looked at this 7 days ago). I've been trying to
> set aside Friday as "cubieboard hacking day", but I've ended up with a
> bunch of meetings today, hopefully I'll still manage to investigate a
> bit.
>
> Ian.
>
>

Re: bootwrapper can't be compiled for cubieboard2

[Ian Campbell]

On Fri, 2013-08-30 at 10:27 +0200, Andre Przywara wrote:
> On 08/30/2013 10:06 AM, Ian Campbell wrote:
> > On Fri, 2013-08-30 at 15:12 +0800, Josh Zhao wrote:
> >> Hi Ian,
> >> I try compile the bootwrapper for cubieboard2 like this:
> >
> > Hi Josh,
> >
> > cubieboards2 support is not really in place yet, so I'm afraid there
> > will currently be rough edges like this.
> >
> > That said I think this worked for me, but based on the discussions I
> > think the better approach is to merge Andre's u-boot patches to support
> > entry in HYP mode into the cubieboard branch. I'll only work on
> > bootwrapper any further if some interesting platform comes along for
> > which there is no possibility of updating the bootloader, so for now you
> > should consider it dead/dormant.
> >
> > Anyhow, I've tried inserting Andre's patches into the cubieboard2 uboot
> > and stumbled across a problem -- it turns out that u-boot has enabled
> > the MMU (SCTLR.M), apparently because it is required for the caches to
> > be on (see arch/arm/lib/cache-cp15.c:cache_enable). unfortunately SCTLR
> > is banked so when nonsec-virt.S:_secure_monitor enabled SCR.NS
> > everything falls apart. Naïvely enabling SCTLR.M while in monitor mode
> > didn't seem to help.
> 
> Ah, good point. My whole code approach was relying on the fact that 
> u-boot runs with MMU _off_. With the MMU differences in HYP mode we may 
> run into troubles here, it may work if we switch late enough, though.

This new "Falcon mode" thing seems to be driving some changes here, in
particular it seems that turning on the caches can dramatically speed up
boot in some circumstance.

> 
> > Calling cleanup_before_linux() early in armv7_switch_nonsec() seemed to
> > do the job -- Andre, do you think that is the right fix?
> 
> Do you have a pointer to a u-boot repository with the cubieboard 
> patches? Then I'd take a look.

https://github.com/hno/u-boot.git branch wip/a20
specifically 80fd9a5c5b87ba2f48f4a71b666839870e780be6 and your hypmode
v4 5068b337518617586f2e51b6d616c54dbec4fa62

It looks like I should switch to the sunxi branch of
https://github.com/linux-sunxi/u-boot-sunxi.git though, it includes the
above and is more up to date.

I'll try that shortly and let you know how I get on.

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: bootwrapper can't be compiled for cubieboard2

[Ian Campbell]

On Fri, 2013-08-30 at 10:51 +0100, Ian Campbell wrote:
> I'll try that shortly and let you know how I get on. 

Success! My failure previously was not getting the kernel command line
in the right place, so no printk...

Starting from
https://github.com/linux-sunxi/u-boot-sunxi.git sunxi
I merged your stuff:
git pull git://git.linaro.org/people/aprzywara/u-boot.git hypmode_v4
and applied the patch below (NB: u-boot doesn't support SMP on this
platform yet, hence stubbing out the kicking of secondaries etc) and
https://groups.google.com/forum/#!topic/linux-sunxi/7oS0CM2WSy4.

Then built with:

$ make Cubieboard2_FEL CROSS_COMPILE=arm-linux-gnueabihf- 

Then I could boot via FEL (http://linux-sunxi.org/FEL/USBBoot) using
usb-boot from git://github.com/linux-sunxi/sunxi-tools.git:
usb-boot ~/devel/u-boot.git/spl/u-boot-spl.bin ~/devel/u-boot.git/u-boot.bin uImage

where uImage is the zImage+dtb (cat into tmp) and mkimage -A arm -O
linux -T kernel -a 0x40008000 -e 0x40008000 -C none -d tmp uImage

Linux says:
[    0.000000] CPU: All CPU(s) started in HYP mode.
[    0.000000] CPU: Virtualization extensions available.

(All == all 1)

Kernel is https://github.com/mripard/linux.git sunxi-next-a20-cubie2
multi_v7_defconfig + CONFIG_DEBUG_LL w/ CONFIG_DEBUG_SUNXI_UART0 and
CONFIG_EARLY_PRINTK.
I used appended dtb and set /chosen/bootargs = "console=ttyS0,115200
earlyprintk" in arch/arm/boot/dts/sun7i-a20-cubieboard2.dts (only
because usb-boot doesn't happen to load dtb and pass it to bootm, will
fix that up...)

Now to try Xen!

Ian.

commit 9267730ab1c3b6be5cf9118b2ce4e708e9d4ba66
Author: Ian Campbell <ijc@hellion.org.uk>
Date:   Fri Aug 30 13:52:07 2013 +0100

    sunxi: Enable hyp mode boot on sunxi platforms
    
    Required a call to cleanup_before_linux in order to disable caches before
    switching to NS world, otherwise we switch to an incorrecly configured banked
    version of SCTLR.
    
    Signed-off-by: Ian Campbell <ijc@hellion.org.uk>

diff --git a/arch/arm/cpu/armv7/virt-v7.c b/arch/arm/cpu/armv7/virt-v7.c
index 4dc8c45..a76dcf7 100644
--- a/arch/arm/cpu/armv7/virt-v7.c
+++ b/arch/arm/cpu/armv7/virt-v7.c
@@ -132,6 +132,8 @@ int armv7_switch_nonsec(void)
 	unsigned int reg;
 	unsigned itlinesnr, i;
 
+	cleanup_before_linux();
+
 	/* check whether the CPU supports the security extensions */
 	reg = read_id_pfr1();
 	if ((reg & 0xF0) == 0) {
diff --git a/board/sunxi/Makefile b/board/sunxi/Makefile
index 4780c99..d756091 100644
--- a/board/sunxi/Makefile
+++ b/board/sunxi/Makefile
@@ -76,9 +76,13 @@ COBJS-$(CONFIG_UHOST_U1A)	+= dram_sun4i_360_1024_iow8.o
 COBJS-$(CONFIG_WOBO_I5)		+= dram_wobo_i5.o
 COBJS-$(CONFIG_XZPAD700)	+= dram_xzpad700.o
 
+ifneq ($(CONFIG_ARMV7_NONSEC)$(CONFIG_ARMV7_VIRT),)
+SOBJS                           := smp.o
+endif
+
 COBJS	:= $(COBJS-y)
-SRCS	:= $(COBJS:.o=.c)
-OBJS	:= $(addprefix $(obj),$(COBJS))
+SRCS	:= $(COBJS:.o=.c) $(SOBJS:.o=.S)
+OBJS	:= $(addprefix $(obj),$(COBJS) $(SOBJS))
 
 all:	$(LIB)
 
diff --git a/board/sunxi/board.c b/board/sunxi/board.c
index 50c07e2..4b8c732 100755
--- a/board/sunxi/board.c
+++ b/board/sunxi/board.c
@@ -40,6 +40,20 @@
 
 DECLARE_GLOBAL_DATA_PTR;
 
+/* Setting the address at which secondary cores start from.
+ * SMP currently unsupported.
+ */
+void smp_set_core_boot_addr(unsigned long addr, int corenr)
+{
+	printf("WARNING: ignoring attempt to set core boot address %lx on core %d\n",
+	       addr, corenr);
+}
+
+void smp_kick_all_cpus(void)
+{
+	printf("WARNING: Not kicking secondary CPUs\n");
+}
+
 /* add board specific code here */
 int board_init(void)
 {
diff --git a/board/sunxi/smp.S b/board/sunxi/smp.S
new file mode 100644
index 0000000..6a6e08c
--- /dev/null
+++ b/board/sunxi/smp.S
@@ -0,0 +1,33 @@
+/*
+ * code for redirecting secondary cores to their start address
+ *
+ * Copyright (c) 2013	Andre Przywara <andre.przywara@linaro.org>
+ *
+ * See file CREDITS for list of people who contributed to this
+ * project.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation; either version 2 of
+ * the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.	 See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+ * MA 02111-1307 USA
+ */
+
+#include <config.h>
+#include <linux/linkage.h>
+
+/* void _smp_waitloop(unsigned previous_address); */
+ENTRY(smp_waitloop)
+	/* SMP currently unsupported */
+1:	wfi
+	b 1b
+ENDPROC(smp_waitloop)
diff --git a/include/configs/sun7i.h b/include/configs/sun7i.h
index a6ede2a..f0d9222 100644
--- a/include/configs/sun7i.h
+++ b/include/configs/sun7i.h
@@ -34,6 +34,8 @@
 #define CONFIG_SYS_PROMPT		"sun7i# "
 #define CONFIG_MACH_TYPE		4283
 
+#define CONFIG_ARMV7_VIRT
+
 #if defined(CONFIG_SYS_SECONDARY_ON)
 #define CONFIG_BOARD_POSTCLK_INIT 1
 #endif

Re: bootwrapper can't be compiled for cubieboard2

[Andre Przywara]

On 08/30/2013 03:12 PM, Ian Campbell wrote:
> On Fri, 2013-08-30 at 10:51 +0100, Ian Campbell wrote:
>> I'll try that shortly and let you know how I get on.
>
> Success! My failure previously was not getting the kernel command line
> in the right place, so no printk...

Excellent! Great news! Seems I need to shell out those 70 EUR now ;-)

Thanks!
Andre.

P.S. Keep my fingers cross for the Xen boot.

> Starting from
> https://github.com/linux-sunxi/u-boot-sunxi.git sunxi
> I merged your stuff:
> git pull git://git.linaro.org/people/aprzywara/u-boot.git hypmode_v4
> and applied the patch below (NB: u-boot doesn't support SMP on this
> platform yet, hence stubbing out the kicking of secondaries etc) and
> https://groups.google.com/forum/#!topic/linux-sunxi/7oS0CM2WSy4.
>
> Then built with:
>
> $ make Cubieboard2_FEL CROSS_COMPILE=arm-linux-gnueabihf-
>
> Then I could boot via FEL (http://linux-sunxi.org/FEL/USBBoot) using
> usb-boot from git://github.com/linux-sunxi/sunxi-tools.git:
> usb-boot ~/devel/u-boot.git/spl/u-boot-spl.bin ~/devel/u-boot.git/u-boot.bin uImage
>
> where uImage is the zImage+dtb (cat into tmp) and mkimage -A arm -O
> linux -T kernel -a 0x40008000 -e 0x40008000 -C none -d tmp uImage
>
> Linux says:
> [    0.000000] CPU: All CPU(s) started in HYP mode.
> [    0.000000] CPU: Virtualization extensions available.
>
> (All == all 1)
>
> Kernel is https://github.com/mripard/linux.git sunxi-next-a20-cubie2
> multi_v7_defconfig + CONFIG_DEBUG_LL w/ CONFIG_DEBUG_SUNXI_UART0 and
> CONFIG_EARLY_PRINTK.
> I used appended dtb and set /chosen/bootargs = "console=ttyS0,115200
> earlyprintk" in arch/arm/boot/dts/sun7i-a20-cubieboard2.dts (only
> because usb-boot doesn't happen to load dtb and pass it to bootm, will
> fix that up...)
>
> Now to try Xen!
>
> Ian.
>
> commit 9267730ab1c3b6be5cf9118b2ce4e708e9d4ba66
> Author: Ian Campbell <ijc@hellion.org.uk>
> Date:   Fri Aug 30 13:52:07 2013 +0100
>
>      sunxi: Enable hyp mode boot on sunxi platforms
>
>      Required a call to cleanup_before_linux in order to disable caches before
>      switching to NS world, otherwise we switch to an incorrecly configured banked
>      version of SCTLR.
>
>      Signed-off-by: Ian Campbell <ijc@hellion.org.uk>
>
> diff --git a/arch/arm/cpu/armv7/virt-v7.c b/arch/arm/cpu/armv7/virt-v7.c
> index 4dc8c45..a76dcf7 100644
> --- a/arch/arm/cpu/armv7/virt-v7.c
> +++ b/arch/arm/cpu/armv7/virt-v7.c
> @@ -132,6 +132,8 @@ int armv7_switch_nonsec(void)
>   	unsigned int reg;
>   	unsigned itlinesnr, i;
>
> +	cleanup_before_linux();
> +
>   	/* check whether the CPU supports the security extensions */
>   	reg = read_id_pfr1();
>   	if ((reg & 0xF0) == 0) {
> diff --git a/board/sunxi/Makefile b/board/sunxi/Makefile
> index 4780c99..d756091 100644
> --- a/board/sunxi/Makefile
> +++ b/board/sunxi/Makefile
> @@ -76,9 +76,13 @@ COBJS-$(CONFIG_UHOST_U1A)	+= dram_sun4i_360_1024_iow8.o
>   COBJS-$(CONFIG_WOBO_I5)		+= dram_wobo_i5.o
>   COBJS-$(CONFIG_XZPAD700)	+= dram_xzpad700.o
>
> +ifneq ($(CONFIG_ARMV7_NONSEC)$(CONFIG_ARMV7_VIRT),)
> +SOBJS                           := smp.o
> +endif
> +
>   COBJS	:= $(COBJS-y)
> -SRCS	:= $(COBJS:.o=.c)
> -OBJS	:= $(addprefix $(obj),$(COBJS))
> +SRCS	:= $(COBJS:.o=.c) $(SOBJS:.o=.S)
> +OBJS	:= $(addprefix $(obj),$(COBJS) $(SOBJS))
>
>   all:	$(LIB)
>
> diff --git a/board/sunxi/board.c b/board/sunxi/board.c
> index 50c07e2..4b8c732 100755
> --- a/board/sunxi/board.c
> +++ b/board/sunxi/board.c
> @@ -40,6 +40,20 @@
>
>   DECLARE_GLOBAL_DATA_PTR;
>
> +/* Setting the address at which secondary cores start from.
> + * SMP currently unsupported.
> + */
> +void smp_set_core_boot_addr(unsigned long addr, int corenr)
> +{
> +	printf("WARNING: ignoring attempt to set core boot address %lx on core %d\n",
> +	       addr, corenr);
> +}
> +
> +void smp_kick_all_cpus(void)
> +{
> +	printf("WARNING: Not kicking secondary CPUs\n");
> +}
> +
>   /* add board specific code here */
>   int board_init(void)
>   {
> diff --git a/board/sunxi/smp.S b/board/sunxi/smp.S
> new file mode 100644
> index 0000000..6a6e08c
> --- /dev/null
> +++ b/board/sunxi/smp.S
> @@ -0,0 +1,33 @@
> +/*
> + * code for redirecting secondary cores to their start address
> + *
> + * Copyright (c) 2013	Andre Przywara <andre.przywara@linaro.org>
> + *
> + * See file CREDITS for list of people who contributed to this
> + * project.
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License as
> + * published by the Free Software Foundation; either version 2 of
> + * the License, or (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.	 See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program; if not, write to the Free Software
> + * Foundation, Inc., 59 Temple Place, Suite 330, Boston,
> + * MA 02111-1307 USA
> + */
> +
> +#include <config.h>
> +#include <linux/linkage.h>
> +
> +/* void _smp_waitloop(unsigned previous_address); */
> +ENTRY(smp_waitloop)
> +	/* SMP currently unsupported */
> +1:	wfi
> +	b 1b
> +ENDPROC(smp_waitloop)
> diff --git a/include/configs/sun7i.h b/include/configs/sun7i.h
> index a6ede2a..f0d9222 100644
> --- a/include/configs/sun7i.h
> +++ b/include/configs/sun7i.h
> @@ -34,6 +34,8 @@
>   #define CONFIG_SYS_PROMPT		"sun7i# "
>   #define CONFIG_MACH_TYPE		4283
>
> +#define CONFIG_ARMV7_VIRT
> +
>   #if defined(CONFIG_SYS_SECONDARY_ON)
>   #define CONFIG_BOARD_POSTCLK_INIT 1
>   #endif
>
>

Re: bootwrapper can't be compiled for cubieboard2

[Josh Zhao]

2013/8/30 Ian Campbell <ian.campbell@citrix.com>:
> On Fri, 2013-08-30 at 10:51 +0100, Ian Campbell wrote:
>> I'll try that shortly and let you know how I get on.
>
> Success! My failure previously was not getting the kernel command line
> in the right place, so no printk...
>
> Starting from
> https://github.com/linux-sunxi/u-boot-sunxi.git sunxi
> I merged your stuff:
> git pull git://git.linaro.org/people/aprzywara/u-boot.git hypmode_v4
> and applied the patch below (NB: u-boot doesn't support SMP on this
> platform yet, hence stubbing out the kicking of secondaries etc) and
> https://groups.google.com/forum/#!topic/linux-sunxi/7oS0CM2WSy4.
>
> Then built with:
>
> $ make Cubieboard2_FEL CROSS_COMPILE=arm-linux-gnueabihf-
>
> Then I could boot via FEL (http://linux-sunxi.org/FEL/USBBoot) using
> usb-boot from git://github.com/linux-sunxi/sunxi-tools.git:
> usb-boot ~/devel/u-boot.git/spl/u-boot-spl.bin ~/devel/u-boot.git/u-boot.bin uImage
>
> where uImage is the zImage+dtb (cat into tmp) and mkimage -A arm -O
> linux -T kernel -a 0x40008000 -e 0x40008000 -C none -d tmp uImage
>
> Linux says:
> [    0.000000] CPU: All CPU(s) started in HYP mode.
> [    0.000000] CPU: Virtualization extensions available.
>
> (All == all 1)
>
> Kernel is https://github.com/mripard/linux.git sunxi-next-a20-cubie2
> multi_v7_defconfig + CONFIG_DEBUG_LL w/ CONFIG_DEBUG_SUNXI_UART0 and
> CONFIG_EARLY_PRINTK.
> I used appended dtb and set /chosen/bootargs = "console=ttyS0,115200
> earlyprintk" in arch/arm/boot/dts/sun7i-a20-cubieboard2.dts (only
> because usb-boot doesn't happen to load dtb and pass it to bootm, will
> fix that up...)
>

That's cool. Following your guide,  the u-boot can enter into hyper mode now.

Re: bootwrapper can't be compiled for cubieboard2

[Ian Campbell]

On Mon, 2013-09-02 at 11:59 +0800, Josh Zhao wrote:
> That's cool. Following your guide,  the u-boot can enter into hyper mode now.

Great!

I've got it to the point of dom0 failing to find a root filesystem,
which is because there is no MMC, SATA or USB driver in mainline yet --
so no storage.

I'm rather busy at the moment, so I kind of hoping that by the time I
get back to this someone will have picked up the allwinner sata driver
and made it fit upstream.

Ian.