xen/arm: Bootwrapper update to support PSCI and GICv3

xen/arm: Bootwrapper update to support PSCI and GICv3

[Julien Grall]

Hello all,

I've been working on updating our aarch64 bootwrapper
to support new feature such as PSCI and GICv3.

Rather than porting the feature from the Linux bootwrapper [1].
I've added support of Xen on top of the Linux repo.

Below an example to configure bootwrapper with GICv3 and PSCI for
the foundation model:

42sh> ./configure --host=aarch64-linux-gnu		\
--with-kernel-dir=$HOME/linux-build/aarch64	\
--with-dtb=$HOME/arm-trusted-firmware/fdts/fvp-foundation-gicv3-psci.dtb \
--with-cmdline="console=hvc0 earlycon=pl011,0x1c090000 init=/root/init.sh root=/dev/vda" \
--enable-psci --with-xen-cmdline="dtuart=serial0 console=dtuart no-bootscrub"		 \
--with-xen="$HOME/xen" --enable-gicv3
42sh> make

Make will produce a xen-system.axf which is the image used to boot
Xen on the model.

The branch with the new version is:
git://xenbits.xen.org/people/julieng/boot-wrapper-aarch64.git branch xen

Ian, can you update your repo with this new version?

Regards,

[1] git://git.kernel.org/pub/scm/linux/kernel/git/cmarinas/boot-wrapper-aarch64.git

-- 
Julien Grall

Re: xen/arm: Bootwrapper update to support PSCI and GICv3

[Ian Campbell]

On Thu, 2014-11-06 at 12:48 +0000, Julien Grall wrote:
> Hello all,
> 
> I've been working on updating our aarch64 bootwrapper
> to support new feature such as PSCI and GICv3.
> 
> Rather than porting the feature from the Linux bootwrapper [1].
> I've added support of Xen on top of the Linux repo.
> 
> Below an example to configure bootwrapper with GICv3 and PSCI for
> the foundation model:
> 
> 42sh> ./configure --host=aarch64-linux-gnu		\
> --with-kernel-dir=$HOME/linux-build/aarch64	\
> --with-dtb=$HOME/arm-trusted-firmware/fdts/fvp-foundation-gicv3-psci.dtb \
> --with-cmdline="console=hvc0 earlycon=pl011,0x1c090000 init=/root/init.sh root=/dev/vda" \
> --enable-psci --with-xen-cmdline="dtuart=serial0 console=dtuart no-bootscrub"		 \
> --with-xen="$HOME/xen" --enable-gicv3
> 42sh> make
> 
> Make will produce a xen-system.axf which is the image used to boot
> Xen on the model.
> 
> The branch with the new version is:
> git://xenbits.xen.org/people/julieng/boot-wrapper-aarch64.git branch xen
> 
> Ian, can you update your repo with this new version?

FWIW I've been happily using
https://git.linaro.org/people/christoffer.dall/boot-wrapper-aarch64.git/shortlog/refs/heads/xen-psci-support at 7e702c7892d0965f459a61d36e4c8f1a9d6ee6df plus the following fixup (which I've been remiss in not sending out).

Given that we are now in a state where the patches appear to be nicely
in keeping with the wrapper's architecture and therefore potentially
upstreamable I'd like to at least have that conversation with the
maintainers (probably via a patch set submission) before we carry on
with a fork.

Ian.

diff --git a/Makefile.am b/Makefile.am
index 9b6c7e3..6c2786e 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -69,8 +69,9 @@ XEN_OFFSET	:= 0xA00000
 DOM0_OFFSET	:= $(shell echo $$(($(PHYS_OFFSET) + $(KERNEL_OFFSET))))
 XEN_BOOTARGS	:= xen,xen-bootargs = \"$(BOOTARGS)\";			\
 		   module@1 {						\
+			bootargs = \"$(CMDLINE)\";			\
 			compatible = \"xen,linux-zimage\", \"xen,multiboot-module\"; \
-			reg = <$(DOM0_OFFSET) 0x800000>;		\
+			reg = <0 $(DOM0_OFFSET) 0 0x800000>;		\
 		   };
 endif
 
@@ -97,7 +98,10 @@ all: $(IMAGE) $(XIMAGE)
 
 CLEANFILES = $(IMAGE) boot.o cache.o $(GIC) mmu.o ns.o $(BOOTMETHOD) model.lds fdt.dtb
 
-$(IMAGE): boot.o cache.o $(GIC) mmu.o ns.o $(BOOTMETHOD) model.lds fdt.dtb $(KERNEL_IMAGE) $(FILESYSTEM) $(XEN_IMAGE)
+if XEN
+XEN_IMAGE_DEP = $(XEN_IMAGE)
+endif
+$(IMAGE): boot.o cache.o $(GIC) mmu.o ns.o $(BOOTMETHOD) model.lds fdt.dtb $(KERNEL_IMAGE) $(FILESYSTEM) $(XEN_IMAGE_DEP)
 	$(LD) -o $@ --script=model.lds
 
 %.o: %.S Makefile
diff --git a/configure.ac b/configure.ac
index 2f31fab..44b3bf0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -75,13 +75,14 @@ AC_ARG_WITH([initrd],
 AC_SUBST([FILESYSTEM], [$USE_INITRD])
 AM_CONDITIONAL([INITRD], [test "x$USE_INITRD" != "x"])
 
-C_CMDLINE="console=ttyAMA0 earlyprintk=pl011,0x1c090000"
+AS_IF([test "x$XEN_IMAGE" = "no"],[C_CONSOLE="ttyAMA0"],[C_CONSOLE="hvc0"])
+C_CMDLINE="console=$C_CONSOLE earlyprintk=pl011,0x1c090000"
 AC_ARG_WITH([cmdline],
 	AS_HELP_STRING([--with-cmdline], [set a command line for the kernel]),
 	[C_CMDLINE=$withval])
 AC_SUBST([CMDLINE], [$C_CMDLINE])
 
-X_BOOTARGS="console=dtuart dtuart=serial0"
+X_BOOTARGS="console=dtuart dtuart=serial0 no-bootscrub"
 AC_ARG_WITH([xen-bootargs],
 	AS_HELP_STRING([--with-xen-bootargs], [set Xen bootargs]),
 	[X_BOOTARGS=$withval])

Re: xen/arm: Bootwrapper update to support PSCI and GICv3

[Christoffer Dall]

On Mon, Nov 10, 2014 at 12:34 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Thu, 2014-11-06 at 12:48 +0000, Julien Grall wrote:
>> Hello all,
>>
>> I've been working on updating our aarch64 bootwrapper
>> to support new feature such as PSCI and GICv3.
>>
>> Rather than porting the feature from the Linux bootwrapper [1].
>> I've added support of Xen on top of the Linux repo.
>>
>> Below an example to configure bootwrapper with GICv3 and PSCI for
>> the foundation model:
>>
>> 42sh> ./configure --host=aarch64-linux-gnu            \
>> --with-kernel-dir=$HOME/linux-build/aarch64   \
>> --with-dtb=$HOME/arm-trusted-firmware/fdts/fvp-foundation-gicv3-psci.dtb \
>> --with-cmdline="console=hvc0 earlycon=pl011,0x1c090000 init=/root/init.sh root=/dev/vda" \
>> --enable-psci --with-xen-cmdline="dtuart=serial0 console=dtuart no-bootscrub"          \
>> --with-xen="$HOME/xen" --enable-gicv3
>> 42sh> make
>>
>> Make will produce a xen-system.axf which is the image used to boot
>> Xen on the model.
>>
>> The branch with the new version is:
>> git://xenbits.xen.org/people/julieng/boot-wrapper-aarch64.git branch xen
>>
>> Ian, can you update your repo with this new version?
>
> FWIW I've been happily using
> https://git.linaro.org/people/christoffer.dall/boot-wrapper-aarch64.git/shortlog/refs/heads/xen-psci-support at 7e702c7892d0965f459a61d36e4c8f1a9d6ee6df plus the following fixup (which I've been remiss in not sending out).

can you send this to me as a proper patch, then I'll include it in my
tree and test this out and send a patch set upstream?

>
> Given that we are now in a state where the patches appear to be nicely
> in keeping with the wrapper's architecture and therefore potentially
> upstreamable I'd like to at least have that conversation with the
> maintainers (probably via a patch set submission) before we carry on
> with a fork.
>
Agreed.

-Christoffer

[PATCH 1/4] xen: Correction to module@1 (dom0 kernel) DT node.

[Ian Campbell]

- Include bootargs (kernel command line) property.
- Update reg property to match #address-cells and #size-cells in the DTB (both
  2).

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Makefile.am |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/Makefile.am b/Makefile.am
index 9b6c7e3..ed5acbb 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -69,8 +69,9 @@ XEN_OFFSET	:= 0xA00000
 DOM0_OFFSET	:= $(shell echo $$(($(PHYS_OFFSET) + $(KERNEL_OFFSET))))
 XEN_BOOTARGS	:= xen,xen-bootargs = \"$(BOOTARGS)\";			\
 		   module@1 {						\
+			bootargs = \"$(CMDLINE)\";			\
 			compatible = \"xen,linux-zimage\", \"xen,multiboot-module\"; \
-			reg = <$(DOM0_OFFSET) 0x800000>;		\
+			reg = <0 $(DOM0_OFFSET) 0 0x800000>;		\
 		   };
 endif
 
-- 
1.7.10.4

[PATCH 2/4] Fix build when Xen is not enabled

[Ian Campbell]

If Xen isn't enabled then XEN_IMAGE ends up as "no.o", which obviously doesn't
exist. Handle the dependency explicitly.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Makefile.am |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/Makefile.am b/Makefile.am
index ed5acbb..6c2786e 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -98,7 +98,10 @@ all: $(IMAGE) $(XIMAGE)
 
 CLEANFILES = $(IMAGE) boot.o cache.o $(GIC) mmu.o ns.o $(BOOTMETHOD) model.lds fdt.dtb
 
-$(IMAGE): boot.o cache.o $(GIC) mmu.o ns.o $(BOOTMETHOD) model.lds fdt.dtb $(KERNEL_IMAGE) $(FILESYSTEM) $(XEN_IMAGE)
+if XEN
+XEN_IMAGE_DEP = $(XEN_IMAGE)
+endif
+$(IMAGE): boot.o cache.o $(GIC) mmu.o ns.o $(BOOTMETHOD) model.lds fdt.dtb $(KERNEL_IMAGE) $(FILESYSTEM) $(XEN_IMAGE_DEP)
 	$(LD) -o $@ --script=model.lds
 
 %.o: %.S Makefile
-- 
1.7.10.4

[PATCH 3/4] Select correct dom0 console depending on whether Xen is enabled or not

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 configure.ac |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/configure.ac b/configure.ac
index 2f31fab..d292f4d 100644
--- a/configure.ac
+++ b/configure.ac
@@ -75,7 +75,8 @@ AC_ARG_WITH([initrd],
 AC_SUBST([FILESYSTEM], [$USE_INITRD])
 AM_CONDITIONAL([INITRD], [test "x$USE_INITRD" != "x"])
 
-C_CMDLINE="console=ttyAMA0 earlyprintk=pl011,0x1c090000"
+AS_IF([test "x$XEN_IMAGE" = "no"],[C_CONSOLE="ttyAMA0"],[C_CONSOLE="hvc0"])
+C_CMDLINE="console=$C_CONSOLE earlyprintk=pl011,0x1c090000"
 AC_ARG_WITH([cmdline],
 	AS_HELP_STRING([--with-cmdline], [set a command line for the kernel]),
 	[C_CMDLINE=$withval])
-- 
1.7.10.4

[PATCH 4/4] xen: Disable boot scrub

[Ian Campbell]

It's really (really!) slow on models and the security concerns don't really
apply.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 configure.ac |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/configure.ac b/configure.ac
index d292f4d..44b3bf0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -82,7 +82,7 @@ AC_ARG_WITH([cmdline],
 	[C_CMDLINE=$withval])
 AC_SUBST([CMDLINE], [$C_CMDLINE])
 
-X_BOOTARGS="console=dtuart dtuart=serial0"
+X_BOOTARGS="console=dtuart dtuart=serial0 no-bootscrub"
 AC_ARG_WITH([xen-bootargs],
 	AS_HELP_STRING([--with-xen-bootargs], [set Xen bootargs]),
 	[X_BOOTARGS=$withval])
-- 
1.7.10.4

Re: xen/arm: Bootwrapper update to support PSCI and GICv3

[Ian Campbell]

On Mon, 2014-11-10 at 14:35 +0100, Christoffer Dall wrote:

> can you send this to me as a proper patch, then I'll include it in my
> tree and test this out and send a patch set upstream?

I've just invoked git send-email, resulting in 4 patches.

Re: [PATCH 1/4] xen: Correction to module@1 (dom0 kernel) DT node.

[Christoffer Dall]

Hi Ian,

On Mon, Nov 10, 2014 at 02:09:58PM +0000, Ian Campbell wrote:
> - Include bootargs (kernel command line) property.

Why?

The logic I was going for was to reduce duplicate code/entries in the
DT, and if I read docs/misc/arm/device-tree/booting.txt, the fact that
we have xen,xen-bootargs but no xen,dom0-bootargs then bootargs added
further down in Makefile.am will be used.

Is this not correct or not preferred for some reason?

-Christoffer

> - Update reg property to match #address-cells and #size-cells in the DTB (both
>   2).
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  Makefile.am |    3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/Makefile.am b/Makefile.am
> index 9b6c7e3..ed5acbb 100644
> --- a/Makefile.am
> +++ b/Makefile.am
> @@ -69,8 +69,9 @@ XEN_OFFSET	:= 0xA00000
>  DOM0_OFFSET	:= $(shell echo $$(($(PHYS_OFFSET) + $(KERNEL_OFFSET))))
>  XEN_BOOTARGS	:= xen,xen-bootargs = \"$(BOOTARGS)\";			\
>  		   module@1 {						\
> +			bootargs = \"$(CMDLINE)\";			\
>  			compatible = \"xen,linux-zimage\", \"xen,multiboot-module\"; \
> -			reg = <$(DOM0_OFFSET) 0x800000>;		\
> +			reg = <0 $(DOM0_OFFSET) 0 0x800000>;		\
>  		   };
>  endif
>  
> -- 
> 1.7.10.4
> 

Re: [PATCH 2/4] Fix build when Xen is not enabled

[Christoffer Dall]

On Mon, Nov 10, 2014 at 02:09:59PM +0000, Ian Campbell wrote:
> If Xen isn't enabled then XEN_IMAGE ends up as "no.o", which obviously doesn't
> exist. Handle the dependency explicitly.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  Makefile.am |    5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/Makefile.am b/Makefile.am
> index ed5acbb..6c2786e 100644
> --- a/Makefile.am
> +++ b/Makefile.am
> @@ -98,7 +98,10 @@ all: $(IMAGE) $(XIMAGE)
>  
>  CLEANFILES = $(IMAGE) boot.o cache.o $(GIC) mmu.o ns.o $(BOOTMETHOD) model.lds fdt.dtb
>  
> -$(IMAGE): boot.o cache.o $(GIC) mmu.o ns.o $(BOOTMETHOD) model.lds fdt.dtb $(KERNEL_IMAGE) $(FILESYSTEM) $(XEN_IMAGE)
> +if XEN
> +XEN_IMAGE_DEP = $(XEN_IMAGE)
> +endif
> +$(IMAGE): boot.o cache.o $(GIC) mmu.o ns.o $(BOOTMETHOD) model.lds fdt.dtb $(KERNEL_IMAGE) $(FILESYSTEM) $(XEN_IMAGE_DEP)
>  	$(LD) -o $@ --script=model.lds
>  
>  %.o: %.S Makefile
> -- 
> 1.7.10.4
> 
I fixed this differently, see the 'xen-psci-support-for-upstream' branch in:
http://git.linaro.org/people/christoffer.dall/boot-wrapper-aarch64.git

Hopefully you're ok with that change.

-Christoffer

Re: [PATCH 1/4] xen: Correction to module@1 (dom0 kernel) DT node.

[Ian Campbell]

On Sun, 2014-11-16 at 12:26 -0800, Christoffer Dall wrote:
> Hi Ian,
> 
> On Mon, Nov 10, 2014 at 02:09:58PM +0000, Ian Campbell wrote:
> > - Include bootargs (kernel command line) property.
> 
> Why?
> 
> The logic I was going for was to reduce duplicate code/entries in the
> DT, and if I read docs/misc/arm/device-tree/booting.txt, the fact that
> we have xen,xen-bootargs but no xen,dom0-bootargs then bootargs added
> further down in Makefile.am will be used.
> 
> Is this not correct or not preferred for some reason?

I thought it wasn't working for me, but it's possible I was tripping
over something else which I fixed by hacking around in this stuff around
the same time and misattributed the fix to this change.

Looking at it now I agree that the code looks like it should work
without this change, so why don't you drop this hunk and if I find
something not working right in the new version I'll investigate again
from scratch.

Ian.

> 
> -Christoffer
> 
> > - Update reg property to match #address-cells and #size-cells in the DTB (both
> >   2).
> > 
> > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > ---
> >  Makefile.am |    3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> > 
> > diff --git a/Makefile.am b/Makefile.am
> > index 9b6c7e3..ed5acbb 100644
> > --- a/Makefile.am
> > +++ b/Makefile.am
> > @@ -69,8 +69,9 @@ XEN_OFFSET	:= 0xA00000
> >  DOM0_OFFSET	:= $(shell echo $$(($(PHYS_OFFSET) + $(KERNEL_OFFSET))))
> >  XEN_BOOTARGS	:= xen,xen-bootargs = \"$(BOOTARGS)\";			\
> >  		   module@1 {						\
> > +			bootargs = \"$(CMDLINE)\";			\
> >  			compatible = \"xen,linux-zimage\", \"xen,multiboot-module\"; \
> > -			reg = <$(DOM0_OFFSET) 0x800000>;		\
> > +			reg = <0 $(DOM0_OFFSET) 0 0x800000>;		\
> >  		   };
> >  endif
> >  
> > -- 
> > 1.7.10.4
> > 

Re: [PATCH 2/4] Fix build when Xen is not enabled

[Ian Campbell]

On Sun, 2014-11-16 at 13:03 -0800, Christoffer Dall wrote:
> On Mon, Nov 10, 2014 at 02:09:59PM +0000, Ian Campbell wrote:
> > If Xen isn't enabled then XEN_IMAGE ends up as "no.o", which obviously doesn't
> > exist. Handle the dependency explicitly.
> > 
> > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > ---
> >  Makefile.am |    5 ++++-
> >  1 file changed, 4 insertions(+), 1 deletion(-)
> > 
> > diff --git a/Makefile.am b/Makefile.am
> > index ed5acbb..6c2786e 100644
> > --- a/Makefile.am
> > +++ b/Makefile.am
> > @@ -98,7 +98,10 @@ all: $(IMAGE) $(XIMAGE)
> >  
> >  CLEANFILES = $(IMAGE) boot.o cache.o $(GIC) mmu.o ns.o $(BOOTMETHOD) model.lds fdt.dtb
> >  
> > -$(IMAGE): boot.o cache.o $(GIC) mmu.o ns.o $(BOOTMETHOD) model.lds fdt.dtb $(KERNEL_IMAGE) $(FILESYSTEM) $(XEN_IMAGE)
> > +if XEN
> > +XEN_IMAGE_DEP = $(XEN_IMAGE)
> > +endif
> > +$(IMAGE): boot.o cache.o $(GIC) mmu.o ns.o $(BOOTMETHOD) model.lds fdt.dtb $(KERNEL_IMAGE) $(FILESYSTEM) $(XEN_IMAGE_DEP)
> >  	$(LD) -o $@ --script=model.lds
> >  
> >  %.o: %.S Makefile
> > -- 
> > 1.7.10.4
> > 
> I fixed this differently, see the 'xen-psci-support-for-upstream' branch in:
> http://git.linaro.org/people/christoffer.dall/boot-wrapper-aarch64.git
> 
> Hopefully you're ok with that change.

Sure.

Ian,