[PATCH V2 for-4.5] EFI: Always use EFI command line

[PATCH V2 for-4.5] EFI: Always use EFI command line

[Roy Franz]

This patch changes the ARM EFI boot code to always use the EFI commandline,
even when loaded by GRUB, which makes it consistent with Linux EFI booting.
The code previously incorrectly skipped processing of the EFI command line when
modules are present in the loader supplied FDT and the config file is not used.
There is no change in behavior for x86 since it unconditionally uses the config
file.
Update documentation to clarify command line handling for EFI Xen.

Signed-off-by: Roy Franz <roy.franz@linaro.org>
---
Changes since V1:
* Fix use of data after free.
* Update documentation.
  (I'm not sure how I'm supposed to refer to to docs/misc/efi.markdown in
   the booting.txt.  Should I assume that it will be processed to html
   and refrence it that way?)

 docs/misc/arm/booting.txt | 3 +++
 docs/misc/efi.markdown    | 8 ++++++--
 xen/common/efi/boot.c     | 4 +++-
 3 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/docs/misc/arm/booting.txt b/docs/misc/arm/booting.txt
index 9802e5e..1193ae0 100644
--- a/docs/misc/arm/booting.txt
+++ b/docs/misc/arm/booting.txt
@@ -23,6 +23,9 @@ The exceptions to this on 32-bit ARM are as follows:
 
 There are no exception on 64-bit ARM.
 
+For 64-bit ARM, EFI booting is also supported.  See docs/misc/efi.markdown for
+a description of EFI boot.
+
 [1] linux/Documentation/arm/Booting
 Latest version: http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/arm/Booting
 
diff --git a/docs/misc/efi.markdown b/docs/misc/efi.markdown
index 5e48aa3..5a783b1 100644
--- a/docs/misc/efi.markdown
+++ b/docs/misc/efi.markdown
@@ -13,7 +13,9 @@ configuration file as described below unless a bootloader, such as GRUB, has
 loaded the modules and describes them in the device tree provided to Xen.  If a
 bootloader provides a device tree containing modules then any configuration
 files are ignored, and the bootloader is responsible for populating all
-relevant device tree nodes.
+relevant module device tree nodes, and doing any required video mode setting.
+The EFI command line is always used for Xen arguments.
+
 
 Once built, `make install-xen` will place the resulting binary directly into
 the EFI boot partition, provided `EFI_VENDOR` is set in the environment (and
@@ -104,4 +106,6 @@ and really not meant to be used together with the `-cfg=` command line option.
 Filenames must be specified relative to the location of the EFI binary.
 
 Extra options to be passed to Xen can also be specified on the command line,
-following a `--` separator option.
+following a `--` separator option.  These options from the EFI command line are
+appended to the options contained in the configuration file "options" value,
+if any, before being passed to Xen by the EFI boot code.
diff --git a/xen/common/efi/boot.c b/xen/common/efi/boot.c
index 4257341..ad9dd3a 100644
--- a/xen/common/efi/boot.c
+++ b/xen/common/efi/boot.c
@@ -930,8 +930,10 @@ efi_start(EFI_HANDLE ImageHandle, EFI_SYSTEM_TABLE *SystemTable)
         cfg.addr = 0;
 
         dir_handle->Close(dir_handle);
-
     }
+    else
+        efi_arch_handle_cmdline(argc ? *argv : NULL, options, NULL);
+
 
     if ( gop && !base_video )
     {
-- 
2.1.0

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Ian Campbell]

On Fri, 2014-10-24 at 16:41 -0700, Roy Franz wrote:
>   (I'm not sure how I'm supposed to refer to to docs/misc/efi.markdown in
>    the booting.txt.  Should I assume that it will be processed to html
>    and refrence it that way?)

This stuff kinda sucks because it all depends on where you build and
install to etc. What you've done in naming the file as it appears in the
source is fine. Other places use a reference to the canonical online
copy at http://xenbits.xen.org/docs/unstable/ which is fine too I think.

> @@ -104,4 +106,6 @@ and really not meant to be used together with the `-cfg=` command line option.
>  Filenames must be specified relative to the location of the EFI binary.
>  
>  Extra options to be passed to Xen can also be specified on the command line,
> -following a `--` separator option.
> +following a `--` separator option.

Will grub automatically insert this "--" marker? Or is there some
special handling in Xen when none of -cfg/-help/-videothing are present?
(Doesn't look to be the case)

Ian.

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Fu Wei]

On 10/25/2014 05:06 PM, Ian Campbell wrote:
> On Fri, 2014-10-24 at 16:41 -0700, Roy Franz wrote:
>>   (I'm not sure how I'm supposed to refer to to docs/misc/efi.markdown in
>>    the booting.txt.  Should I assume that it will be processed to html
>>    and refrence it that way?)
> 
> This stuff kinda sucks because it all depends on where you build and
> install to etc. What you've done in naming the file as it appears in the
> source is fine. Other places use a reference to the canonical online
> copy at http://xenbits.xen.org/docs/unstable/ which is fine too I think.
> 
>> @@ -104,4 +106,6 @@ and really not meant to be used together with the `-cfg=` command line option.
>>  Filenames must be specified relative to the location of the EFI binary.
>>  
>>  Extra options to be passed to Xen can also be specified on the command line,
>> -following a `--` separator option.
>> +following a `--` separator option.
> 
> Will grub automatically insert this "--" marker? Or is there some
> special handling in Xen when none of -cfg/-help/-videothing are present?
> (Doesn't look to be the case)

For now, I didn't make this automatically "--" injection.

Do we need to do this, or just let user do it by themselves?

In my opinion, according to the wiki page, we should make the injection?
http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/Multiboot

any thought and suggestion?? :-)

> 
> Ian.
> 


-- 
Best regards,

Fu Wei
Software Engineer From Red Hat
LEG Team
Linaro.org | Open source software for ARM SoCs
Ph: +86 186 2020 4684 (mobile)
IRC: fuwei
Skype: tekkamanninja
Room 1512, Regus One Corporate Avenue,Level 15,
One Corporate Avenue,222 Hubin Road,Huangpu District,
Shanghai,China 200021 

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Fu Wei]

On 10/25/2014 05:27 PM, Fu Wei wrote:
> On 10/25/2014 05:06 PM, Ian Campbell wrote:
>> On Fri, 2014-10-24 at 16:41 -0700, Roy Franz wrote:
>>>   (I'm not sure how I'm supposed to refer to to docs/misc/efi.markdown in
>>>    the booting.txt.  Should I assume that it will be processed to html
>>>    and refrence it that way?)
>>
>> This stuff kinda sucks because it all depends on where you build and
>> install to etc. What you've done in naming the file as it appears in the
>> source is fine. Other places use a reference to the canonical online
>> copy at http://xenbits.xen.org/docs/unstable/ which is fine too I think.
>>
>>> @@ -104,4 +106,6 @@ and really not meant to be used together with the `-cfg=` command line option.
>>>  Filenames must be specified relative to the location of the EFI binary.
>>>  
>>>  Extra options to be passed to Xen can also be specified on the command line,
>>> -following a `--` separator option.
>>> +following a `--` separator option.
>>
>> Will grub automatically insert this "--" marker? Or is there some
>> special handling in Xen when none of -cfg/-help/-videothing are present?
>> (Doesn't look to be the case)
> 
> For now, I didn't make this automatically "--" injection.
> 
> Do we need to do this, or just let user do it by themselves?
> 
> In my opinion, according to the wiki page, we should make the injection?
> http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/Multiboot
> 
> any thought and suggestion?? :-)

I have made a patch for this:
If the first argument is not "--", grub will add "--" as first argument.

But I don't think it is a good fix.

So please let me know what is your decision, then I can make the multiboot behavior right :-)
before I get your info, my multiboot patch won't take care of "--"

> 
>>
>> Ian.
>>
> 
> 


-- 
Best regards,

Fu Wei
Software Engineer From Red Hat
LEG Team
Linaro.org | Open source software for ARM SoCs
Ph: +86 186 2020 4684 (mobile)
IRC: fuwei
Skype: tekkamanninja
Room 1512, Regus One Corporate Avenue,Level 15,
One Corporate Avenue,222 Hubin Road,Huangpu District,
Shanghai,China 200021 

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Roy Franz]

On Sat, Oct 25, 2014 at 2:06 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Fri, 2014-10-24 at 16:41 -0700, Roy Franz wrote:
>>   (I'm not sure how I'm supposed to refer to to docs/misc/efi.markdown in
>>    the booting.txt.  Should I assume that it will be processed to html
>>    and refrence it that way?)
>
> This stuff kinda sucks because it all depends on where you build and
> install to etc. What you've done in naming the file as it appears in the
> source is fine. Other places use a reference to the canonical online
> copy at http://xenbits.xen.org/docs/unstable/ which is fine too I think.
>
>> @@ -104,4 +106,6 @@ and really not meant to be used together with the `-cfg=` command line option.
>>  Filenames must be specified relative to the location of the EFI binary.
>>
>>  Extra options to be passed to Xen can also be specified on the command line,
>> -following a `--` separator option.
>> +following a `--` separator option.
>
> Will grub automatically insert this "--" marker? Or is there some
> special handling in Xen when none of -cfg/-help/-videothing are present?
> (Doesn't look to be the case)
>
> Ian.
>

I'm not sure what GRUB should do.  I think that the root of the
problem is that the command line
given to Xen when booted as an "Image" file is different than when it
is booted as an EFI application.
In the Image case, the entire string is processed by Xen itself, while
in the EFI boot case, only the
portions after a "--" separator are processed by Xen, the stuff before
that is consumed by the EFI boot
code.  Switching between and EFI and Image boot in GRUB would either
mean the user changing
the command line, or GRUB doing some magic/heuristics to fix things up
behind the scenes.  Neither
of these seems like a great solution.  As I understand it, Fu's GRUB
work is only supporting EFI boot
of Xen for arm64, so we only have one case to consider, but if booting
an Image format Xen on arm64
is going to be supported in GRUB in the future then this is something
that we will have to deal with.

Jan - do you know how this command line handling in GRUB for x86 is
handled?  I would think that x86
would have the same issue with the command line.

Roy

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Jan Beulich]

>>> On 25.10.14 at 21:18, <roy.franz@linaro.org> wrote:
> Jan - do you know how this command line handling in GRUB for x86 is
> handled?  I would think that x86
> would have the same issue with the command line.

I guess the question isn't really relevant on x86 since we don't try to
boot the same image two ways.

And anyway, only when booting from UEFI (without GrUB involved)
does anything prior to an eventual -- matter. With GrUB it shouldn't
(and hence the separator is then also pointless and should neither
get inserted nor looked for), as these options only control the EFI
application behavior of the binary. At least that's how it was
intended to be originally.

Jan

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Roy Franz]

On Mon, Oct 27, 2014 at 3:55 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 25.10.14 at 21:18, <roy.franz@linaro.org> wrote:
>> Jan - do you know how this command line handling in GRUB for x86 is
>> handled?  I would think that x86
>> would have the same issue with the command line.
>
> I guess the question isn't really relevant on x86 since we don't try to
> boot the same image two ways.

I think this is related to wanting to be able to boot both and EFI executable
and a Linux Image using GRUB.  Whether these are the same file or not
doesn't affect how the command lines are different.

>
> And anyway, only when booting from UEFI (without GrUB involved)
> does anything prior to an eventual -- matter. With GrUB it shouldn't
> (and hence the separator is then also pointless and should neither
> get inserted nor looked for), as these options only control the EFI
> application behavior of the binary. At least that's how it was
> intended to be originally.
>
> Jan
>
So I guess on x86 GRUB is not used to boot the EFI version of Xen?  My
understanding
is that for arm64 servers all the distros want to use GRUB to boot, as
that is primarily how
things work now, even though UEFI provides boot menu support directly.
So for arm64 we want
to support booting EFI both directly and using GRUB.  While there is
no need now to control the
behavior of the EFI application portion of Xen when booting via GRUB I
think this would be useful to
support.

Roy

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Konrad Rzeszutek Wilk]

On Mon, Oct 27, 2014 at 03:29:18PM -0700, Roy Franz wrote:
> On Mon, Oct 27, 2014 at 3:55 AM, Jan Beulich <JBeulich@suse.com> wrote:
> >>>> On 25.10.14 at 21:18, <roy.franz@linaro.org> wrote:
> >> Jan - do you know how this command line handling in GRUB for x86 is
> >> handled?  I would think that x86
> >> would have the same issue with the command line.
> >
> > I guess the question isn't really relevant on x86 since we don't try to
> > boot the same image two ways.
> 
> I think this is related to wanting to be able to boot both and EFI executable
> and a Linux Image using GRUB.  Whether these are the same file or not
> doesn't affect how the command lines are different.
> 
> >
> > And anyway, only when booting from UEFI (without GrUB involved)
> > does anything prior to an eventual -- matter. With GrUB it shouldn't
> > (and hence the separator is then also pointless and should neither
> > get inserted nor looked for), as these options only control the EFI
> > application behavior of the binary. At least that's how it was
> > intended to be originally.
> >
> > Jan
> >
> So I guess on x86 GRUB is not used to boot the EFI version of Xen?  My
> understanding
> is that for arm64 servers all the distros want to use GRUB to boot, as
> that is primarily how
> things work now, even though UEFI provides boot menu support directly.
> So for arm64 we want
> to support booting EFI both directly and using GRUB.  While there is
> no need now to control the
> behavior of the EFI application portion of Xen when booting via GRUB I
> think this would be useful to
> support.

CC-ing Daniel as that is what he has been working on.
> 
> Roy
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Jan Beulich]

>>> On 27.10.14 at 23:29, <roy.franz@linaro.org> wrote:
> On Mon, Oct 27, 2014 at 3:55 AM, Jan Beulich <JBeulich@suse.com> wrote:
>> And anyway, only when booting from UEFI (without GrUB involved)
>> does anything prior to an eventual -- matter. With GrUB it shouldn't
>> (and hence the separator is then also pointless and should neither
>> get inserted nor looked for), as these options only control the EFI
>> application behavior of the binary. At least that's how it was
>> intended to be originally.
>>
> So I guess on x86 GRUB is not used to boot the EFI version of Xen?

Correct - supporting this is what Daniel Kiper has been working on
for the last several months.

>  My understanding
> is that for arm64 servers all the distros want to use GRUB to boot, as
> that is primarily how
> things work now, even though UEFI provides boot menu support directly.
> So for arm64 we want
> to support booting EFI both directly and using GRUB.  While there is
> no need now to control the
> behavior of the EFI application portion of Xen when booting via GRUB I
> think this would be useful to support.

With the goal/purpose of what? The EFI application portion really is
being replaced by GrUB (or any other more advanced boot loader,
i.e. excluding the EFI boot manager). There are pieces of efi_start()
that serve a purpose in both environments, but that's merely
because that function does more than just the work attributable to
being an EFI application.

That doesn't extend to the command line though if the mechanism
by which GrUB communicates it to its descendants is via the Loaded
Image protocol rather than MB2 (albeit that seems dubious to me).
But that's still only the "core" command line, not the part the EFI
application would see/handle prior to the first --. I.e. that part of
efi_start() really also should become conditional upon whether a
config file is being used (as well as the code leading to the
StdOut->SetMode() call).

Jan

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Roy Franz]

On Tue, Oct 28, 2014 at 12:48 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 27.10.14 at 23:29, <roy.franz@linaro.org> wrote:
>> On Mon, Oct 27, 2014 at 3:55 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>> And anyway, only when booting from UEFI (without GrUB involved)
>>> does anything prior to an eventual -- matter. With GrUB it shouldn't
>>> (and hence the separator is then also pointless and should neither
>>> get inserted nor looked for), as these options only control the EFI
>>> application behavior of the binary. At least that's how it was
>>> intended to be originally.
>>>
>> So I guess on x86 GRUB is not used to boot the EFI version of Xen?
>
> Correct - supporting this is what Daniel Kiper has been working on
> for the last several months.
>
>>  My understanding
>> is that for arm64 servers all the distros want to use GRUB to boot, as
>> that is primarily how
>> things work now, even though UEFI provides boot menu support directly.
>> So for arm64 we want
>> to support booting EFI both directly and using GRUB.  While there is
>> no need now to control the
>> behavior of the EFI application portion of Xen when booting via GRUB I
>> think this would be useful to support.
>
> With the goal/purpose of what?
Are you referring to using GRUB with EFI, or to being able to pass EFI specific
arguments to the EFI code in Xen?

> The EFI application portion really is
> being replaced by GrUB (or any other more advanced boot loader,
> i.e. excluding the EFI boot manager). There are pieces of efi_start()
> that serve a purpose in both environments, but that's merely
> because that function does more than just the work attributable to
> being an EFI application.
Yup.  I think the only reason to pass EFI specific arguments in the GRUB case
is if the Xen EFI boot code is doing more than just handling basic EFI
OS startup.
If Xen doesn't do this in the GRUB case, then I don't see a need for
passing arguements
to the EFI startup code from GRUB.

>
> That doesn't extend to the command line though if the mechanism
> by which GrUB communicates it to its descendants is via the Loaded
> Image protocol rather than MB2 (albeit that seems dubious to me).
I don't really understand what you are saying here.

> But that's still only the "core" command line, not the part the EFI
> application would see/handle prior to the first --. I.e. that part of
> efi_start() really also should become conditional upon whether a
> config file is being used (as well as the code leading to the
> StdOut->SetMode() call).
>
> Jan

If I understand you correctly, you are suggesting something along the lines of:
* If a bootloader (GRUB, etc.) is used to load EFI Xen, the bootloader
is responsible for taking care of all of
 the module loading and other setup such as video/console modes, and
the EFI boot code in Xen
 just does the EFI OS starting bits (memory map, ExitBootServices, etc.)
* This requires some additional code in efi_start() to be
conditionalized based on the need for
  a config file.  (which is really whether a bootloader has loaded XEN
vs. a native EFI load.)

This resolves the whole "--" in the commandline issue - the command
line specified in GRUB is for
Xen itself, no options are processed by the EFI boot code.


The two open issues I see are:
* Should the Xen commandline be put into the MBD2/FDT, or giving in
the EFI command line?  In the arm64 case,
passing it in the EFI commandline is what EFI Linux does and allows
more code sharing in GRUB.
  Jan - it was unclear to me which of these you favor.
* How to determine if Xen is loaded by GRUB?  For arm64, I currently
use the presence of a module in the
FDT provided.  Is it possible/useful to boot Xen without any modules?
If it is then a more robust mechanism
needs to be used.

If possible I'd like to get this settled so that adjustments to the
EFI boot code can be made for the 4.5 release.  It
would be nice to have the EFI boot from GRUB interface be settled and
not change after the release.

Thanks,
Roy

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Jan Beulich]

>>> On 29.10.14 at 04:23, <roy.franz@linaro.org> wrote:
> On Tue, Oct 28, 2014 at 12:48 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 27.10.14 at 23:29, <roy.franz@linaro.org> wrote:
>>>  My understanding
>>> is that for arm64 servers all the distros want to use GRUB to boot, as
>>> that is primarily how
>>> things work now, even though UEFI provides boot menu support directly.
>>> So for arm64 we want
>>> to support booting EFI both directly and using GRUB.  While there is
>>> no need now to control the
>>> behavior of the EFI application portion of Xen when booting via GRUB I
>>> think this would be useful to support.
>>
>> With the goal/purpose of what?
> Are you referring to using GRUB with EFI, or to being able to pass EFI 
> specific
> arguments to the EFI code in Xen?

And I don't see how this counter question relates to the original one.

>> That doesn't extend to the command line though if the mechanism
>> by which GrUB communicates it to its descendants is via the Loaded
>> Image protocol rather than MB2 (albeit that seems dubious to me).
> I don't really understand what you are saying here.

There are two ways to communicate command line arguments under
GrUB: The MB2 way and the EFI way. I don't see why we would want/
need to honor the ones coming the EFI way when (supposedly) the
same ones get passed via MB2.

>> But that's still only the "core" command line, not the part the EFI
>> application would see/handle prior to the first --. I.e. that part of
>> efi_start() really also should become conditional upon whether a
>> config file is being used (as well as the code leading to the
>> StdOut->SetMode() call).
> 
> If I understand you correctly, you are suggesting something along the lines 
> of:
> * If a bootloader (GRUB, etc.) is used to load EFI Xen, the bootloader
> is responsible for taking care of all of
>  the module loading and other setup such as video/console modes, and
> the EFI boot code in Xen
>  just does the EFI OS starting bits (memory map, ExitBootServices, etc.)
> * This requires some additional code in efi_start() to be
> conditionalized based on the need for
>   a config file.  (which is really whether a bootloader has loaded XEN
> vs. a native EFI load.)
> 
> This resolves the whole "--" in the commandline issue - the command
> line specified in GRUB is for
> Xen itself, no options are processed by the EFI boot code.

Yes.

> The two open issues I see are:
> * Should the Xen commandline be put into the MBD2/FDT, or giving in
> the EFI command line?  In the arm64 case,
> passing it in the EFI commandline is what EFI Linux does and allows
> more code sharing in GRUB.
>   Jan - it was unclear to me which of these you favor.

The natural (to GrUB) one - MB2 (or FDT if that's the MB2 equivalent
on ARM). I'm not sure about Linux on ARM, but Linux on x86 gets
special treatment by GrUB anyway, and Xen (being an MB client)
clearly shouldn't be treated like Linux. In fact I've always been
viewing it as at least conceptually bogus for Linux to be a one-off
in GrUB (as opposed to e.g. lilo, which is specifically a loader for
Linux).

> * How to determine if Xen is loaded by GRUB?  For arm64, I currently
> use the presence of a module in the
> FDT provided.  Is it possible/useful to boot Xen without any modules?
> If it is then a more robust mechanism
> needs to be used.

How would that be useful, as that would mean no Dom0? Brief
inspection of ARM code doesn't show an equivalent of x86's

    /* Check that we have at least one Multiboot module. */
    if ( !(mbi->flags & MBI_MODULES) || (mbi->mods_count == 0) )
        panic("dom0 kernel not specified. Check bootloader configuration.");

but I nevertheless can't see how Xen without Dom0 would be
useful on ARM either.

> If possible I'd like to get this settled so that adjustments to the
> EFI boot code can be made for the 4.5 release.  It
> would be nice to have the EFI boot from GRUB interface be settled and
> not change after the release.

I fully agree - interfaces should be stable once we release 4.5.

Jan

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Daniel Kiper]

On Tue, Oct 28, 2014 at 08:23:48PM -0700, Roy Franz wrote:
> On Tue, Oct 28, 2014 at 12:48 AM, Jan Beulich <JBeulich@suse.com> wrote:
> >>>> On 27.10.14 at 23:29, <roy.franz@linaro.org> wrote:
> >> On Mon, Oct 27, 2014 at 3:55 AM, Jan Beulich <JBeulich@suse.com> wrote:
> >>> And anyway, only when booting from UEFI (without GrUB involved)
> >>> does anything prior to an eventual -- matter. With GrUB it shouldn't
> >>> (and hence the separator is then also pointless and should neither
> >>> get inserted nor looked for), as these options only control the EFI
> >>> application behavior of the binary. At least that's how it was
> >>> intended to be originally.
> >>>
> >> So I guess on x86 GRUB is not used to boot the EFI version of Xen?
> >
> > Correct - supporting this is what Daniel Kiper has been working on
> > for the last several months.

It looks that we we will be able to load PE image using EFI loader,
multiboot(1) and multiboot2 protocol. I did some initial tests
and everything looks promising. However, there is an open question
how to build final PE image with required properties. I have at
least two ideas how to do that I am going to present them shortly
on Xen-devel.

[...]

> > That doesn't extend to the command line though if the mechanism
> > by which GrUB communicates it to its descendants is via the Loaded
> > Image protocol rather than MB2 (albeit that seems dubious to me).
> I don't really understand what you are saying here.
>
> > But that's still only the "core" command line, not the part the EFI
> > application would see/handle prior to the first --. I.e. that part of
> > efi_start() really also should become conditional upon whether a
> > config file is being used (as well as the code leading to the
> > StdOut->SetMode() call).
> >
> > Jan
>
> If I understand you correctly, you are suggesting something along the lines of:
> * If a bootloader (GRUB, etc.) is used to load EFI Xen, the bootloader
> is responsible for taking care of all of
>  the module loading and other setup such as video/console modes, and
> the EFI boot code in Xen
>  just does the EFI OS starting bits (memory map, ExitBootServices, etc.)

More or less (I am inclining to less and IIRC it was agreed during last
Xen Hackathon). I am working on this issue right now.

> * This requires some additional code in efi_start() to be
> conditionalized based on the need for
>   a config file.  (which is really whether a bootloader has loaded XEN
> vs. a native EFI load.)
>
> This resolves the whole "--" in the commandline issue - the command
> line specified in GRUB is for
> Xen itself, no options are processed by the EFI boot code.
>
>
> The two open issues I see are:
> * Should the Xen commandline be put into the MBD2/FDT, or giving in
> the EFI command line?  In the arm64 case,
> passing it in the EFI commandline is what EFI Linux does and allows
> more code sharing in GRUB.

I think that in case of Xen loaded by GRUB we should not mangle in any
way command line passed to Xen and modules. Even GRUB should not mangle
command lines. All of them should be passed and parsed as is. This way we
will be in line with current versions of multiboot protocols and GRUB.

>   Jan - it was unclear to me which of these you favor.
> * How to determine if Xen is loaded by GRUB?  For arm64, I currently
> use the presence of a module in the
> FDT provided.  Is it possible/useful to boot Xen without any modules?
> If it is then a more robust mechanism
> needs to be used.

efi_start() should be called only by EFI loader. There should be
separate entry point in EFI init code for multiboot protocol. It
should take into account that we have some knowledge from GRUB
(or any mutliboot protocol aware loader) and act as required.
However, I suppose that somewhere, probably quite early, both
code paths (let's call it EFI and GRUB) will merge. Currently
I am working on this and I hope that I will be able to post
something in November.

> If possible I'd like to get this settled so that adjustments to the
> EFI boot code can be made for the 4.5 release. It would be nice to
> have the EFI boot from GRUB interface be settled and not change
> after the release.

Hmmm... Do you mean 4.6? 4.5 is in freezing state so only bugfixes are allowed.

Daniel

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Ian Campbell]

On Wed, 2014-10-29 at 13:44 +0100, Daniel Kiper wrote:
> On Tue, Oct 28, 2014 at 08:23:48PM -0700, Roy Franz wrote:
> > If possible I'd like to get this settled so that adjustments to the
> > EFI boot code can be made for the 4.5 release. It would be nice to
> > have the EFI boot from GRUB interface be settled and not change
> > after the release.
> 
> Hmmm... Do you mean 4.6? 4.5 is in freezing state so only bugfixes are allowed.
> 

It would be a bug to release Xen on ARM 4.5 with broken handling of the
provision of the command line when booted via EFI.

Ian.

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Daniel Kiper]

On Wed, Oct 29, 2014 at 03:26:40PM +0000, Ian Campbell wrote:
> On Wed, 2014-10-29 at 13:44 +0100, Daniel Kiper wrote:
> > On Tue, Oct 28, 2014 at 08:23:48PM -0700, Roy Franz wrote:
> > > If possible I'd like to get this settled so that adjustments to the
> > > EFI boot code can be made for the 4.5 release. It would be nice to
> > > have the EFI boot from GRUB interface be settled and not change
> > > after the release.
> >
> > Hmmm... Do you mean 4.6? 4.5 is in freezing state so only bugfixes are allowed.
> >
>
> It would be a bug to release Xen on ARM 4.5 with broken handling of the
> provision of the command line when booted via EFI.

What is wrong with current implementation? I thought that it works
in the same way on x86 and ARM. However, maybe I missing something.

Daniel

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Roy Franz]

On Wed, Oct 29, 2014 at 9:55 AM, Daniel Kiper <daniel.kiper@oracle.com> wrote:
> On Wed, Oct 29, 2014 at 03:26:40PM +0000, Ian Campbell wrote:
>> On Wed, 2014-10-29 at 13:44 +0100, Daniel Kiper wrote:
>> > On Tue, Oct 28, 2014 at 08:23:48PM -0700, Roy Franz wrote:
>> > > If possible I'd like to get this settled so that adjustments to the
>> > > EFI boot code can be made for the 4.5 release. It would be nice to
>> > > have the EFI boot from GRUB interface be settled and not change
>> > > after the release.
>> >
>> > Hmmm... Do you mean 4.6? 4.5 is in freezing state so only bugfixes are allowed.
>> >
>>
>> It would be a bug to release Xen on ARM 4.5 with broken handling of the
>> provision of the command line when booted via EFI.
>
> What is wrong with current implementation? I thought that it works
> in the same way on x86 and ARM. However, maybe I missing something.
>
> Daniel

I'll try to provide a brief summary.

Currently, when booted from GRUB, arm64 EFI Xen:
1) processes the EFI command line for EFI related options (-basevideo,
etc), ie those before the "--"
2) expects the Xen commandline to be in the GRUB provided FDT.
Anything past a "--" on the
    EFI commandline that would be given to Xen when booted natively
from EFI is ignored.


The GRUB work to support EFI Xen booting on arm64 is in progress, and
when Leif was reviewing
the code he noticed that this diverged from how EFI Linux was booted.
EFI Linux takes it's commandline
via the EFI commandline - GRUB does not put it in the FDT.  For the
EFI case, having both Xen and Linux
accept their commandlines the same way allows more shared code in
GRUB.  This is what motivated this
patch to change where the commandline was taken from.

The discussion of this patch brought up the related issue of how EFI
boot code specific options, and the
handling of the "--" separator.  This I think has been resolved -
there is no use for EFI boot specific options
when booting from GRUB, as the EFI boot code should not do anything
that requires configuration.  Therefore
there is no "--" required, and GRUB just deals with the Xen
commandline.  (Note that there are some small changes
required to remove some code from the GRUB boot case.)

So the open question is, when booted from GRUB (or other bootloader),
should Xen get it's commandline via the EFI
commandline, or via the MB2 protocol?  (and for arm64, this means the
FDT based multiboot.)  I felt that the shared code
in GRUB was a reasonable reason to follow what Linux did in the EFI case.

Roy

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Jan Beulich]

>>> On 30.10.14 at 03:21, <roy.franz@linaro.org> wrote:
> So the open question is, when booted from GRUB (or other bootloader),
> should Xen get it's commandline via the EFI
> commandline, or via the MB2 protocol?  (and for arm64, this means the
> FDT based multiboot.)  I felt that the shared code
> in GRUB was a reasonable reason to follow what Linux did in the EFI case.

Consistency would call for MB2/FDT, but in the end this can very well
be arch-specific imo (but in that case the patch also needs to reflect
this).

Jan

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Ian Campbell]

On Thu, 2014-10-30 at 09:49 +0000, Jan Beulich wrote:
> >>> On 30.10.14 at 03:21, <roy.franz@linaro.org> wrote:
> > So the open question is, when booted from GRUB (or other bootloader),
> > should Xen get it's commandline via the EFI
> > commandline, or via the MB2 protocol?  (and for arm64, this means the
> > FDT based multiboot.)  I felt that the shared code
> > in GRUB was a reasonable reason to follow what Linux did in the EFI case.

Isn't the "unshared" code here an if statement (or maybe two)? I don't
see why this one difference would require a troublesome amount of
separation the implementations.

> Consistency would call for MB2/FDT, but in the end this can very well
> be arch-specific imo (but in that case the patch also needs to reflect
> this).

I don't have a strong opinion either way, but if it is to change for
arm64 I'd like to see a proposed update to
http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/Multiboot
as a starting point for deciding.

It needs to be careful not to break the "multiboot as a series of u-boot
fdt commands" case too.

Ian.

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Daniel Kiper]

On Wed, Oct 29, 2014 at 07:21:23PM -0700, Roy Franz wrote:
> On Wed, Oct 29, 2014 at 9:55 AM, Daniel Kiper <daniel.kiper@oracle.com> wrote:
> > On Wed, Oct 29, 2014 at 03:26:40PM +0000, Ian Campbell wrote:
> >> On Wed, 2014-10-29 at 13:44 +0100, Daniel Kiper wrote:
> >> > On Tue, Oct 28, 2014 at 08:23:48PM -0700, Roy Franz wrote:
> >> > > If possible I'd like to get this settled so that adjustments to the
> >> > > EFI boot code can be made for the 4.5 release. It would be nice to
> >> > > have the EFI boot from GRUB interface be settled and not change
> >> > > after the release.
> >> >
> >> > Hmmm... Do you mean 4.6? 4.5 is in freezing state so only bugfixes are allowed.
> >> >
> >>
> >> It would be a bug to release Xen on ARM 4.5 with broken handling of the
> >> provision of the command line when booted via EFI.
> >
> > What is wrong with current implementation? I thought that it works
> > in the same way on x86 and ARM. However, maybe I missing something.
> >
> > Daniel
>
> I'll try to provide a brief summary.

Thanks!

> Currently, when booted from GRUB, arm64 EFI Xen:
> 1) processes the EFI command line for EFI related options (-basevideo,
> etc), ie those before the "--"
> 2) expects the Xen commandline to be in the GRUB provided FDT.
> Anything past a "--" on the
>     EFI commandline that would be given to Xen when booted natively
> from EFI is ignored.
>
>
> The GRUB work to support EFI Xen booting on arm64 is in progress, and
> when Leif was reviewing
> the code he noticed that this diverged from how EFI Linux was booted.
> EFI Linux takes it's commandline
> via the EFI commandline - GRUB does not put it in the FDT.  For the
> EFI case, having both Xen and Linux
> accept their commandlines the same way allows more shared code in
> GRUB.  This is what motivated this
> patch to change where the commandline was taken from.
>
> The discussion of this patch brought up the related issue of how EFI
> boot code specific options, and the
> handling of the "--" separator.  This I think has been resolved -
> there is no use for EFI boot specific options
> when booting from GRUB, as the EFI boot code should not do anything
> that requires configuration.  Therefore
> there is no "--" required, and GRUB just deals with the Xen
> commandline.  (Note that there are some small changes
> required to remove some code from the GRUB boot case.)
>
> So the open question is, when booted from GRUB (or other bootloader),
> should Xen get it's commandline via the EFI
> commandline, or via the MB2 protocol?  (and for arm64, this means the
> FDT based multiboot.)  I felt that the shared code
> in GRUB was a reasonable reason to follow what Linux did in the EFI case.

After reading some GRUB2 code it looks that on ARM64 Linux Kernel PE image
(ARM64 GRUB2 does not accept Linux Kernel in other formats) is executed as
regular EFI application. It means that all arguments for kernel itself are
found in usual place. However, arguments for initrd and ramdisk are passed
via FDT. After reading your email I realized that you are going in the same
direction. Am I right? If yes then I can see three solutions for this issue:
  - GRUB2 should pass e.g. --multiboot as first argument to Xen EFI
    application and after -- all arguments for Xen itself; Xen EFI app
    should not parse xen.cfg files in this case,
  - GRUB2 should pass e.g. --multiboot as first argument to Xen EFI
    application and all arguments for Xen itself (and relevant modules)
    in FDT; Xen EFI app should not parse xen.cfg files in this case too,
  - GRUB2 should pass all arguments to EFI application as is via standard
    way (as GRUB2 linux loader does on ARM64); arguments for modules should
    be passed via FDT; Xen EFI application should parse all arguments as
    regular xen.gz; -cfg option should be changed to cfg, -basevideo option
    should be changed to basevideo, etc. At first sight this looks as most
    natural thing from new multiboot protocol for ARM64 definition standpoint.

Anyway, I think that this thing should be be described in multiboot for ARM64
spec as IanC earlier told.

I think that this should be ARM specific solution because we will be using
multiboot2 protocol on x86 which works on completely different way. Hmmm...
Maybe third solution (excluding FDT of course), if we choose that one,
should be common thing and work on any platform including x86.

By the way, as I saw GRUB2 for ARM is completely different thing. Are
you going to support loading Xen using GRUB2 on ARM EFI platforms?

I do not know ARM stuff well so please correct me if I am wrong.

Daniel

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Roy Franz]

On Thu, Oct 30, 2014 at 10:57 AM, Daniel Kiper <daniel.kiper@oracle.com> wrote:
> On Wed, Oct 29, 2014 at 07:21:23PM -0700, Roy Franz wrote:
>> On Wed, Oct 29, 2014 at 9:55 AM, Daniel Kiper <daniel.kiper@oracle.com> wrote:
>> > On Wed, Oct 29, 2014 at 03:26:40PM +0000, Ian Campbell wrote:
>> >> On Wed, 2014-10-29 at 13:44 +0100, Daniel Kiper wrote:
>> >> > On Tue, Oct 28, 2014 at 08:23:48PM -0700, Roy Franz wrote:
>> >> > > If possible I'd like to get this settled so that adjustments to the
>> >> > > EFI boot code can be made for the 4.5 release. It would be nice to
>> >> > > have the EFI boot from GRUB interface be settled and not change
>> >> > > after the release.
>> >> >
>> >> > Hmmm... Do you mean 4.6? 4.5 is in freezing state so only bugfixes are allowed.
>> >> >
>> >>
>> >> It would be a bug to release Xen on ARM 4.5 with broken handling of the
>> >> provision of the command line when booted via EFI.
>> >
>> > What is wrong with current implementation? I thought that it works
>> > in the same way on x86 and ARM. However, maybe I missing something.
>> >
>> > Daniel
>>
>> I'll try to provide a brief summary.
>
> Thanks!
>
>> Currently, when booted from GRUB, arm64 EFI Xen:
>> 1) processes the EFI command line for EFI related options (-basevideo,
>> etc), ie those before the "--"
>> 2) expects the Xen commandline to be in the GRUB provided FDT.
>> Anything past a "--" on the
>>     EFI commandline that would be given to Xen when booted natively
>> from EFI is ignored.
>>
>>
>> The GRUB work to support EFI Xen booting on arm64 is in progress, and
>> when Leif was reviewing
>> the code he noticed that this diverged from how EFI Linux was booted.
>> EFI Linux takes it's commandline
>> via the EFI commandline - GRUB does not put it in the FDT.  For the
>> EFI case, having both Xen and Linux
>> accept their commandlines the same way allows more shared code in
>> GRUB.  This is what motivated this
>> patch to change where the commandline was taken from.
>>
>> The discussion of this patch brought up the related issue of how EFI
>> boot code specific options, and the
>> handling of the "--" separator.  This I think has been resolved -
>> there is no use for EFI boot specific options
>> when booting from GRUB, as the EFI boot code should not do anything
>> that requires configuration.  Therefore
>> there is no "--" required, and GRUB just deals with the Xen
>> commandline.  (Note that there are some small changes
>> required to remove some code from the GRUB boot case.)
>>
>> So the open question is, when booted from GRUB (or other bootloader),
>> should Xen get it's commandline via the EFI
>> commandline, or via the MB2 protocol?  (and for arm64, this means the
>> FDT based multiboot.)  I felt that the shared code
>> in GRUB was a reasonable reason to follow what Linux did in the EFI case.
>
> After reading some GRUB2 code it looks that on ARM64 Linux Kernel PE image
> (ARM64 GRUB2 does not accept Linux Kernel in other formats) is executed as
> regular EFI application. It means that all arguments for kernel itself are
> found in usual place.
Yes, and here for a PE application the usual place is the EFI command line.


> However, arguments for initrd and ramdisk are passed
> via FDT. After reading your email I realized that you are going in the same
> direction. Am I right?
Yes

> If yes then I can see three solutions for this issue:
>   - GRUB2 should pass e.g. --multiboot as first argument to Xen EFI
>     application and after -- all arguments for Xen itself; Xen EFI app
>     should not parse xen.cfg files in this case,
>   - GRUB2 should pass e.g. --multiboot as first argument to Xen EFI
>     application and all arguments for Xen itself (and relevant modules)
>     in FDT; Xen EFI app should not parse xen.cfg files in this case too,
>   - GRUB2 should pass all arguments to EFI application as is via standard
>     way (as GRUB2 linux loader does on ARM64); arguments for modules should
>     be passed via FDT; Xen EFI application should parse all arguments as
>     regular xen.gz; -cfg option should be changed to cfg, -basevideo option
>     should be changed to basevideo, etc. At first sight this looks as most
>     natural thing from new multiboot protocol for ARM64 definition standpoint.
>
> Anyway, I think that this thing should be be described in multiboot for ARM64
> spec as IanC earlier told.
>
> I think that this should be ARM specific solution because we will be using
> multiboot2 protocol on x86 which works on completely different way. Hmmm...
> Maybe third solution (excluding FDT of course), if we choose that one,
> should be common thing and work on any platform including x86.

The multiboot2 on x86 and the FDT on arm64 are very different, which I think
confuses/complicates these discussions.  The open issue of "is the EFI command
line used to pass the Xen command line when GRUB boots EFI Xen" applies equally
to x86 and arm64.  I think that consistency between x86/arm64 Xen in this regard
is more important than following what Linux does, so I'm dropping that
suggestion.

So I'm going to propose the following which I think is in line with
your 2nd option
(I'm not familiar with the GRUB multiboot syntax, so it might not be),
and I think
is along the lines of what Jan has also suggested:

When booting EFI Xen via GRUB (for both x86 and arm64):
1) the EFI command line is not used by Xen, and is ignored.
2) the Xen commandline is provided to Xen in the multiboot2/FDT data
3) The Xen EFI configuration file is not used
4) The EFI Xen boot code does not do any console/video or other initialization.
    There is no way to provide EFI boot code specific options, as it
doesn't and shouldn't
    need any.

I think aligning on the above issues for both architectures in the EFI
would be a good
thing.
The existing arm64 implementation already does 2 and 3 from above. 1 and 4 would
require some minor changes, and I think the EFI booting documentation
should be update
to note that it does not describe EFI booting via GRUB.

>
> By the way, as I saw GRUB2 for ARM is completely different thing. Are
> you going to support loading Xen using GRUB2 on ARM EFI platforms?
Yes, this is currently being worked on by Leif and Fu, and this is expected
to be the typical boot method on those platforms.

>
> I do not know ARM stuff well so please correct me if I am wrong.
>
> Daniel

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Daniel Kiper]

On Thu, Oct 30, 2014 at 12:29:05PM -0700, Roy Franz wrote:
> On Thu, Oct 30, 2014 at 10:57 AM, Daniel Kiper <daniel.kiper@oracle.com> wrote:

[...]

> > If yes then I can see three solutions for this issue:
> >   - GRUB2 should pass e.g. --multiboot as first argument to Xen EFI
> >     application and after -- all arguments for Xen itself; Xen EFI app
> >     should not parse xen.cfg files in this case,
> >   - GRUB2 should pass e.g. --multiboot as first argument to Xen EFI
> >     application and all arguments for Xen itself (and relevant modules)
> >     in FDT; Xen EFI app should not parse xen.cfg files in this case too,
> >   - GRUB2 should pass all arguments to EFI application as is via standard
> >     way (as GRUB2 linux loader does on ARM64); arguments for modules should
> >     be passed via FDT; Xen EFI application should parse all arguments as
> >     regular xen.gz; -cfg option should be changed to cfg, -basevideo option
> >     should be changed to basevideo, etc. At first sight this looks as most
> >     natural thing from new multiboot protocol for ARM64 definition standpoint.
> >
> > Anyway, I think that this thing should be be described in multiboot for ARM64
> > spec as IanC earlier told.
> >
> > I think that this should be ARM specific solution because we will be using
> > multiboot2 protocol on x86 which works on completely different way. Hmmm...
> > Maybe third solution (excluding FDT of course), if we choose that one,
> > should be common thing and work on any platform including x86.
>
> The multiboot2 on x86 and the FDT on arm64 are very different, which I think

I know.

> confuses/complicates these discussions.  The open issue of "is the EFI command

Yep.

> line used to pass the Xen command line when GRUB boots EFI Xen" applies equally
> to x86 and arm64.  I think that consistency between x86/arm64 Xen in this regard

No, PE image will not be executed in the same way (simple EFI application load)
by GRUB2 on x86 like it happens in case of GRUB2 on ARM64. It means that I will
not be entering directly via efi_start() to efi boot code when Xen will be started
by GRUB2 on x86_64 EFI platform. I am going to use separate entry point. However, I am
going to use most of currently existing EFI boot code too. So, as I wrote in earlier
email, I suppose that both paths (GRUB2 and native EFI application) will merge somewhere
quite quickly. However, I do not have the details right now. I am working on it.

> is more important than following what Linux does, so I'm dropping that
> suggestion.

OK.

> So I'm going to propose the following which I think is in line with
> your 2nd option
> (I'm not familiar with the GRUB multiboot syntax, so it might not be),
> and I think
> is along the lines of what Jan has also suggested:
>
> When booting EFI Xen via GRUB (for both x86 and arm64):
> 1) the EFI command line is not used by Xen, and is ignored.

As I said above, multiboot2 protocol on x86 does not execute PE
application as EFI application. It means that we do not have
EFI command line per se.

> 2) the Xen commandline is provided to Xen in the multiboot2/FDT data

OK.

> 3) The Xen EFI configuration file is not used

OK.

> 4) The EFI Xen boot code does not do any console/video or other initialization.
>     There is no way to provide EFI boot code specific options, as it
> doesn't and shouldn't
>     need any.

Please see comment for 1. It means that I will skip code which parse EFI
application command line. Hmmm... How are you going to detect that Xen
was started by GRUB2 if you in both cases entering via efi_start()?
Maybe you should look for arguments in FDT first? Or check FDT for
special flag which means that EFI app was executed by GRUB? Later
looks better because if Xen command line will be empty then EFI
application will try read one of xen.cfg files.

> I think aligning on the above issues for both architectures in the EFI
> would be a good thing.

Yep.

> The existing arm64 implementation already does 2 and 3 from above. 1 and 4 would
> require some minor changes, and I think the EFI booting documentation
> should be update to note that it does not describe EFI booting via GRUB.

Great. By the way, when I was reading http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/Multiboot
it was not obvious for me that multiboot on ARM64 will require PE image
and it will be started as generic EFI application but via GRUB2 (and
all arguments will be in FDT). Could you add this thing there and
in other related docs?

> > By the way, as I saw GRUB2 for ARM is completely different thing. Are
> > you going to support loading Xen using GRUB2 on ARM EFI platforms?
>
> Yes, this is currently being worked on by Leif and Fu, and this is expected
> to be the typical boot method on those platforms.

Doest it mean that multiboot on ARM and ARM64 will work in the same way?
Will it just execute PE image as EFI application and it will look for
arguments in FDT?

Daniel

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Roy Franz]

On Thu, Oct 30, 2014 at 3:18 PM, Daniel Kiper <daniel.kiper@oracle.com> wrote:
> On Thu, Oct 30, 2014 at 12:29:05PM -0700, Roy Franz wrote:
>> On Thu, Oct 30, 2014 at 10:57 AM, Daniel Kiper <daniel.kiper@oracle.com> wrote:
>
> [...]
>
>> > If yes then I can see three solutions for this issue:
>> >   - GRUB2 should pass e.g. --multiboot as first argument to Xen EFI
>> >     application and after -- all arguments for Xen itself; Xen EFI app
>> >     should not parse xen.cfg files in this case,
>> >   - GRUB2 should pass e.g. --multiboot as first argument to Xen EFI
>> >     application and all arguments for Xen itself (and relevant modules)
>> >     in FDT; Xen EFI app should not parse xen.cfg files in this case too,
>> >   - GRUB2 should pass all arguments to EFI application as is via standard
>> >     way (as GRUB2 linux loader does on ARM64); arguments for modules should
>> >     be passed via FDT; Xen EFI application should parse all arguments as
>> >     regular xen.gz; -cfg option should be changed to cfg, -basevideo option
>> >     should be changed to basevideo, etc. At first sight this looks as most
>> >     natural thing from new multiboot protocol for ARM64 definition standpoint.
>> >
>> > Anyway, I think that this thing should be be described in multiboot for ARM64
>> > spec as IanC earlier told.
>> >
>> > I think that this should be ARM specific solution because we will be using
>> > multiboot2 protocol on x86 which works on completely different way. Hmmm...
>> > Maybe third solution (excluding FDT of course), if we choose that one,
>> > should be common thing and work on any platform including x86.
>>
>> The multiboot2 on x86 and the FDT on arm64 are very different, which I think
>
> I know.
>
>> confuses/complicates these discussions.  The open issue of "is the EFI command
>
> Yep.
>
>> line used to pass the Xen command line when GRUB boots EFI Xen" applies equally
>> to x86 and arm64.  I think that consistency between x86/arm64 Xen in this regard
>
> No, PE image will not be executed in the same way (simple EFI application load)
> by GRUB2 on x86 like it happens in case of GRUB2 on ARM64. It means that I will
> not be entering directly via efi_start() to efi boot code when Xen will be started
> by GRUB2 on x86_64 EFI platform. I am going to use separate entry point. However, I am
> going to use most of currently existing EFI boot code too. So, as I wrote in earlier
> email, I suppose that both paths (GRUB2 and native EFI application) will merge somewhere
> quite quickly. However, I do not have the details right now. I am working on it.

OK, this is the part I was missing.

>
>> is more important than following what Linux does, so I'm dropping that
>> suggestion.
>
> OK.
>
>> So I'm going to propose the following which I think is in line with
>> your 2nd option
>> (I'm not familiar with the GRUB multiboot syntax, so it might not be),
>> and I think
>> is along the lines of what Jan has also suggested:
>>
>> When booting EFI Xen via GRUB (for both x86 and arm64):
>> 1) the EFI command line is not used by Xen, and is ignored.
>
> As I said above, multiboot2 protocol on x86 does not execute PE
> application as EFI application. It means that we do not have
> EFI command line per se.

OK, I understand now it makes things clearer.  My assumptions where
wrong about how GRUB would start the PE Xen on x86, so I was asking questions
that didn't really make sense.

>
>> 2) the Xen commandline is provided to Xen in the multiboot2/FDT data
>
> OK.
>
>> 3) The Xen EFI configuration file is not used
>
> OK.
>
>> 4) The EFI Xen boot code does not do any console/video or other initialization.
>>     There is no way to provide EFI boot code specific options, as it
>> doesn't and shouldn't
>>     need any.
>
> Please see comment for 1. It means that I will skip code which parse EFI
> application command line. Hmmm... How are you going to detect that Xen
> was started by GRUB2 if you in both cases entering via efi_start()?
> Maybe you should look for arguments in FDT first? Or check FDT for
> special flag which means that EFI app was executed by GRUB? Later
> looks better because if Xen command line will be empty then EFI
> application will try read one of xen.cfg files.

The current implementation is to examine the FDT passed to Xen to see
if it contains any modules.  If it does, then this indicates to the
EFI boot code
that it is being run by a bootloader, and not directly from the EFI bootmanager
or shell.

>
>> I think aligning on the above issues for both architectures in the EFI
>> would be a good thing.
>
> Yep.
>
>> The existing arm64 implementation already does 2 and 3 from above. 1 and 4 would
>> require some minor changes, and I think the EFI booting documentation
>> should be update to note that it does not describe EFI booting via GRUB.
>
> Great. By the way, when I was reading http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/Multiboot
> it was not obvious for me that multiboot on ARM64 will require PE image
> and it will be started as generic EFI application but via GRUB2 (and
> all arguments will be in FDT). Could you add this thing there and
> in other related docs?

In the ARM case, the multiboot FDT definitions are bootloader agnostic
- all it requires is
an FDT. U-boot can load modules, update the FDT, and start Xen using
this without any
PE/EFI support.  For the EFI case, we use an EFI configuration table
to pass the FDT
to Xen, as opposed to the FDT address in x0 for the "Image" boot method.

What I do want to add is an explicit mention that in the EFI case the
EFI commandline is
not used.

>
>> > By the way, as I saw GRUB2 for ARM is completely different thing. Are
>> > you going to support loading Xen using GRUB2 on ARM EFI platforms?
>>
>> Yes, this is currently being worked on by Leif and Fu, and this is expected
>> to be the typical boot method on those platforms.
>
> Doest it mean that multiboot on ARM and ARM64 will work in the same way?
> Will it just execute PE image as EFI application and it will look for
> arguments in FDT?

Sorry, I responded to the above for arm64, which is getting PE support
in GRUB and Xen.
I'm not aware are any plans to add EFI support for arm32.

>
> Daniel

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Daniel Kiper]

On Thu, Oct 30, 2014 at 04:54:17PM -0700, Roy Franz wrote:
> On Thu, Oct 30, 2014 at 3:18 PM, Daniel Kiper <daniel.kiper@oracle.com> wrote:
> > On Thu, Oct 30, 2014 at 12:29:05PM -0700, Roy Franz wrote:
> >> On Thu, Oct 30, 2014 at 10:57 AM, Daniel Kiper <daniel.kiper@oracle.com> wrote:
> >
> > [...]
> >
> >> > If yes then I can see three solutions for this issue:
> >> >   - GRUB2 should pass e.g. --multiboot as first argument to Xen EFI
> >> >     application and after -- all arguments for Xen itself; Xen EFI app
> >> >     should not parse xen.cfg files in this case,
> >> >   - GRUB2 should pass e.g. --multiboot as first argument to Xen EFI
> >> >     application and all arguments for Xen itself (and relevant modules)
> >> >     in FDT; Xen EFI app should not parse xen.cfg files in this case too,
> >> >   - GRUB2 should pass all arguments to EFI application as is via standard
> >> >     way (as GRUB2 linux loader does on ARM64); arguments for modules should
> >> >     be passed via FDT; Xen EFI application should parse all arguments as
> >> >     regular xen.gz; -cfg option should be changed to cfg, -basevideo option
> >> >     should be changed to basevideo, etc. At first sight this looks as most
> >> >     natural thing from new multiboot protocol for ARM64 definition standpoint.
> >> >
> >> > Anyway, I think that this thing should be be described in multiboot for ARM64
> >> > spec as IanC earlier told.
> >> >
> >> > I think that this should be ARM specific solution because we will be using
> >> > multiboot2 protocol on x86 which works on completely different way. Hmmm...
> >> > Maybe third solution (excluding FDT of course), if we choose that one,
> >> > should be common thing and work on any platform including x86.
> >>
> >> The multiboot2 on x86 and the FDT on arm64 are very different, which I think
> >
> > I know.
> >
> >> confuses/complicates these discussions.  The open issue of "is the EFI command
> >
> > Yep.
> >
> >> line used to pass the Xen command line when GRUB boots EFI Xen" applies equally
> >> to x86 and arm64.  I think that consistency between x86/arm64 Xen in this regard
> >
> > No, PE image will not be executed in the same way (simple EFI application load)
> > by GRUB2 on x86 like it happens in case of GRUB2 on ARM64. It means that I will
> > not be entering directly via efi_start() to efi boot code when Xen will be started
> > by GRUB2 on x86_64 EFI platform. I am going to use separate entry point. However, I am
> > going to use most of currently existing EFI boot code too. So, as I wrote in earlier
> > email, I suppose that both paths (GRUB2 and native EFI application) will merge somewhere
> > quite quickly. However, I do not have the details right now. I am working on it.
>
> OK, this is the part I was missing.
>
> >
> >> is more important than following what Linux does, so I'm dropping that
> >> suggestion.
> >
> > OK.
> >
> >> So I'm going to propose the following which I think is in line with
> >> your 2nd option
> >> (I'm not familiar with the GRUB multiboot syntax, so it might not be),
> >> and I think
> >> is along the lines of what Jan has also suggested:
> >>
> >> When booting EFI Xen via GRUB (for both x86 and arm64):
> >> 1) the EFI command line is not used by Xen, and is ignored.
> >
> > As I said above, multiboot2 protocol on x86 does not execute PE
> > application as EFI application. It means that we do not have
> > EFI command line per se.
>
> OK, I understand now it makes things clearer.  My assumptions where
> wrong about how GRUB would start the PE Xen on x86, so I was asking questions
> that didn't really make sense.
>
> >
> >> 2) the Xen commandline is provided to Xen in the multiboot2/FDT data
> >
> > OK.
> >
> >> 3) The Xen EFI configuration file is not used
> >
> > OK.
> >
> >> 4) The EFI Xen boot code does not do any console/video or other initialization.
> >>     There is no way to provide EFI boot code specific options, as it
> >> doesn't and shouldn't
> >>     need any.
> >
> > Please see comment for 1. It means that I will skip code which parse EFI
> > application command line. Hmmm... How are you going to detect that Xen
> > was started by GRUB2 if you in both cases entering via efi_start()?
> > Maybe you should look for arguments in FDT first? Or check FDT for
> > special flag which means that EFI app was executed by GRUB? Later
> > looks better because if Xen command line will be empty then EFI
> > application will try read one of xen.cfg files.
>
> The current implementation is to examine the FDT passed to Xen to see
> if it contains any modules.  If it does, then this indicates to the
> EFI boot code
> that it is being run by a bootloader, and not directly from the EFI bootmanager
> or shell.

I think that it is wrong. I am able to imagine such situation in which
only small binary is loaded to do something specific and this binary
does not require any modules. If we do what you suggest then we must
load an additional module which will not be used by binary itself but
it just signals that binary was loaded by multiboot protocol. Not nice.
That is why I am still thinking that multiboot protocol aware loader
should put a flag in FDT telling that binary was loaded that way.

I am aware that Xen have to have at least one module (kernel image for dom0)
but I think that new protocol should be generic as much as possible and
Xen should not be its only one user.

Daniel

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Roy Franz]

On Tue, Nov 4, 2014 at 1:49 PM, Daniel Kiper <daniel.kiper@oracle.com> wrote:
> On Thu, Oct 30, 2014 at 04:54:17PM -0700, Roy Franz wrote:
>> On Thu, Oct 30, 2014 at 3:18 PM, Daniel Kiper <daniel.kiper@oracle.com> wrote:
>> > On Thu, Oct 30, 2014 at 12:29:05PM -0700, Roy Franz wrote:
>> >> On Thu, Oct 30, 2014 at 10:57 AM, Daniel Kiper <daniel.kiper@oracle.com> wrote:
>> >
>> > [...]
>> >
>> >> > If yes then I can see three solutions for this issue:
>> >> >   - GRUB2 should pass e.g. --multiboot as first argument to Xen EFI
>> >> >     application and after -- all arguments for Xen itself; Xen EFI app
>> >> >     should not parse xen.cfg files in this case,
>> >> >   - GRUB2 should pass e.g. --multiboot as first argument to Xen EFI
>> >> >     application and all arguments for Xen itself (and relevant modules)
>> >> >     in FDT; Xen EFI app should not parse xen.cfg files in this case too,
>> >> >   - GRUB2 should pass all arguments to EFI application as is via standard
>> >> >     way (as GRUB2 linux loader does on ARM64); arguments for modules should
>> >> >     be passed via FDT; Xen EFI application should parse all arguments as
>> >> >     regular xen.gz; -cfg option should be changed to cfg, -basevideo option
>> >> >     should be changed to basevideo, etc. At first sight this looks as most
>> >> >     natural thing from new multiboot protocol for ARM64 definition standpoint.
>> >> >
>> >> > Anyway, I think that this thing should be be described in multiboot for ARM64
>> >> > spec as IanC earlier told.
>> >> >
>> >> > I think that this should be ARM specific solution because we will be using
>> >> > multiboot2 protocol on x86 which works on completely different way. Hmmm...
>> >> > Maybe third solution (excluding FDT of course), if we choose that one,
>> >> > should be common thing and work on any platform including x86.
>> >>
>> >> The multiboot2 on x86 and the FDT on arm64 are very different, which I think
>> >
>> > I know.
>> >
>> >> confuses/complicates these discussions.  The open issue of "is the EFI command
>> >
>> > Yep.
>> >
>> >> line used to pass the Xen command line when GRUB boots EFI Xen" applies equally
>> >> to x86 and arm64.  I think that consistency between x86/arm64 Xen in this regard
>> >
>> > No, PE image will not be executed in the same way (simple EFI application load)
>> > by GRUB2 on x86 like it happens in case of GRUB2 on ARM64. It means that I will
>> > not be entering directly via efi_start() to efi boot code when Xen will be started
>> > by GRUB2 on x86_64 EFI platform. I am going to use separate entry point. However, I am
>> > going to use most of currently existing EFI boot code too. So, as I wrote in earlier
>> > email, I suppose that both paths (GRUB2 and native EFI application) will merge somewhere
>> > quite quickly. However, I do not have the details right now. I am working on it.
>>
>> OK, this is the part I was missing.
>>
>> >
>> >> is more important than following what Linux does, so I'm dropping that
>> >> suggestion.
>> >
>> > OK.
>> >
>> >> So I'm going to propose the following which I think is in line with
>> >> your 2nd option
>> >> (I'm not familiar with the GRUB multiboot syntax, so it might not be),
>> >> and I think
>> >> is along the lines of what Jan has also suggested:
>> >>
>> >> When booting EFI Xen via GRUB (for both x86 and arm64):
>> >> 1) the EFI command line is not used by Xen, and is ignored.
>> >
>> > As I said above, multiboot2 protocol on x86 does not execute PE
>> > application as EFI application. It means that we do not have
>> > EFI command line per se.
>>
>> OK, I understand now it makes things clearer.  My assumptions where
>> wrong about how GRUB would start the PE Xen on x86, so I was asking questions
>> that didn't really make sense.
>>
>> >
>> >> 2) the Xen commandline is provided to Xen in the multiboot2/FDT data
>> >
>> > OK.
>> >
>> >> 3) The Xen EFI configuration file is not used
>> >
>> > OK.
>> >
>> >> 4) The EFI Xen boot code does not do any console/video or other initialization.
>> >>     There is no way to provide EFI boot code specific options, as it
>> >> doesn't and shouldn't
>> >>     need any.
>> >
>> > Please see comment for 1. It means that I will skip code which parse EFI
>> > application command line. Hmmm... How are you going to detect that Xen
>> > was started by GRUB2 if you in both cases entering via efi_start()?
>> > Maybe you should look for arguments in FDT first? Or check FDT for
>> > special flag which means that EFI app was executed by GRUB? Later
>> > looks better because if Xen command line will be empty then EFI
>> > application will try read one of xen.cfg files.
>>
>> The current implementation is to examine the FDT passed to Xen to see
>> if it contains any modules.  If it does, then this indicates to the
>> EFI boot code
>> that it is being run by a bootloader, and not directly from the EFI bootmanager
>> or shell.
>
> I think that it is wrong. I am able to imagine such situation in which
> only small binary is loaded to do something specific and this binary
> does not require any modules. If we do what you suggest then we must
> load an additional module which will not be used by binary itself but
> it just signals that binary was loaded by multiboot protocol. Not nice.
> That is why I am still thinking that multiboot protocol aware loader
> should put a flag in FDT telling that binary was loaded that way.
>
> I am aware that Xen have to have at least one module (kernel image for dom0)
> but I think that new protocol should be generic as much as possible and
> Xen should not be its only one user.

I think that what is being done for Xen is robust for Xen, and likely will be
for others as well.  The FDT-multiboot bindings are just a way to pass extra
module information to an application - if there are no
"multiboot,module" compatible nodes,
then the FDT-multiboot bindings are not in use.  Multiboot2 seems to
imply much more
than that - I need to investigate that a bit more so I understand how
this is done on
x86.

>
> Daniel

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Daniel Kiper]

On Tue, Nov 04, 2014 at 02:48:59PM -0800, Roy Franz wrote:
> On Tue, Nov 4, 2014 at 1:49 PM, Daniel Kiper <daniel.kiper@oracle.com> wrote:
> > On Thu, Oct 30, 2014 at 04:54:17PM -0700, Roy Franz wrote:

[...]

> >> The current implementation is to examine the FDT passed to Xen to see
> >> if it contains any modules.  If it does, then this indicates to the
> >> EFI boot code
> >> that it is being run by a bootloader, and not directly from the EFI bootmanager
> >> or shell.
> >
> > I think that it is wrong. I am able to imagine such situation in which
> > only small binary is loaded to do something specific and this binary
> > does not require any modules. If we do what you suggest then we must
> > load an additional module which will not be used by binary itself but
> > it just signals that binary was loaded by multiboot protocol. Not nice.
> > That is why I am still thinking that multiboot protocol aware loader
> > should put a flag in FDT telling that binary was loaded that way.
> >
> > I am aware that Xen have to have at least one module (kernel image for dom0)
> > but I think that new protocol should be generic as much as possible and
> > Xen should not be its only one user.
>
> I think that what is being done for Xen is robust for Xen, and likely will be
> for others as well.  The FDT-multiboot bindings are just a way to pass extra
> module information to an application - if there are no "multiboot,module" compatible nodes,
> then the FDT-multiboot bindings are not in use.  Multiboot2 seems to imply much more

OK, however, lack of "multiboot,module" does not mean that image was not
loaded by multiboot aware loader on ARM64 platform. As I can see there
are also other valid FDT multiboot strings in spec, e.g. "multiboot,kernel".
They could not be used by Xen itself but it does not change anything in
boot protocol. Hence, I think that it is much easier to look for one string
or magic value which has meaning "loaded by multiboot aware loader" than
test all possible values to be sure that image was not loaded by "multiboot
aware loader".

> than that - I need to investigate that a bit more so I understand how
> this is done on x86.

multiboot2 aware loader on x86 puts special magic value in %eax to
mark that image should expect multiboot2 as a boot protocol. Early
x86 code looks for this value and do all things accordingly.

Daniel

Re: [PATCH V2 for-4.5] EFI: Always use EFI command line

[Roy Franz]

On Wed, Nov 5, 2014 at 6:30 AM, Daniel Kiper <daniel.kiper@oracle.com> wrote:
> On Tue, Nov 04, 2014 at 02:48:59PM -0800, Roy Franz wrote:
>> On Tue, Nov 4, 2014 at 1:49 PM, Daniel Kiper <daniel.kiper@oracle.com> wrote:
>> > On Thu, Oct 30, 2014 at 04:54:17PM -0700, Roy Franz wrote:
>
> [...]
>
>> >> The current implementation is to examine the FDT passed to Xen to see
>> >> if it contains any modules.  If it does, then this indicates to the
>> >> EFI boot code
>> >> that it is being run by a bootloader, and not directly from the EFI bootmanager
>> >> or shell.
>> >
>> > I think that it is wrong. I am able to imagine such situation in which
>> > only small binary is loaded to do something specific and this binary
>> > does not require any modules. If we do what you suggest then we must
>> > load an additional module which will not be used by binary itself but
>> > it just signals that binary was loaded by multiboot protocol. Not nice.
>> > That is why I am still thinking that multiboot protocol aware loader
>> > should put a flag in FDT telling that binary was loaded that way.
>> >
>> > I am aware that Xen have to have at least one module (kernel image for dom0)
>> > but I think that new protocol should be generic as much as possible and
>> > Xen should not be its only one user.
>>
>> I think that what is being done for Xen is robust for Xen, and likely will be
>> for others as well.  The FDT-multiboot bindings are just a way to pass extra
>> module information to an application - if there are no "multiboot,module" compatible nodes,
>> then the FDT-multiboot bindings are not in use.  Multiboot2 seems to imply much more
>
> OK, however, lack of "multiboot,module" does not mean that image was not
> loaded by multiboot aware loader on ARM64 platform. As I can see there
> are also other valid FDT multiboot strings in spec, e.g. "multiboot,kernel".
> They could not be used by Xen itself but it does not change anything in
> boot protocol. Hence, I think that it is much easier to look for one string
> or magic value which has meaning "loaded by multiboot aware loader" than
> test all possible values to be sure that image was not loaded by "multiboot
> aware loader".

I think that this is worth further discussion, but I don't think this
additional FDT node
will be agreed upon and implemented in GRUB and Xen in time for any changes
in 4.5. (and I don't think it is urgently needed.)  Also, I know I
used the term
"loaded by multiboot aware loader", but this may not be the precise thing the
arm64 EFI Xen boot code cares about.  I think what it really cares about is
whether the FDT it has been given has modules (ie dom0, initrd)
already in it, or if the boot code needs to loade them based on a config file.

>
>> than that - I need to investigate that a bit more so I understand how
>> this is done on x86.
>
> multiboot2 aware loader on x86 puts special magic value in %eax to
> mark that image should expect multiboot2 as a boot protocol. Early
> x86 code looks for this value and do all things accordingly.
>
> Daniel