Relocatable Xen early boot code

Relocatable Xen early boot code

[Daniel Kiper]

Hey,

During work on multiboot2 protocol support for Xen on EFI platform
I discovered that we need relocatable Xen early boot code (which is
mostly 32-bit code). More you can find here:

http://lists.xen.org/archives/html/xen-devel/2015-02/msg01257.html

I would like to focus on solution #1 described above. I have working PoC
which implements it. We also discussed various solutions for this issue
during Xen Hackhaton in Shanghai. As Andrew and Jan asked I tried to
implement solution based on segment registers. However, after revisiting
this issue and further investigation I still have some doubts. You can
read about my findings below.

Here I do not want to discuss GRUB2 and multiboot2 protocol support details
for relocatable images. It is not needed. It is sufficient to know that it
is able to put loaded image anywhere in available memory below 4 GiB. Loaded
image is informed about its base address according to multiboot2 protocol via
special tag. This is new feature not available in upstream GRUB2. I work on
upstreaming it in parallel. Relevant patches will be posted together with
Xen patches.

1) My PoC uses %ebp (last unused register available globally in Xen early
   boot code; there is assumption that %ds == %es == %ss) as a storage for
   Xen image base address. If boot protocol do not support relocatable images
   it is filled with static value calculated during build. Otherwise it is
   taken from special multiboot2 tag.

   We need Xen image base address to access memory in Xen image area in at
   least three different cases:
     a) direct memory access, e.g.: mov %eax,sym_offset(boot_tsc_stamp)(%ebp)
     b) memory address calculation, e.g.: lea sym_offset(__page_tables_start)(%ebp),%edx,
     c) update static addresses calculated during build, e.g. prebuild page tables.

   If we have Xen image base address in a register then all mentioned operations
   are quite simple.

   This idea is based on x86_64 addressing mode which uses %rip as a reference.

2) Andrew and Jan suggested that we can use segment descriptor to store Xen
   image base address. This way all references to variables in Xen image can
   be easily calculated during build and they will be static. However, relevant
   segment descriptor must be updated during Xen start.

   At first sight, it looks that %cs, %ds, %es, %ss should be initialized as is
   (start: 0, size: 4 GiB, ...). This way we will not break references to trampoline
   and all other variables living outside of Xen image. Additionally, we are sure
   that all references to variables from C code (xen/arch/x86/boot/reloc.c) are
   pointing always to the same place regardless of x86 instruction generated by
   compiler and segment register used by this instruction for addressing. So, it
   looks that potentially we can use %fs or %gs as segment register to access
   variables living in Xen image. Looks good... Case a and b from solution #1 seams
   easy to resolve by prefixing by chosen segment register. Case c requires an extra
   register to store temporarily Xen image base address. Looks quite easy... However,
   looking at xen/arch/x86/boot/cmdline.S (at first sight this is the worst thing)
   I am not so happy with segment register solution. Unfortunately all functions
   get as argument just displacement in segment. This means that we must rework
   all stuff there and pass segment(s) as additional argument(s) because most of
   code checks memory in and out of Xen image in parallel. Of course we can use
   solution similar to case c described above but I think that then whole stuff
   move closer to idea #1. This way we will have something which uses things
   from #1 and #2. I do not think that this is solution which we want.

3) There is a third solution which is a mixture of #1 and #2. We can use e.g.
   %fs:0 (e.g. located somewhere in Xen image) to store Xen image base address.
   If this value is needed then we can access it directly, e.g. add %fs:0,%esi,
   or copy to temporary register and use as required.

So, I am still in favor of #1. It is clean and easy. It does not require a lot of work.
#2 has a potential but requires a lot of changes in fragile cmdline.S (maybe others
difficult places). Is it worth? I think that #3 is a backup solution in case we choose
#1 and later it will appear that we need a globally unused register.

Daniel

Re: Relocatable Xen early boot code

[Jan Beulich]

>>> On 12.06.15 at 13:14, <daniel.kiper@oracle.com> wrote:
> Here I do not want to discuss GRUB2 and multiboot2 protocol support details
> for relocatable images. It is not needed. It is sufficient to know that it
> is able to put loaded image anywhere in available memory below 4 GiB. Loaded
> image is informed about its base address according to multiboot2 protocol 
> via
> special tag. This is new feature not available in upstream GRUB2. I work on
> upstreaming it in parallel. Relevant patches will be posted together with
> Xen patches.

Before going into any detail on what you write later on - if this isn't in
upstream grub2, why can't you do what you want to do first without
another change needed in the boot loader? In which case, if I'm reading
this correctly, you wouldn't need our boot code to become relocatable
either.

Jan

Re: Relocatable Xen early boot code

[Daniel Kiper]

On Fri, Jun 12, 2015 at 01:08:41PM +0100, Jan Beulich wrote:
> >>> On 12.06.15 at 13:14, <daniel.kiper@oracle.com> wrote:
> > Here I do not want to discuss GRUB2 and multiboot2 protocol support details
> > for relocatable images. It is not needed. It is sufficient to know that it
> > is able to put loaded image anywhere in available memory below 4 GiB. Loaded
> > image is informed about its base address according to multiboot2 protocol
> > via
> > special tag. This is new feature not available in upstream GRUB2. I work on
> > upstreaming it in parallel. Relevant patches will be posted together with
> > Xen patches.
>
> Before going into any detail on what you write later on - if this isn't in
> upstream grub2, why can't you do what you want to do first without
> another change needed in the boot loader? In which case, if I'm reading
> this correctly, you wouldn't need our boot code to become relocatable
> either.

multiboot protocol (any version) puts image at address as it requests.
Always, without any exception. So, in our case starting from 1 MiB.
On legacy BIOS platforms it works. However, on EFI platforms it is not
always possible because sometimes some parts of EFI boot/runtime services
live somewhere around 1 MiB. So, first of all we must change multiboot
protocol behavior (in our case version 2 because it is more flexible)
and learn it to relocate images into free memory region. On the other
hand we must learn Xen early boot code to live at different addresses.
We must do all this changes in GRUB2 and in Xen in parallel.

Please check http://lists.xen.org/archives/html/xen-devel/2015-02/msg01257.html
for more details.

Daniel

Re: Relocatable Xen early boot code

[Jan Beulich]

>>> On 12.06.15 at 15:51, <daniel.kiper@oracle.com> wrote:
> On Fri, Jun 12, 2015 at 01:08:41PM +0100, Jan Beulich wrote:
>> >>> On 12.06.15 at 13:14, <daniel.kiper@oracle.com> wrote:
>> > Here I do not want to discuss GRUB2 and multiboot2 protocol support details
>> > for relocatable images. It is not needed. It is sufficient to know that it
>> > is able to put loaded image anywhere in available memory below 4 GiB. 
> Loaded
>> > image is informed about its base address according to multiboot2 protocol
>> > via
>> > special tag. This is new feature not available in upstream GRUB2. I work on
>> > upstreaming it in parallel. Relevant patches will be posted together with
>> > Xen patches.
>>
>> Before going into any detail on what you write later on - if this isn't in
>> upstream grub2, why can't you do what you want to do first without
>> another change needed in the boot loader? In which case, if I'm reading
>> this correctly, you wouldn't need our boot code to become relocatable
>> either.
> 
> multiboot protocol (any version) puts image at address as it requests.
> Always, without any exception. So, in our case starting from 1 MiB.
> On legacy BIOS platforms it works. However, on EFI platforms it is not
> always possible because sometimes some parts of EFI boot/runtime services
> live somewhere around 1 MiB. So, first of all we must change multiboot
> protocol behavior (in our case version 2 because it is more flexible)
> and learn it to relocate images into free memory region. On the other
> hand we must learn Xen early boot code to live at different addresses.
> We must do all this changes in GRUB2 and in Xen in parallel.

Right, I now recall (the huge time difference between individual
steps is of course kind of problematic here).

Jan

Re: Relocatable Xen early boot code

[Daniel Kiper]

On Fri, Jun 12, 2015 at 02:57:29PM +0100, Jan Beulich wrote:
> >>> On 12.06.15 at 15:51, <daniel.kiper@oracle.com> wrote:
> > On Fri, Jun 12, 2015 at 01:08:41PM +0100, Jan Beulich wrote:
> >> >>> On 12.06.15 at 13:14, <daniel.kiper@oracle.com> wrote:
> >> > Here I do not want to discuss GRUB2 and multiboot2 protocol support details
> >> > for relocatable images. It is not needed. It is sufficient to know that it
> >> > is able to put loaded image anywhere in available memory below 4 GiB.
> > Loaded
> >> > image is informed about its base address according to multiboot2 protocol
> >> > via
> >> > special tag. This is new feature not available in upstream GRUB2. I work on
> >> > upstreaming it in parallel. Relevant patches will be posted together with
> >> > Xen patches.
> >>
> >> Before going into any detail on what you write later on - if this isn't in
> >> upstream grub2, why can't you do what you want to do first without
> >> another change needed in the boot loader? In which case, if I'm reading
> >> this correctly, you wouldn't need our boot code to become relocatable
> >> either.
> >
> > multiboot protocol (any version) puts image at address as it requests.
> > Always, without any exception. So, in our case starting from 1 MiB.
> > On legacy BIOS platforms it works. However, on EFI platforms it is not
> > always possible because sometimes some parts of EFI boot/runtime services
> > live somewhere around 1 MiB. So, first of all we must change multiboot
> > protocol behavior (in our case version 2 because it is more flexible)
> > and learn it to relocate images into free memory region. On the other
> > hand we must learn Xen early boot code to live at different addresses.
> > We must do all this changes in GRUB2 and in Xen in parallel.
>
> Right, I now recall (the huge time difference between individual
> steps is of course kind of problematic here).

Great! Sorry for "huge time difference" but I was not always able to
devote all my time to upstream work or things were complicating more
then expected and delaying all stuff. Now it looks that all major issues
are solved and I will be able to work most of the time on upstreaming
GRUB2 and Xen EFI related stuff.

Daniel

Re: Relocatable Xen early boot code

[Andrew Cooper]

On 12/06/15 12:14, Daniel Kiper wrote:
> Hey,
>
> During work on multiboot2 protocol support for Xen on EFI platform
> I discovered that we need relocatable Xen early boot code (which is
> mostly 32-bit code). More you can find here:
>
> http://lists.xen.org/archives/html/xen-devel/2015-02/msg01257.html
>
> I would like to focus on solution #1 described above. I have working PoC
> which implements it. We also discussed various solutions for this issue
> during Xen Hackhaton in Shanghai. As Andrew and Jan asked I tried to
> implement solution based on segment registers. However, after revisiting
> this issue and further investigation I still have some doubts. You can
> read about my findings below.
>
> Here I do not want to discuss GRUB2 and multiboot2 protocol support details
> for relocatable images. It is not needed. It is sufficient to know that it
> is able to put loaded image anywhere in available memory below 4 GiB. Loaded
> image is informed about its base address according to multiboot2 protocol via
> special tag. This is new feature not available in upstream GRUB2. I work on
> upstreaming it in parallel. Relevant patches will be posted together with
> Xen patches.
>
> 1) My PoC uses %ebp (last unused register available globally in Xen early
>    boot code; there is assumption that %ds == %es == %ss) as a storage for
>    Xen image base address. If boot protocol do not support relocatable images
>    it is filled with static value calculated during build. Otherwise it is
>    taken from special multiboot2 tag.
>
>    We need Xen image base address to access memory in Xen image area in at
>    least three different cases:
>      a) direct memory access, e.g.: mov %eax,sym_offset(boot_tsc_stamp)(%ebp)
>      b) memory address calculation, e.g.: lea sym_offset(__page_tables_start)(%ebp),%edx,
>      c) update static addresses calculated during build, e.g. prebuild page tables.
>
>    If we have Xen image base address in a register then all mentioned operations
>    are quite simple.
>
>    This idea is based on x86_64 addressing mode which uses %rip as a reference.
>
> 2) Andrew and Jan suggested that we can use segment descriptor to store Xen
>    image base address. This way all references to variables in Xen image can
>    be easily calculated during build and they will be static. However, relevant
>    segment descriptor must be updated during Xen start.
>
>    At first sight, it looks that %cs, %ds, %es, %ss should be initialized as is
>    (start: 0, size: 4 GiB, ...). This way we will not break references to trampoline
>    and all other variables living outside of Xen image. Additionally, we are sure
>    that all references to variables from C code (xen/arch/x86/boot/reloc.c) are
>    pointing always to the same place regardless of x86 instruction generated by
>    compiler and segment register used by this instruction for addressing. So, it
>    looks that potentially we can use %fs or %gs as segment register to access
>    variables living in Xen image. Looks good... Case a and b from solution #1 seams
>    easy to resolve by prefixing by chosen segment register. Case c requires an extra
>    register to store temporarily Xen image base address. Looks quite easy... However,
>    looking at xen/arch/x86/boot/cmdline.S (at first sight this is the worst thing)
>    I am not so happy with segment register solution. Unfortunately all functions
>    get as argument just displacement in segment. This means that we must rework
>    all stuff there and pass segment(s) as additional argument(s) because most of
>    code checks memory in and out of Xen image in parallel. Of course we can use
>    solution similar to case c described above but I think that then whole stuff
>    move closer to idea #1. This way we will have something which uses things
>    from #1 and #2. I do not think that this is solution which we want.
>
> 3) There is a third solution which is a mixture of #1 and #2. We can use e.g.
>    %fs:0 (e.g. located somewhere in Xen image) to store Xen image base address.
>    If this value is needed then we can access it directly, e.g. add %fs:0,%esi,
>    or copy to temporary register and use as required.
>
> So, I am still in favor of #1. It is clean and easy. It does not require a lot of work.
> #2 has a potential but requires a lot of changes in fragile cmdline.S (maybe others
> difficult places). Is it worth? I think that #3 is a backup solution in case we choose
> #1 and later it will appear that we need a globally unused register.

I really don't understand why we have command line parsing written asm. 
I had half a mind to rewrite it all in C, (similar to the existing reloc
code, but have gcc just create the .s and #include it as before, so
external references work in a rational way).  It would be far clearer
like that.

As stated at the hackathon, the problem with using %ebp is that it turns
all implicit %ds references into implicit %ss references, and tends to
add a SIB+imm32 to each instruction with a memory reference (ebp
relative memory references, and r13 for that matter, have restrictions
in the way in which they can be encoded).

~Andrew

Re: Relocatable Xen early boot code

[Daniel Kiper]

On Fri, Jun 12, 2015 at 02:19:01PM +0100, Andrew Cooper wrote:
> On 12/06/15 12:14, Daniel Kiper wrote:
> > Hey,
> >
> > During work on multiboot2 protocol support for Xen on EFI platform
> > I discovered that we need relocatable Xen early boot code (which is
> > mostly 32-bit code). More you can find here:
> >
> > http://lists.xen.org/archives/html/xen-devel/2015-02/msg01257.html
> >
> > I would like to focus on solution #1 described above. I have working PoC
> > which implements it. We also discussed various solutions for this issue
> > during Xen Hackhaton in Shanghai. As Andrew and Jan asked I tried to
> > implement solution based on segment registers. However, after revisiting
> > this issue and further investigation I still have some doubts. You can
> > read about my findings below.
> >
> > Here I do not want to discuss GRUB2 and multiboot2 protocol support details
> > for relocatable images. It is not needed. It is sufficient to know that it
> > is able to put loaded image anywhere in available memory below 4 GiB. Loaded
> > image is informed about its base address according to multiboot2 protocol via
> > special tag. This is new feature not available in upstream GRUB2. I work on
> > upstreaming it in parallel. Relevant patches will be posted together with
> > Xen patches.
> >
> > 1) My PoC uses %ebp (last unused register available globally in Xen early
> >    boot code; there is assumption that %ds == %es == %ss) as a storage for
> >    Xen image base address. If boot protocol do not support relocatable images
> >    it is filled with static value calculated during build. Otherwise it is
> >    taken from special multiboot2 tag.
> >
> >    We need Xen image base address to access memory in Xen image area in at
> >    least three different cases:
> >      a) direct memory access, e.g.: mov %eax,sym_offset(boot_tsc_stamp)(%ebp)
> >      b) memory address calculation, e.g.: lea sym_offset(__page_tables_start)(%ebp),%edx,
> >      c) update static addresses calculated during build, e.g. prebuild page tables.
> >
> >    If we have Xen image base address in a register then all mentioned operations
> >    are quite simple.
> >
> >    This idea is based on x86_64 addressing mode which uses %rip as a reference.
> >
> > 2) Andrew and Jan suggested that we can use segment descriptor to store Xen
> >    image base address. This way all references to variables in Xen image can
> >    be easily calculated during build and they will be static. However, relevant
> >    segment descriptor must be updated during Xen start.
> >
> >    At first sight, it looks that %cs, %ds, %es, %ss should be initialized as is
> >    (start: 0, size: 4 GiB, ...). This way we will not break references to trampoline
> >    and all other variables living outside of Xen image. Additionally, we are sure
> >    that all references to variables from C code (xen/arch/x86/boot/reloc.c) are
> >    pointing always to the same place regardless of x86 instruction generated by
> >    compiler and segment register used by this instruction for addressing. So, it
> >    looks that potentially we can use %fs or %gs as segment register to access
> >    variables living in Xen image. Looks good... Case a and b from solution #1 seams
> >    easy to resolve by prefixing by chosen segment register. Case c requires an extra
> >    register to store temporarily Xen image base address. Looks quite easy... However,
> >    looking at xen/arch/x86/boot/cmdline.S (at first sight this is the worst thing)
> >    I am not so happy with segment register solution. Unfortunately all functions
> >    get as argument just displacement in segment. This means that we must rework
> >    all stuff there and pass segment(s) as additional argument(s) because most of
> >    code checks memory in and out of Xen image in parallel. Of course we can use
> >    solution similar to case c described above but I think that then whole stuff
> >    move closer to idea #1. This way we will have something which uses things
> >    from #1 and #2. I do not think that this is solution which we want.
> >
> > 3) There is a third solution which is a mixture of #1 and #2. We can use e.g.
> >    %fs:0 (e.g. located somewhere in Xen image) to store Xen image base address.
> >    If this value is needed then we can access it directly, e.g. add %fs:0,%esi,
> >    or copy to temporary register and use as required.
> >
> > So, I am still in favor of #1. It is clean and easy. It does not require a lot of work.
> > #2 has a potential but requires a lot of changes in fragile cmdline.S (maybe others
> > difficult places). Is it worth? I think that #3 is a backup solution in case we choose
> > #1 and later it will appear that we need a globally unused register.
>
> I really don't understand why we have command line parsing written asm.
> I had half a mind to rewrite it all in C, (similar to the existing reloc
> code, but have gcc just create the .s and #include it as before, so
> external references work in a rational way).  It would be far clearer
> like that.

Yep, I remember that you said that once. Do you suggest that we should
do that right now? For me it make sense. Maybe we should also remove
other crazy stuff in that way.

> As stated at the hackathon, the problem with using %ebp is that it turns
> all implicit %ds references into implicit %ss references, and tends to

I am aware of that. However, I think that this is not a problem right
now because %ds == %ss. Additionally, I do not think that it will change
in the future.

> add a SIB+imm32 to each instruction with a memory reference (ebp
> relative memory references, and r13 for that matter, have restrictions
> in the way in which they can be encoded).

I am not sure what are you talking about here. What do you mean by r13?
Register name? In 32-bit mode? Could you point me a paragraph in Intel
or AMD docs which says about this restrictions?

Daniel

Re: Relocatable Xen early boot code

[Andrew Cooper]

On 12/06/15 15:36, Daniel Kiper wrote:
>> As stated at the hackathon, the problem with using %ebp is that it turns
>> all implicit %ds references into implicit %ss references, and tends to
> I am aware of that. However, I think that this is not a problem right
> now because %ds == %ss.

Some data segment attributes have different meanings when loaded into
%ss, than when loaded into any of the plain data segment registers.  I
believe what we have is compatible between the two. but...

> Additionally, I do not think that it will change in the future.

Stack references behave differently to data references.  For example, a
non-canonical addresses generate a #GP fault for data references, or a
#SS fault for stack references.

The concern with implicit stack references comes from the subtle
behaviour changes, not because it won't work in the general case.

>
>> add a SIB+imm32 to each instruction with a memory reference (ebp
>> relative memory references, and r13 for that matter, have restrictions
>> in the way in which they can be encoded).
> I am not sure what are you talking about here. What do you mean by r13?
> Register name? In 32-bit mode? Could you point me a paragraph in Intel
> or AMD docs which says about this restrictions?

This isn't a restriction pe say.  It is a properly of ModRM/SIB
encoding.  It is described in both the Intel and AMD instruction
reference manuals.

~Andrew