Hypervisor compilation for arm without debug=y

Hypervisor compilation for arm without debug=y

[Andrii Anisov]

[-- Attachment #1.1: Type: text/plain, Size: 872 bytes --]

Hello,

I tried to get hypervisor built without debug=y option.
Unfortunately built binary doesn't boot.
I've localized a problem and get a workaround for it:

diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
index d1290cd..aa8e057 100644
--- a/xen/arch/arm/mm.c
+++ b/xen/arch/arm/mm.c
@@ -332,7 +332,7 @@ void __cpuinit setup_virt_paging(void)

 /* Boot-time pagetable setup.
  * Changes here may need matching changes in head.S */
-void __init setup_pagetables(unsigned long boot_phys_offset, paddr_t
xen_paddr)
+void __attribute__((optimize("-fno-omit-frame-pointer"))) __init
setup_pagetables(unsigned long boot_phys_offset, paddr_t xen_paddr)
 {
     unsigned long dest_va;
     lpae_t pte, *p;

Is it a known issue? Is there a better solution for it?
I work with 4.3 release, build with gcc version 4.6.3 (Ubuntu/Linaro
4.6.3-1ubuntu5)

Sincerely,
Andrii Anisov.

[-- Attachment #1.2: Type: text/html, Size: 1214 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Hypervisor compilation for arm without debug=y

[Ian Campbell]

On Mon, 2013-08-19 at 14:44 +0300, Andrii Anisov wrote:
> Hello, 
> 
> 
> I tried to get hypervisor built without debug=y option.
> Unfortunately built binary doesn't boot.
> I've localized a problem and get a workaround for it:

Thanks.

> diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
> index d1290cd..aa8e057 100644
> --- a/xen/arch/arm/mm.c
> +++ b/xen/arch/arm/mm.c
> @@ -332,7 +332,7 @@ void __cpuinit setup_virt_paging(void)
>  
>  /* Boot-time pagetable setup.
>   * Changes here may need matching changes in head.S */
> -void __init setup_pagetables(unsigned long boot_phys_offset, paddr_t xen_paddr)
> +void __attribute__((optimize("-fno-omit-frame-pointer"))) __init setup_pagetables(unsigned long boot_phys_offset, paddr_t xen_paddr)
>  {
>      unsigned long dest_va;
>      lpae_t pte, *p;
> 
> 
> Is it a known issue?

I suspect none of us has ever really used a non-debug build!

> Is there a better solution for it?

So the confusing double negative -fno-omit-frame-pointer is causing the
frame pointer to be omitted for this function?

I wonder why the frame pointer is a problem here. This function is a bit
sensitive since it relocates Xen in physical RAM but it doesn't change
the virtual address space so I would expect fp, sp etc to remain valid.

The danger is if something is written to the stack between the initial
copy and the switch over, but I'd have thought that the frame pointer
would be written to the stack near the start and popped at the end and
mostly not touched in the middle.

I guess what I'm trying to say is I'm not confident that adding that
flag has "fixed" the issue, rather than just causing the compiler to
behave a bit differently and avoid it.

Are you able to post any details about the boot failure? Is it a crash
or something more subtle? Are you able to localise it to a particular
instruction or section of code?

> I work with 4.3 release, build with gcc version 4.6.3 (Ubuntu/Linaro
> 4.6.3-1ubuntu5) 

For ARM stuff you are probably better off tracking mainline (unstable)
xen. Depending on what you are doing of course.

Ian.

Re: Hypervisor compilation for arm without debug=y

[Andrii Anisov]

[-- Attachment #1.1: Type: text/plain, Size: 1778 bytes --]

>
> So the confusing double negative -fno-omit-frame-pointer is causing the
> frame pointer to be omitted for this function?

Not exactly, debug=y forces -fno-omit-frame-pointer into CFLAG. Without
debug, it is built without this option.
I just forced -fno-omit-frame-pointer for the particular function.


> I wonder why the frame pointer is a problem here. This function is a bit
> sensitive since it relocates Xen in physical RAM but it doesn't change
> the virtual address space so I would expect fp, sp etc to remain valid.
>
> The danger is if something is written to the stack between the initial
> copy and the switch over, but I'd have thought that the frame pointer
> would be written to the stack near the start and popped at the end and
> mostly not touched in the middle.
>
 Well, I roughly compared the function dumps for both cases (with frame
pointer and without), and noticed more aggressive pc-related data
manipulation if built without fp. This could be a cause, f.e. if data
modified in the old place, while than read from new location.

I guess what I'm trying to say is I'm not confident that adding that
> flag has "fixed" the issue, rather than just causing the compiler to
> behave a bit differently and avoid it.

I haven't root caused the issue so I do not affirm it's a fix. It's just a
WA works for me.


> Are you able to post any details about the boot failure? Is it a crash
> or something more subtle? Are you able to localise it to a particular
> instruction or section of code?

Unfortunately I can't localize particular instruction. JTAG I have does not
allow stepping through and use HW BP in HYP mode. With SW BP I just
localized an issue in this function, but it isn't right tool to debug that
relocation stuff.

Sincerely,
Andrii Anisov.

[-- Attachment #1.2: Type: text/html, Size: 2775 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Hypervisor compilation for arm without debug=y

[Andrii Anisov]

[-- Attachment #1.1: Type: text/plain, Size: 394 bytes --]

BTW,

> Is it a known issue?
>
> I suspect none of us has ever really used a non-debug build!
>

Wiki says:

Once you have a suitable cross compiler you can compile Xen with:
>    $ make dist-xen XEN_TARGET_ARCH=arm32
> CROSS_COMPILE=arm-unknown-linux-gnueabi-
> or:
>    $ make dist-xen XEN_TARGET_ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-


Without debug=y flag.

Sincerely,
Andrii Anisov.

[-- Attachment #1.2: Type: text/html, Size: 1473 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Hypervisor compilation for arm without debug=y

[Julien Grall]

On 08/20/2013 05:03 PM, Andrii Anisov wrote:
> BTW,
> 
>     > Is it a known issue?
> 
>     I suspect none of us has ever really used a non-debug build!
> 
> 
> Wiki says: 
> 
>     Once you have a suitable cross compiler you can compile Xen with:
>        $ make dist-xen XEN_TARGET_ARCH=arm32
>     CROSS_COMPILE=arm-unknown-linux-gnueabi-
>     or:
>        $ make dist-xen XEN_TARGET_ARCH=arm64
>     CROSS_COMPILE=aarch64-linux-gnu-
> 
> 
> Without debug=y flag.

On unstable branch by default debug=y. On release branches have debug=n
by default.

I have tried to boot a non-debug Xen on the Versatile Express:
   gcc 4.8.1: Xen boots
   gcc 4.6.3: Xen hangs with no output log (even early printk)

Cheers,

-- 
Julien Grall

Re: Hypervisor compilation for arm without debug=y

[Julien Grall]

On 08/20/2013 05:59 PM, Julien Grall wrote:
> On 08/20/2013 05:03 PM, Andrii Anisov wrote:
>> BTW,
>>
>>     > Is it a known issue?
>>
>>     I suspect none of us has ever really used a non-debug build!
>>
>>
>> Wiki says: 
>>
>>     Once you have a suitable cross compiler you can compile Xen with:
>>        $ make dist-xen XEN_TARGET_ARCH=arm32
>>     CROSS_COMPILE=arm-unknown-linux-gnueabi-
>>     or:
>>        $ make dist-xen XEN_TARGET_ARCH=arm64
>>     CROSS_COMPILE=aarch64-linux-gnu-
>>
>>
>> Without debug=y flag.
> 
> On unstable branch by default debug=y. On release branches have debug=n
> by default.
> 
> I have tried to boot a non-debug Xen on the Versatile Express:
>    gcc 4.8.1: Xen boots
>    gcc 4.6.3: Xen hangs with no output log (even early printk)

I was wrong, when debug=n it's not possible to have early printk. So Xen
only hangs.

But, if I enable early printk (by hacking xen/arch/arm/Rules.mk) Xen is
able to boot...


-- 
Julien Grall

Re: Hypervisor compilation for arm without debug=y

[Andrii Anisov]

[-- Attachment #1.1: Type: text/plain, Size: 278 bytes --]

>
> But, if I enable early printk (by hacking xen/arch/arm/Rules.mk) Xen is
> able to boot...
>

Yep,

I've seen this as well. But we are experimenting with boot time so early
printk is not an option for us.
Disabling debug gave us -30ms from 600 ms.

Sincerely,
Andrii Anisov.

[-- Attachment #1.2: Type: text/html, Size: 893 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel