[Qemu-devel] vexpress-a9: coreboot is unable to push any data on stack

[Qemu-devel] vexpress-a9: coreboot is unable to push any data on stack

[Piotr Król]

Hi all,
I'm trying to boot latest coreboot on qemu-system-arm (-M vexpress-a9).
I compiled qemu arm-softmmu from source (latest code) and trying to boot
with:

qemu-system-arm -M vexpress-a9 -m 1024M -nographic -bios build/coreboot.rom

First of all mtree looks strange because there are two aliases to one
memory region:

0000000000000000-0000000003ffffff (prio 0, RW): alias  vexpress.flashalias @vexpress.flash0 0000000000000000-0000000003ffffff
0000000000000000-0000000003ffffff (prio 0, RW): alias  vexpress.lowmem @vexpress.highmem 0000000000000000-0000000003ffffff

Is this ok ?

Second, because VE_NORFLASHALIAS was change to 0 here:
http://git.qemu.org/?p=qemu.git;a=commit;h=6ec1588e09770ac7e9c60194faff6101111fc7f0

I was unable to boot coreboot using:
qemu-system-arm -M vexpress-a9 -m 1024M -nographic -kernel build/coreboot.rom

If I change VE_NORFLASHALIAS to -1 like it was before coreboot boots using
'-kernel' parameter, but '-bios' flag even then doesn't work.

Third, (for '-bios' case) thing is that coreboot is unable to push any value on
stack. Instructions related to stack change stack pointer but stack address
memory dump show only 0xffffffff. Coreboot perform stack initialization on its
own.

Disassembled coreboot bootblock with stack initialization is here:
https://gist.github.com/pietrushnic/f83d5d0e5d8d1b75d4c2

The code flow is:
_rom
|-> reset
    |-> init_stack_loop
        |-> call_bootblock
            |-> main
                |-> armv7_invalidate_caches
                    |-> icache_invalidate_all
                    |-> dcache_invalidate_all
                      |-> dcache_foreach

At the begging of dcache_foreach we execute:
stmdb  sp!, {r0, r1, r4, r5, r6, r7, r9, sl, fp, lr}

And at the end there is:
ldmia.w sp!, {r2, r3, r4, r5, r6, r7, r9, r10, r11, pc}

Unfortunately, as I wrote, on stack we have all 0xffffffff that's why
dcache_foreach finish emulation with:

qemu: fatal: Trying to execute code outside RAM or ROM at 0xfffffffe

Any idea how to debug this issue or why it happens in that way ?
What could be the reason of unusable stack memory ?

P.S. Before I started thread here there was a discussion on coreboot mailing list:
http://www.coreboot.org/pipermail/coreboot/2014-August/078378.html

Thanks,
Piotr

Re: [Qemu-devel] vexpress-a9: coreboot is unable to push any data on stack

[Peter Maydell]

On 15 August 2014 11:07, Piotr Król <pietrushnic@gmail.com> wrote:
> Hi all,
> I'm trying to boot latest coreboot on qemu-system-arm (-M vexpress-a9).
> I compiled qemu arm-softmmu from source (latest code) and trying to boot
> with:
>
> qemu-system-arm -M vexpress-a9 -m 1024M -nographic -bios build/coreboot.rom
>
> First of all mtree looks strange because there are two aliases to one
> memory region:
>
> 0000000000000000-0000000003ffffff (prio 0, RW): alias  vexpress.flashalias @vexpress.flash0 0000000000000000-0000000003ffffff
> 0000000000000000-0000000003ffffff (prio 0, RW): alias  vexpress.lowmem @vexpress.highmem 0000000000000000-0000000003ffffff
>
> Is this ok ?

Well, yes and no. It's not intentional, but the effect is that
we'll happen to map the flash at address zero and ignore
the RAM alias.

I'm afraid when I put in the code to add the flash alias at
address zero I missed that the reason for it not being
already in the code was the legacy aliasing of RAM at zero.

For your stack issues, it looks like your code is trying to
use the area which is the flash as the stack. Since flash
isn't writeable, we ignore the writes and it's not very
useful for stack. It looks like your code is assuming that
the low memory is RAM, not flash -- so how does your
code work on real hardware? Do you try to use the
software controllable remapping to copy from the flash
into RAM before using the stack, or something else?

In terms of where we go from here, we have two
choices:
 (1) leave address 0 as RAM, not flash; this means
legacy guest binaries that work only on QEMU and
not on real hardware will still work, but the -bios
option won't be of much use. (This is more or less
reverting to the 2.0 situation.)
 (2) bring it in to line with vexpress-a15 (which is
effectively how 2.1 shipped), so 0 is always flash
and never RAM. This is consistent but (as you've
found) binaries assuming that 0 is a RAM alias
will stop working.

thanks
-- PMM

Re: [Qemu-devel] vexpress-a9: coreboot is unable to push any data on stack

[Piotr Król]

On Fri, Aug 15, 2014 at 05:10:04PM +0100, Peter Maydell wrote:
> For your stack issues, it looks like your code is trying to
> use the area which is the flash as the stack. Since flash
> isn't writeable, we ignore the writes and it's not very
> useful for stack. It looks like your code is assuming that
> the low memory is RAM, not flash -- so how does your
> code work on real hardware? Do you try to use the
> software controllable remapping to copy from the flash
> into RAM before using the stack, or something else?

Peter thanks for your reply. I'm not coreboot developer so cannot
advocate for their decision. Bootblock that I sent is from build
targeted on qemu and developer who initially wrote the code suggest
using it with '-kernel' parameter. I would like to fix this code
according to correct memory map.

Comment in hw/arm/vexpress.c say that, as you wrote below, Versatile
Express got two possible memory maps. Can you point me to exact
documentation that you use as reference for vexpress implementation ?

> 
> In terms of where we go from here, we have two
> choices:
>  (1) leave address 0 as RAM, not flash; this means
> legacy guest binaries that work only on QEMU and
> not on real hardware will still work, but the -bios
> option won't be of much use. (This is more or less
> reverting to the 2.0 situation.)
>  (2) bring it in to line with vexpress-a15 (which is
> effectively how 2.1 shipped), so 0 is always flash
> and never RAM. This is consistent but (as you've
> found) binaries assuming that 0 is a RAM alias
> will stop working.
> 

Assuming option (2) contain strategy for future releases it should be
priority. I will try to fix coreboot binary according to this advice.

One more time thanks for explanation.

Thanks,
Piotr Król

Re: [Qemu-devel] vexpress-a9: coreboot is unable to push any data on stack

[Peter Maydell]

On 15 August 2014 19:02, Piotr Król <pietrushnic@gmail.com> wrote:
> On Fri, Aug 15, 2014 at 05:10:04PM +0100, Peter Maydell wrote:
>> For your stack issues, it looks like your code is trying to
>> use the area which is the flash as the stack. Since flash
>> isn't writeable, we ignore the writes and it's not very
>> useful for stack. It looks like your code is assuming that
>> the low memory is RAM, not flash -- so how does your
>> code work on real hardware? Do you try to use the
>> software controllable remapping to copy from the flash
>> into RAM before using the stack, or something else?
>
> Peter thanks for your reply. I'm not coreboot developer so cannot
> advocate for their decision.

I'm not looking for advocacy, just for an explanation of
what it's doing, and in particular whether anybody's
ever tested this on real hardware. (Running guest code
only on QEMU is a recipe for bugs, because our emulation
is often not very accurate and if you only test on QEMU
you can end up accidentally relying on our bugs.)

> Bootblock that I sent is from build
> targeted on qemu and developer who initially wrote the code suggest
> using it with '-kernel' parameter. I would like to fix this code
> according to correct memory map.
>
> Comment in hw/arm/vexpress.c say that, as you wrote below, Versatile
> Express got two possible memory maps. Can you point me to exact
> documentation that you use as reference for vexpress implementation ?

The common motherboard docs:
http://infocenter.arm.com/help/topic/com.arm.doc.dui0447j/index.html
A9 daughterboard docs:
http://infocenter.arm.com/help/topic/com.arm.doc.dui0448h/index.html
A15 daughterboard docs:
http://infocenter.arm.com/help/topic/com.arm.doc.dui0604e/index.html

thanks
-- PMM

Re: [Qemu-devel] vexpress-a9: coreboot is unable to push any data on stack

[Piotr Król]

On Fri, Aug 15, 2014 at 11:54:55PM +0100, Peter Maydell wrote:
> I'm not looking for advocacy, just for an explanation of what it's
> doing, and in particular whether anybody's ever tested this on real
> hardware. (Running guest code only on QEMU is a recipe for bugs,
> because our emulation is often not very accurate and if you only test
> on QEMU you can end up accidentally relying on our bugs.)
> 
Creboot is an open source replacement for BIOS. Almost every supported
board contain its own memory map. According to commit message from
Chromium developer coreboot build for qemu vexpress-a9 was created to
'simplify testing ARM implementation (... and) to verify the boot loader
functionality'. I think that code for real hardware got little in common
with qemu implementation.

I'm trying to use this implementation for pure educational purposes.

Going back to my initial issues with stack, changing stack location to
SRAM improves situation but does not fix booting. It looks like whole
implementation assumes running from RAM.

> The common motherboard docs:
> http://infocenter.arm.com/help/topic/com.arm.doc.dui0447j/index.html
> A9 daughterboard docs:
> http://infocenter.arm.com/help/topic/com.arm.doc.dui0448h/index.html
> A15 daughterboard docs:
> http://infocenter.arm.com/help/topic/com.arm.doc.dui0604e/index.html
> 
Thanks,
Piotr

Re: [Qemu-devel] vexpress-a9: coreboot is unable to push any data on stack

[Peter Maydell]

On 16 August 2014 10:50, Piotr Król <pietrushnic@gmail.com> wrote:
> On Fri, Aug 15, 2014 at 11:54:55PM +0100, Peter Maydell wrote:
>> I'm not looking for advocacy, just for an explanation of what it's
>> doing, and in particular whether anybody's ever tested this on real
>> hardware. (Running guest code only on QEMU is a recipe for bugs,
>> because our emulation is often not very accurate and if you only test
>> on QEMU you can end up accidentally relying on our bugs.)
>>
> Creboot is an open source replacement for BIOS. Almost every supported
> board contain its own memory map. According to commit message from
> Chromium developer coreboot build for qemu vexpress-a9 was created to
> 'simplify testing ARM implementation (... and) to verify the boot loader
> functionality'. I think that code for real hardware got little in common
> with qemu implementation.

> I'm trying to use this implementation for pure educational purposes.
>
> Going back to my initial issues with stack, changing stack location to
> SRAM improves situation but does not fix booting. It looks like whole
> implementation assumes running from RAM.

It sounds like this guest code is just broken and needs fixing to
relocate itself into RAM somehow, then...

-- PMM