[Qemu-devel] boot arm fedora via u-boot in qemu

[Qemu-devel] boot arm fedora via u-boot in qemu

[Gerd Hoffmann]

  Hi,

Played around with u-boot a bit, trying to get it work in qemu, so you
don't have to hop through loops copying the kernel+initrd from the disk
image to pass it to qemu on the command line.

Guess why I'm asking here?  No so success so far :(

What I did:  Started with the fedora uboot-tools package, then tried to
adapt vexpress.  Look here:

https://www.kraxel.org/cgit/jenkins/u-boot/tree/

Patches 1-7 come straight from the fedora package.
Patch #8 is a bugfix.
Patch #9 is a debug patch (send logs to console).
Patch #10 adapts vexpress to the distro patches.
Patch #11 makes u-boot pick up the fdt from qemu.

All looks fine, uboot finds the virtual sdcard, loads kernel + initrd,
passes on the fdt.  kernel doesn't boot though, without printing any
helpful clues to the serial console.

Any ideas what this might be?
Any hints how to go on with debugging?

thanks,
  Gerd

--------- [ cut here: log from a boot attempt ] -------------
+ exec qemu-system-arm -nographic -M vexpress-a9 -m 1G \
  -kernel build-vexpress_ca9x4/u-boot \
  -dtb /path/to/vexpress-v2p-ca9.dtb -sd f21.img


U-Boot 2014.10-rc3-00275-g30ed9a4 (Oct 21 2014 - 12:49:54)

DRAM:  1 GiB
WARNING: Caches not enabled
Flash: 128 MiB
MMC:   MMC: 0
*** Warning - bad CRC, using default environment

In:    serial
Out:   serial
Err:   serial
QEMU: fdt found at 60000000, using it.
Net:   smc911x-0
Warning: smc911x-0 using MAC address from net device

Warning: Your board does not use generic board. Please read
doc/README.generic-board and take action. Boards not
upgraded by the late 2014 may break or be removed.
Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found /extlinux/extlinux.conf
Retrieving file: /extlinux/extlinux.conf
528 bytes read in 34 ms (14.6 KiB/s)
Ignoring unknown command: ui
Ignoring malformed menu command:  autoboot
Ignoring malformed menu command:  hidden
Ignoring unknown command: totaltimeout
Fedora-Minimal-armhfp-21-20140905 Boot Options.
1:      Fedora-Minimal-armhfp-21-20140905 (3.16.1-301.fc21.armv7hl)
Enter choice: 1
1:      Fedora-Minimal-armhfp-21-20140905 (3.16.1-301.fc21.armv7hl)
Retrieving file: /initramfs-3.16.1-301.fc21.armv7hl.img
29202689 bytes read in 3788 ms (7.4 MiB/s)
Retrieving file: /vmlinuz-3.16.1-301.fc21.armv7hl
5460688 bytes read in 762 ms (6.8 MiB/s)
append: ro root=UUID=55c3c475-37c8-41f8-ac34-3c153cde02a2
console=ttyAMA0,115200n8
Kernel image @ 0x80008000 [ 0x000000 - 0x5352d0 ]
## Flattened Device Tree blob at 60000000
   Booting using the fdt blob at 0x60000000
   Loading Ramdisk to 7e307000, end 7fee0901 ... OK
   Loading Device Tree to 7e2f8000, end 7e306339 ... OK

Starting kernel ...

Re: [Qemu-devel] boot arm fedora via u-boot in qemu

[Hans de Goede]

Hi Gerd,

On 10/21/2014 03:33 PM, Gerd Hoffmann wrote:
>   Hi,
> 
> Played around with u-boot a bit, trying to get it work in qemu, so you
> don't have to hop through loops copying the kernel+initrd from the disk
> image to pass it to qemu on the command line.
> 
> Guess why I'm asking here?  No so success so far :(
> 
> What I did:  Started with the fedora uboot-tools package, then tried to
> adapt vexpress.  Look here:
> 
> https://www.kraxel.org/cgit/jenkins/u-boot/tree/
> 
> Patches 1-7 come straight from the fedora package.
> Patch #8 is a bugfix.
> Patch #9 is a debug patch (send logs to console).
> Patch #10 adapts vexpress to the distro patches.
> Patch #11 makes u-boot pick up the fdt from qemu.
> 
> All looks fine, uboot finds the virtual sdcard, loads kernel + initrd,
> passes on the fdt.  kernel doesn't boot though, without printing any
> helpful clues to the serial console.
> 
> Any ideas what this might be?
> Any hints how to go on with debugging?
> 
> thanks,
>   Gerd
> 
> --------- [ cut here: log from a boot attempt ] -------------
> + exec qemu-system-arm -nographic -M vexpress-a9 -m 1G \
>   -kernel build-vexpress_ca9x4/u-boot \
>   -dtb /path/to/vexpress-v2p-ca9.dtb -sd f21.img
> 
> 
> U-Boot 2014.10-rc3-00275-g30ed9a4 (Oct 21 2014 - 12:49:54)
> 
> DRAM:  1 GiB
> WARNING: Caches not enabled
> Flash: 128 MiB
> MMC:   MMC: 0
> *** Warning - bad CRC, using default environment
> 
> In:    serial
> Out:   serial
> Err:   serial
> QEMU: fdt found at 60000000, using it.
> Net:   smc911x-0
> Warning: smc911x-0 using MAC address from net device
> 
> Warning: Your board does not use generic board. Please read
> doc/README.generic-board and take action. Boards not
> upgraded by the late 2014 may break or be removed.
> Hit any key to stop autoboot:  0 
> switch to partitions #0, OK
> mmc0 is current device
> Scanning mmc 0:1...
> Found /extlinux/extlinux.conf
> Retrieving file: /extlinux/extlinux.conf
> 528 bytes read in 34 ms (14.6 KiB/s)
> Ignoring unknown command: ui
> Ignoring malformed menu command:  autoboot
> Ignoring malformed menu command:  hidden
> Ignoring unknown command: totaltimeout
> Fedora-Minimal-armhfp-21-20140905 Boot Options.
> 1:      Fedora-Minimal-armhfp-21-20140905 (3.16.1-301.fc21.armv7hl)
> Enter choice: 1
> 1:      Fedora-Minimal-armhfp-21-20140905 (3.16.1-301.fc21.armv7hl)
> Retrieving file: /initramfs-3.16.1-301.fc21.armv7hl.img
> 29202689 bytes read in 3788 ms (7.4 MiB/s)
> Retrieving file: /vmlinuz-3.16.1-301.fc21.armv7hl
> 5460688 bytes read in 762 ms (6.8 MiB/s)
> append: ro root=UUID=55c3c475-37c8-41f8-ac34-3c153cde02a2
> console=ttyAMA0,115200n8
> Kernel image @ 0x80008000 [ 0x000000 - 0x5352d0 ]
> ## Flattened Device Tree blob at 60000000
>    Booting using the fdt blob at 0x60000000
>    Loading Ramdisk to 7e307000, end 7fee0901 ... OK
>    Loading Device Tree to 7e2f8000, end 7e306339 ... OK
> 
> Starting kernel ...

Here is what a working boot on an Allwinner board looks like:

Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0...
Found /extlinux/extlinux.conf
Retrieving file: /extlinux/extlinux.conf
769 bytes read in 100 ms (6.8 KiB/s)
Ignoring unknown command: ui
Ignoring malformed menu command:  autoboot
Ignoring malformed menu command:  hidden
Ignoring unknown command: totaltimeout
Fedora-Minimal-armhfp-rawhide-20140731 Boot Options.
1:      Fedora-Minimal-armhfp-rawhide-20140731 (zImage)
2:      Fedora-Minimal-armhfp-rawhide-20140731 (3.16.0-0.rc7.git1.1.fc22.armv7hl)
Enter choice: 2
2:      Fedora-Minimal-armhfp-rawhide-20140731 (3.16.0-0.rc7.git1.1.fc22.armv7hl)
Retrieving file: /initramfs-3.16.0-0.rc7.git1.1.fc22.armv7hl.img
29519929 bytes read in 1724 ms (16.3 MiB/s)
Retrieving file: /vmlinuz-3.16.0-0.rc7.git1.1.fc22.armv7hl
5932608 bytes read in 456 ms (12.4 MiB/s)
append: ro root=UUID=a6446dcb-320d-421f-9b71-725b68e5a41b console=ttyS0,115200
Retrieving file: /dtb-3.16.0-0.rc7.git1.1.fc22.armv7hl/sun7i-a20-bananapi.dtb
22335 bytes read in 2312 ms (8.8 KiB/s)
Kernel image @ 0x42000000 [ 0x000000 - 0x5a8640 ]
## Flattened Device Tree blob at 43000000
   Booting using the fdt blob at 0x43000000
   Loading Ramdisk to 4e3d8000, end 4ffff039 ... OK
   Loading Device Tree to 4e3cf000, end 4e3d773e ... OK

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.16.0-0.rc7.git1.1.fc22.armv7hl (mockbuild@arm02-builder06.arm.fedoraproject.org) (gcc version 4.9.1 20140717 (Red Hat 4.9.1-2) (GCC) ) #1 SMP Tue Jul 29 17:20:28 UTC 2014

Note how it is also actually reading the dtb from disk, where as in your case
it is not, so that seems to be the culprit.

Regards,

Hans

Re: [Qemu-devel] boot arm fedora via u-boot in qemu

[Peter Maydell]

On 21 October 2014 14:33, Gerd Hoffmann <kraxel@redhat.com> wrote:
> All looks fine, uboot finds the virtual sdcard, loads kernel + initrd,
> passes on the fdt.  kernel doesn't boot though, without printing any
> helpful clues to the serial console.
>
> Any ideas what this might be?

Probably we're not emulating some bit of h/w the way
u-boot is expecting. (Possibly u-boot expects us to support
trustzone and boot in the Secure world?)

> Any hints how to go on with debugging?

Use the gdbstub to find out where the kernel is getting
stuck? "Kernel produces no output" just means "we didn't
get far enough through early boot for the kernel to start
sending output to the UART. You might find it useful to
turn on the earlyprintk support, and/or to build the
kernel with debug-UART-via-semihosting and add the
-semihosting argument to the QEMU command line.

-- PMM

Re: [Qemu-devel] boot arm fedora via u-boot in qemu

[Gerd Hoffmann]

  Hi,

> Retrieving file: /dtb-3.16.0-0.rc7.git1.1.fc22.armv7hl/sun7i-a20-bananapi.dtb
> 22335 bytes read in 2312 ms (8.8 KiB/s)
> Kernel image @ 0x42000000 [ 0x000000 - 0x5a8640 ]
> ## Flattened Device Tree blob at 43000000
>    Booting using the fdt blob at 0x43000000
>    Loading Ramdisk to 4e3d8000, end 4ffff039 ... OK
>    Loading Device Tree to 4e3cf000, end 4e3d773e ... OK
> 
> Starting kernel ...
> 
> [    0.000000] Booting Linux on physical CPU 0x0
> [    0.000000] Initializing cgroup subsys cpuset
> [    0.000000] Initializing cgroup subsys cpu
> [    0.000000] Initializing cgroup subsys cpuacct
> [    0.000000] Linux version 3.16.0-0.rc7.git1.1.fc22.armv7hl (mockbuild@arm02-builder06.arm.fedoraproject.org) (gcc version 4.9.1 20140717 (Red Hat 4.9.1-2) (GCC) ) #1 SMP Tue Jul 29 17:20:28 UTC 2014
> 
> Note how it is also actually reading the dtb from disk, where as in your case
> it is not, so that seems to be the culprit.

If I don't pass the dtb to qemu u-boot will actually load it from disk,
looks like this:

U-Boot 2014.10-rc3-00276-g30fa548 (Oct 21 2014 - 18:56:38)

DRAM:  1 GiB
WARNING: Caches not enabled
Flash: 128 MiB
MMC:   MMC: 0
*** Warning - bad CRC, using default environment

In:    serial
Out:   serial
Err:   serial
QEMU: will try to load vexpress-v2p-ca9.dtb from disk.
Net:   smc911x-0
Warning: smc911x-0 using MAC address from net device

Warning: Your board does not use generic board. Please read
doc/README.generic-board and take action. Boards not
upgraded by the late 2014 may break or be removed.
Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found /extlinux/extlinux.conf
Retrieving file: /extlinux/extlinux.conf
528 bytes read in 34 ms (14.6 KiB/s)
Ignoring unknown command: ui
Ignoring malformed menu command:  autoboot
Ignoring malformed menu command:  hidden
Ignoring unknown command: totaltimeout
Fedora-Minimal-armhfp-21-20140905 Boot Options.
1:      Fedora-Minimal-armhfp-21-20140905 (3.16.1-301.fc21.armv7hl)
Enter choice: 1
1:      Fedora-Minimal-armhfp-21-20140905 (3.16.1-301.fc21.armv7hl)
Retrieving file: /initramfs-3.16.1-301.fc21.armv7hl.img
29202689 bytes read in 3854 ms (7.2 MiB/s)
Retrieving file: /vmlinuz-3.16.1-301.fc21.armv7hl
5460688 bytes read in 793 ms (6.6 MiB/s)
append: ro root=UUID=55c3c475-37c8-41f8-ac34-3c153cde02a2
console=ttyAMA0,115200n8 earlyprintk
Retrieving file: /dtb-3.16.1-301.fc21.armv7hl/vexpress-v2p-ca9.dtb
12941 bytes read in 771 ms (15.6 KiB/s)
Kernel image @ 0x80008000 [ 0x000000 - 0x5352d0 ]
## Flattened Device Tree blob at 87f00000
   Booting using the fdt blob at 0x87f00000
   Loading Ramdisk to 7e307000, end 7fee0901 ... OK
   Loading Device Tree to 7e300000, end 7e30628c ... OK

Starting kernel ...

Still not working.

I think when loading it from disk the dtb lacks the virtio-mmio nodes
which qemu will add to the dtb, so virtio devices will not work.
Shouldn't make much of a difference though when booting from sdcard.

cheers,
  Gerd

Re: [Qemu-devel] boot arm fedora via u-boot in qemu

[Gerd Hoffmann]

  Hi,

> > Any hints how to go on with debugging?
> 
> Use the gdbstub to find out where the kernel is getting
> stuck? "Kernel produces no output" just means "we didn't
> get far enough through early boot for the kernel to start
> sending output to the UART. You might find it useful to
> turn on the earlyprintk support, and/or to build the
> kernel with debug-UART-via-semihosting and add the
> -semihosting argument to the QEMU command line.

earlyprintk doesn't change anything.

Ok, that leaves gdb.  Stuck here:

(gdb) info reg
[ ... ]
lr       0xc0868bf4
pc       0xc04eee18

Looking them up in System.map finds me:

c04eee18 T __loop_delay
c0868a28 T panic

Ok, so the kernel came successfully though the uncompressing +
relocation + enable paging, but paniced before earlyprintk can print
something.

Guess I'll have to fetch the vmlinux with debug info so I can get a full
stack trace and see where it paniced, but not today.

cheers,
  Gerd

Re: [Qemu-devel] boot arm fedora via u-boot in qemu

[Peter Maydell]

On 21 October 2014 19:31, Gerd Hoffmann <kraxel@redhat.com> wrote:
> Looking them up in System.map finds me:
>
> c04eee18 T __loop_delay
> c0868a28 T panic
>
> Ok, so the kernel came successfully though the uncompressing +
> relocation + enable paging, but paniced before earlyprintk can print
> something.
>
> Guess I'll have to fetch the vmlinux with debug info so I can get a full
> stack trace and see where it paniced, but not today.

IIRC you should be able to fish the kernel log out of the
memory buffer where it's currently sitting. I forget what
the symbol name for that is, though.

-- PMM

Re: [Qemu-devel] boot arm fedora via u-boot in qemu

[Gerd Hoffmann]

On Di, 2014-10-21 at 20:11 +0100, Peter Maydell wrote:
> On 21 October 2014 19:31, Gerd Hoffmann <kraxel@redhat.com> wrote:
> > Looking them up in System.map finds me:
> >
> > c04eee18 T __loop_delay
> > c0868a28 T panic
> >
> > Ok, so the kernel came successfully though the uncompressing +
> > relocation + enable paging, but paniced before earlyprintk can print
> > something.
> >
> > Guess I'll have to fetch the vmlinux with debug info so I can get a full
> > stack trace and see where it paniced, but not today.
> 
> IIRC you should be able to fish the kernel log out of the
> memory buffer where it's currently sitting. I forget what
> the symbol name for that is, though.

Hmm, it says "Ignoring memory range 0x60000000 - 0x80000000".
That looks fishy.

"if (base < phys_offset)" in early_init_dt_add_memory_arch triggers.
Seems __pa() isn't working correctly.

Any chance this can be caused by setting some uboot env var incorrectly?
There is a confusing high number of *_addr_* variables ...

cheers,
  Gerd

Re: [Qemu-devel] boot arm fedora via u-boot in qemu

[Gerd Hoffmann]

  Hi,

> "if (base < phys_offset)" in early_init_dt_add_memory_arch triggers.
> Seems __pa() isn't working correctly.
> 
> Any chance this can be caused by setting some uboot env var incorrectly?
> There is a confusing high number of *_addr_* variables ...

Yep, that was it.  Linux kernel wants be loaded at the start of memory,
loading it somewhere else doesn't fly.  And neither kernel_addr nor
kernel_addr_r in upstream u-boot repo is correct.  Or maybe it works
when booting the kernel from flash or building it with different config
options.

Next problem:

When booting f20 image (no extlinux yet) uboot tries the boot script
instead, then fails this way:

[ ... ]
Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
18481 bytes read in 79 ms (227.5 KiB/s)
## Executing script at 67d00000
Welcome to a-b-c 0.59.

Found kernel_addr_r (0x60008000), using for u_k_addr.
Found ramdisk_addr_r (0x68000000), using for u_ramfs_addr.
Found fdt_addr_r (0x67f00000), using for u_dtb_addr.
Using defaults for vexpress-v2p-ca9.dtb, and will load dtb during
startup.
Will scan for kernels on mmc usb scsi sata ide, using ext2 fat
filesystems with bootm

Starting menu init....

syntax error
## Error: "catX" not defined
syntax error
## Error: "catX" not defined
No kernel provides dtb named vexpress-v2p-ca9.dtb
No value for u_kernel.  Getting help.
[ ... ]

Google finds me this:

https://fedoraproject.org/wiki/Architectures/ARM/F20/Installation#For_the_BeagleBone_Black

Suggested fix is to replace uboot.  Heh.

Anyone has clue what is going on and how to fix it?  I suspect it is
simply one of the many CONFIG_* switches in u-boot which must be
flipped?

cheers,
  Gerd


PS: https://www.kraxel.org/repos/ has builds

    yum install -y u-boot.git
    qemu-arm -M vexpress-a9 -m 512M -sd $image \
        -kernel /usr/share/u-boot.git/arm/vexpress-a9/u-boot

Re: [Qemu-devel] boot arm fedora via u-boot in qemu

[Rabin Vincent]

On Wed, Oct 22, 2014 at 11:59:06AM +0200, Gerd Hoffmann wrote:
> syntax error
> ## Error: "catX" not defined
> syntax error
> ## Error: "catX" not defined
> No kernel provides dtb named vexpress-v2p-ca9.dtb
> No value for u_kernel.  Getting help.
> [ ... ]
> 
> Google finds me this:
> 
> https://fedoraproject.org/wiki/Architectures/ARM/F20/Installation#For_the_BeagleBone_Black
> 
> Suggested fix is to replace uboot.  Heh.
> 
> Anyone has clue what is going on and how to fix it?  I suspect it is
> simply one of the many CONFIG_* switches in u-boot which must be
> flipped?

The problem is that commit fe9ca3d3287185 ("hush: fix some quoted
variable expansion issues") in U-Boot breaks this command in boot.scr.

 setenv catcat setenv catout\;'setenv catX "setenv catout '\\\\\\\''\$\$catin'\\\\\\\''"' \; run catX

Reverting that commit from U-Boot fixes the script problem, then you'll
need to set a few more variables to get the full boot working:

diff --git a/include/configs/vexpress_common.h b/include/configs/vexpress_common.h
index 54cb69f..72436d2 100644
--- a/include/configs/vexpress_common.h
+++ b/include/configs/vexpress_common.h
@@ -217,6 +217,9 @@
 		"scriptaddr="     "0x67d00000\0" \
 		"pxefile_addr_r=" "0x67e00000\0" \
 		"fdt_addr_r="     "0x67f00000\0" \
+		"u_boot="     "bootz\0" \
+		"u_kernel="     "3.11.10-301.fc20.armv7hl\0" \
+		"u_extraargs="     "console=ttyAMA0,115200n8\0" \
 		"ramdisk_addr_r=" "0x68000000\0"
 #elif defined(CONFIG_VEXPRESS_EXTENDED_MEMORY_MAP)
 #define CONFIG_PLATFORM_ENV_SETTINGS \