[Qemu-devel] MIPS kernel hanging when loaded through U-Boot in qemu

[Qemu-devel] MIPS kernel hanging when loaded through U-Boot in qemu

[Thomas Petazzoni]

[-- Attachment #1: Type: text/plain, Size: 2891 bytes --]

Hi,

I'm trying to get a MIPS kernel to boot in qemu-system-mips when loaded
through U-Boot, but the kernel boot hangs at random locations. Let me
explain the whole thing. I'm running Qemu SVN-5089.

First, I have a 2.6.24.7 kernel configured for the "qemu" machine of
the MIPS architecture. The config file is available at
 http://toulibre.org/~thomas/qemu/config-2.6.24.7

When I boot this kernel using the -kernel option, it works perfectly,
as can be seen in
 http://toulibre.org/~thomas/qemu/qemu-log-kernel

(well it hangs because it cannot find a root filesystem, but this is
expected)

The ELF binary of this kenel is available at
 http://toulibre.org/~thomas/qemu/vmlinux

Now, to the problem. I compile U-Boot 1.3.4 for the qemu-mips machine,
and boot into it using
~/local/qemu/mips-softmmu/qemu-system-mips -M mips -pflash u-boot.bin
-net nic -net tap -serial stdio

U-Boot boots correctly, I can download the kernel using TFTP, flash it,
and boot it. I use the exact same kernel, except that I use the
binary-only arch/mips/boot/vmlinux.bin instead of the ELF file. Of
course the vmlinux.bin has been prepared using mkimage before being
downloaded by U-Boot. When I boot this kernel in U-Boot using the
'bootm' command, it starts, but then hangs:
 http://toulibre.org/~thomas/qemu/qemu-log-kernel-from-uboot

It always hangs around the same place, but not exactly. Sometimes after
"PID hash table entries", sometimes after "Console: colour dummy
devices", sometimes one or two messages later, or before.

Using the qemu monitor, I can see where the kernel hanged:
 http://toulibre.org/~thomas/qemu/qemu-monitor-showing-hang-location.png

It hanged at 0x80000180, which if I remember correctly my old MIPS
knowledge, is an exception vector location. And the address that
trigerred this exception is 0x80018904, which according to an objdump
of the kernel, is located in handle_sys().

Then, when I use gdbserver, put a breakpoint in handle_sys() to get a
backtrace, the backtrace is different at each boot. Two examples of
backtraces:
 http://toulibre.org/~thomas/qemu/qemu-backtrace-1
 http://toulibre.org/~thomas/qemu/qemu-backtrace-2

Seing handle_sys() being called at that point of the kernel
initialization looks strange to me, as userspace isn't running yet.

For those who want to test, the u-boot.bin image which is the flash
image containing both U-Boot and the kernel can be downloaded from:
 http://toulibre.org/~thomas/qemu/u-boot.bin

Just let U-Boot boot, the default command does the right thing.

Do you have any idea on what's going on ?

Thanks,

Thomas
-- 
Thomas Petazzoni, thomas.petazzoni@enix.org, http://thomas.enix.org
Jabber, thomas.petazzoni@jabber.dk
Toulibre, http://www.toulibre.org - APRIL, http://www.april.org
Fingerprint : 0BE1 4CF3 CEA4 AC9D CC6E  1624 F653 CB30 98D3 F7A7

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [Qemu-devel] MIPS kernel hanging when loaded through U-Boot in qemu

[Thiemo Seufer]

Thomas Petazzoni wrote:
> Hi,
> 
> I'm trying to get a MIPS kernel to boot in qemu-system-mips when loaded
> through U-Boot, but the kernel boot hangs at random locations. Let me
> explain the whole thing. I'm running Qemu SVN-5089.
> 
> First, I have a 2.6.24.7 kernel configured for the "qemu" machine of
> the MIPS architecture. The config file is available at
>  http://toulibre.org/~thomas/qemu/config-2.6.24.7
> 
> When I boot this kernel using the -kernel option, it works perfectly,
> as can be seen in
>  http://toulibre.org/~thomas/qemu/qemu-log-kernel
> 
> (well it hangs because it cannot find a root filesystem, but this is
> expected)
> 
> The ELF binary of this kenel is available at
>  http://toulibre.org/~thomas/qemu/vmlinux
> 
> Now, to the problem. I compile U-Boot 1.3.4 for the qemu-mips machine,
> and boot into it using
> ~/local/qemu/mips-softmmu/qemu-system-mips -M mips -pflash u-boot.bin
> -net nic -net tap -serial stdio
> 
> U-Boot boots correctly, I can download the kernel using TFTP, flash it,
> and boot it. I use the exact same kernel, except that I use the
> binary-only arch/mips/boot/vmlinux.bin instead of the ELF file.

If U-boot jumps just to the start of vmlinux.bin then you need to
have CONFIG_BOOT_RAW enabled. Otherwise you get early exceptions,
which the U-Boot routines attempt to handle.

> Of
> course the vmlinux.bin has been prepared using mkimage before being
> downloaded by U-Boot. When I boot this kernel in U-Boot using the
> 'bootm' command, it starts, but then hangs:
>  http://toulibre.org/~thomas/qemu/qemu-log-kernel-from-uboot
> 
> It always hangs around the same place, but not exactly. Sometimes after
> "PID hash table entries", sometimes after "Console: colour dummy
> devices", sometimes one or two messages later, or before.
> 
> Using the qemu monitor, I can see where the kernel hanged:
>  http://toulibre.org/~thomas/qemu/qemu-monitor-showing-hang-location.png
> 
> It hanged at 0x80000180, which if I remember correctly my old MIPS
> knowledge, is an exception vector location. And the address that
> trigerred this exception is 0x80018904, which according to an objdump
> of the kernel, is located in handle_sys().
> 
> Then, when I use gdbserver, put a breakpoint in handle_sys() to get a
> backtrace, the backtrace is different at each boot. Two examples of
> backtraces:
>  http://toulibre.org/~thomas/qemu/qemu-backtrace-1
>  http://toulibre.org/~thomas/qemu/qemu-backtrace-2
> 
> Seing handle_sys() being called at that point of the kernel
> initialization looks strange to me, as userspace isn't running yet.

Try a breakpoint on kernel_entry and check if it triggers.


Thiemo

Re: [Qemu-devel] MIPS kernel hanging when loaded through U-Boot in qemu

[Thomas Petazzoni]

[-- Attachment #1: Type: text/plain, Size: 1493 bytes --]

Le Thu, 28 Aug 2008 11:00:42 +0200,
Thomas Petazzoni <thomas.petazzoni@enix.org> a écrit :

> It always hangs around the same place, but not exactly. Sometimes
> after "PID hash table entries", sometimes after "Console: colour dummy
> devices", sometimes one or two messages later, or before.

It seems to hang when the first interrupt occurs, which might explain
why it hangs at a random place, but always around the same point.

I've added a small debug message in target-mips/helper.c:do_interrupt()
to see what's happening:
 http://toulibre.org/~thomas/qemu/qemu-interrupt-log-patch

With just the kernel (booted with -kernel), everything is fine, we see
a couple of interrupt of type "interrupt", and the boot proceeds:
 http://toulibre.org/~thomas/qemu/qemu-interrupt-log-kernel-only

But with U-Boot, once the first interrupt of type "interrupt" is fired,
it is followed by an interrupt of type "syscall", and then of type
"interrupt" again, and so on and so on, indefinitely:
 http://toulibre.org/~thomas/qemu/qemu-interrupt-log-with-u-boot

As I'm not a MIPS expert, I don't know what's happening here. Is it a
bug in U-Boot that leaves an incorrect CPU state ? Is it a Qemu
emulation problem ?

Thanks,

Thomas
-- 
Thomas Petazzoni, thomas.petazzoni@enix.org, http://thomas.enix.org
Jabber, thomas.petazzoni@jabber.dk
Toulibre, http://www.toulibre.org - APRIL, http://www.april.org
Fingerprint : 0BE1 4CF3 CEA4 AC9D CC6E  1624 F653 CB30 98D3 F7A7

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [Qemu-devel] MIPS kernel hanging when loaded through U-Boot in qemu

[Thiemo Seufer]

Thomas Petazzoni wrote:
> Le Thu, 28 Aug 2008 11:00:42 +0200,
> Thomas Petazzoni <thomas.petazzoni@enix.org> a écrit :
> 
> > It always hangs around the same place, but not exactly. Sometimes
> > after "PID hash table entries", sometimes after "Console: colour dummy
> > devices", sometimes one or two messages later, or before.
> 
> It seems to hang when the first interrupt occurs, which might explain
> why it hangs at a random place, but always around the same point.
> 
> I've added a small debug message in target-mips/helper.c:do_interrupt()
> to see what's happening:
>  http://toulibre.org/~thomas/qemu/qemu-interrupt-log-patch
> 
> With just the kernel (booted with -kernel), everything is fine, we see
> a couple of interrupt of type "interrupt", and the boot proceeds:
>  http://toulibre.org/~thomas/qemu/qemu-interrupt-log-kernel-only

'Interrupt' at this point should be the normal timer interrupt, "syscall"
are the execve() calls which start kernel threads. On classic mips, both
types of exceptions use the general exception vector at 0x80000180.

> But with U-Boot, once the first interrupt of type "interrupt" is fired,
> it is followed by an interrupt of type "syscall", and then of type
> "interrupt" again, and so on and so on, indefinitely:
>  http://toulibre.org/~thomas/qemu/qemu-interrupt-log-with-u-boot
> 
> As I'm not a MIPS expert, I don't know what's happening here. Is it a
> bug in U-Boot that leaves an incorrect CPU state ? Is it a Qemu
> emulation problem ?

The difference here is that the timer interrupt goes to 0x80000200,
this is controlled by the IV bit in the Cause register. This feature
isn't available on all CPUs. In the kernel the relevant check to test
for it is cpu_has_divec. I figure U-Boot and the Kernel disagree
on the setting.

Qemu always allows to set this Cause bit, independent of the CPU type.
So I figure we have two bugs:
 - The kernel should try to clear the IV bit if it doesn't intend to
   use it
 - Qemu should ignore attempts to set the IV bit when emulating CPUs
  without divec.


Thiemo

Re: [Qemu-devel] MIPS kernel hanging when loaded through U-Boot in qemu

[Thomas Petazzoni]

[-- Attachment #1: Type: text/plain, Size: 1805 bytes --]

Le Wed, 3 Sep 2008 12:54:47 +0200,
Thiemo Seufer <ths@networkno.de> a écrit :

> 'Interrupt' at this point should be the normal timer interrupt,
> "syscall" are the execve() calls which start kernel threads. On
> classic mips, both types of exceptions use the general exception
> vector at 0x80000180.

What's strange about these "syscall" interrupts is that we don't see
them in the kernel-only boot
(http://toulibre.org/~thomas/qemu/qemu-interrupt-log-kernel-only).

Are you sure that the syscall interrupt is used to run do_fork() inside
the kernel ? I'm not so sure.

> The difference here is that the timer interrupt goes to 0x80000200,
> this is controlled by the IV bit in the Cause register. This feature
> isn't available on all CPUs. In the kernel the relevant check to test
> for it is cpu_has_divec. I figure U-Boot and the Kernel disagree
> on the setting.

Hehe, it seems that you're correct.

In U-Boot board/qemu-mips/lowlevel_init.S, we have:

        /*
         * Step 7) Establish Cause
         * (set IV bit)
         */
        li      t1, 0x00800000
        mtc0    t1, CP0_CAUSE

In the kernel include/asm-mips/mach-qemu/cpu-feature-overrides.h, we
have:

#define cpu_has_divec           0

> Qemu always allows to set this Cause bit, independent of the CPU type.
> So I figure we have two bugs:
>  - The kernel should try to clear the IV bit if it doesn't intend to
>    use it
>  - Qemu should ignore attempts to set the IV bit when emulating CPUs
>   without divec.

Probably :-)

Thomas
-- 
Thomas Petazzoni, thomas.petazzoni@enix.org, http://thomas.enix.org
Jabber, thomas.petazzoni@jabber.dk
Toulibre, http://www.toulibre.org - APRIL, http://www.april.org
Fingerprint : 0BE1 4CF3 CEA4 AC9D CC6E  1624 F653 CB30 98D3 F7A7

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [Qemu-devel] MIPS kernel hanging when loaded through U-Boot in qemu

[Thomas Petazzoni]

[-- Attachment #1: Type: text/plain, Size: 1445 bytes --]

Le Wed, 3 Sep 2008 14:03:37 +0200,
Thomas Petazzoni <thomas.petazzoni@enix.org> a écrit :

> > Qemu always allows to set this Cause bit, independent of the CPU
> > type. So I figure we have two bugs:
> >  - The kernel should try to clear the IV bit if it doesn't intend to
> >    use it

I tried this approach, using the following patch to the kernel, and it
now works, the kernel boots successfully !

This solves my problem of demoing U-Boot+kernel on MIPS, but it's quite
useless for others since the qemu-mips platform has been removed from
the kernel between .24 and .25. Except if the bug is present for other
platforms, in which case Qemu could probably be fixed as you suggested ?

Thanks!

Thomas

---
 arch/mips/kernel/traps.c |    3 +++
 1 file changed, 3 insertions(+)

Index: kernel.old/arch/mips/kernel/traps.c
===================================================================
--- kernel.old.orig/arch/mips/kernel/traps.c
+++ kernel.old/arch/mips/kernel/traps.c
@@ -1351,6 +1351,9 @@
 		} else
 			set_c0_cause(CAUSEF_IV);
 	}
+	else {
+		clear_c0_cause(CAUSEF_IV);
+	}
 
 	/*
 	 * Before R2 both interrupt numbers were fixed to 7, so on R2 only:

-- 
Thomas Petazzoni, thomas.petazzoni@enix.org, http://thomas.enix.org
Jabber, thomas.petazzoni@jabber.dk
Toulibre, http://www.toulibre.org - APRIL, http://www.april.org
Fingerprint : 0BE1 4CF3 CEA4 AC9D CC6E  1624 F653 CB30 98D3 F7A7

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]