Kernel panics (in QEMU) when booting with GRUB2 + OVMF + QEMU.

Kernel panics (in QEMU) when booting with GRUB2 + OVMF + QEMU.

[Mark Lee]

[-- Attachment #1: Type: text/plain, Size: 2805 bytes --]

To all,

I've encountered a bug when using GRUB2 + OVMF + QEMU on Arch Linux 64 bit. 
The issue can be reproduced when running the latest git code from grub as 
well.

ISSUE: Kernel panics (in QEMU) when booting with GRUB2 + OVMF + QEMU.
------
    When booting using OVMF (ovmf_x64.bin) + QEMU, GRUB2 manages to load but 
kernel panics when loading its menu entry. This issue is not observed in 
hardware or via bios boot.

DOWNLOADS: test.img
----------
    To speed up testing, one can skip reproducing test.img and grab a copy at 
<https://drive.google.com/file/d/0Bx2cO4TT8XjeSEt2aEVkNEVsZUk/view?usp=sharing>
    
REPRODUCE test.img:
-------------------
# fallocate -l 3G test.img
# gdisk test.img <<<(
o
y
n
1

+1M
ef02

n
2

+100M
ef00

n
3




w
y
)
# kpartx -l test.img      /* Get the name of loop device <loop> */
# kpartx -a test.img
# mkfs.fat /dev/mapper/<loop>p2
# mkfs.ext4 /dev/mapper/<loop>p3 
# mount /dev/mapper/<loop>p3 /mnt
# mkdir /mnt/boot
# mount /dev/mapper/<loop>p2 /mnt/boot

/* This part is Arch Linux specific */
# pacstrap /mnt base grub
/* End Arch Linux specific */

# grub-install --target=i386-pc --boot-directory=/mnt/boot /dev/<loop>
# grub-install --target=x86_64-efi --efi-directory=/mnt/boot --boot-
directory=/mnt/boot --removable
# echo "
insmod all_video
menuentry "Arch Linux" {
  linux /vmlinuz-linux rw root=/dev/sda3
  initrd /initramfs-linux-fallback.img
}
" > /mnt/boot/grub/grub.cfg
# umount -R /mnt
# kpartx -d test.img

REPRODUCE QEMU ERROR:
---------------------
/* Bios works */
# qemu-system-x86_64 -enable-kvm -cpu host test.img

/* UEFI doesn't */
# qemu-system-x86_64 -enable-kvm -cpu host -bios /usr/share/ovmf/ovmf_x64.bin 
test.img

/* I get the following error */
Kernel panic - not syncing : VFS: Unable to mount root fs on Unknown-
block(0,0)
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.1.4-1-ARCH #1
Hardware name : QEMU Standard PC (i440FX + PIIX, 1996), BIOS
0000000000000000 000000007fac344c ffff880006527db8 ffffffff815865be
0000000000000000 ffffffff816f61b0 ffff880006527e38 ffffffff81585453
ffff880000000010 ffff880006527e48 ffff880006527de8 000000007fac344c
Call Trace:
[<ffffffff815865be>] dump_stack+0x4c/0x6e
[<ffffffff81585453>] panic+0xd0/0x203
[<ffffffff818fa621>] mount_block_root+0x297/0x2a6
[<ffffffff818fa684>] mount_root+0x54/0x58
[<ffffffff818fa7c0>] prepare_namespace+0x138/0x171
[<ffffffff818fa22e>] kernel_init_freeable+0x1ec/0x215
[<ffffffff8157cbb0>] ? rest_init+0x90/0x90
[<ffffffff8157cbbe>] kernel_init+0xe/0xf0
[<ffffffff8157c2a2>] ret_fromt_fork+0x42/0x70
[<ffffffff8157cbb0>] ? rest_init+0x90/0x90
Kernel Offset: disabled
---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-
block(0,0)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 213 bytes --]

Re: Kernel panics (in QEMU) when booting with GRUB2 + OVMF + QEMU.

[Andrei Borzenkov]

On 14.08.2015 09:45, Mark Lee wrote:
> To all,
>
> I've encountered a bug when using GRUB2 + OVMF + QEMU on Arch Linux 64 bit.
> The issue can be reproduced when running the latest git code from grub as
> well.
>
> ISSUE: Kernel panics (in QEMU) when booting with GRUB2 + OVMF + QEMU.

Once kernel is loaded, GRUB task is over. Why do you think it is GRUB issue?

> ------
>      When booting using OVMF (ovmf_x64.bin) + QEMU, GRUB2 manages to load but
> kernel panics when loading its menu entry. This issue is not observed in
> hardware or via bios boot.
>
> DOWNLOADS: test.img
> ----------
>      To speed up testing, one can skip reproducing test.img and grab a copy at
> <https://drive.google.com/file/d/0Bx2cO4TT8XjeSEt2aEVkNEVsZUk/view?usp=sharing>
>
> REPRODUCE test.img:
> -------------------
> # fallocate -l 3G test.img
> # gdisk test.img <<<(
> o
> y
> n
> 1
>
> +1M
> ef02
>
> n
> 2
>
> +100M
> ef00
>
> n
> 3
>
>
>
>
> w
> y
> )
> # kpartx -l test.img      /* Get the name of loop device <loop> */
> # kpartx -a test.img
> # mkfs.fat /dev/mapper/<loop>p2
> # mkfs.ext4 /dev/mapper/<loop>p3
> # mount /dev/mapper/<loop>p3 /mnt
> # mkdir /mnt/boot
> # mount /dev/mapper/<loop>p2 /mnt/boot
>
> /* This part is Arch Linux specific */
> # pacstrap /mnt base grub
> /* End Arch Linux specific */
>
> # grub-install --target=i386-pc --boot-directory=/mnt/boot /dev/<loop>
> # grub-install --target=x86_64-efi --efi-directory=/mnt/boot --boot-
> directory=/mnt/boot --removable
> # echo "
> insmod all_video
> menuentry "Arch Linux" {
>    linux /vmlinuz-linux rw root=/dev/sda3
>    initrd /initramfs-linux-fallback.img
> }
> " > /mnt/boot/grub/grub.cfg
> # umount -R /mnt
> # kpartx -d test.img
>
> REPRODUCE QEMU ERROR:
> ---------------------
> /* Bios works */
> # qemu-system-x86_64 -enable-kvm -cpu host test.img
>
> /* UEFI doesn't */
> # qemu-system-x86_64 -enable-kvm -cpu host -bios /usr/share/ovmf/ovmf_x64.bin
> test.img
>
> /* I get the following error */
> Kernel panic - not syncing : VFS: Unable to mount root fs on Unknown-
> block(0,0)
> CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.1.4-1-ARCH #1
> Hardware name : QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> 0000000000000000 000000007fac344c ffff880006527db8 ffffffff815865be
> 0000000000000000 ffffffff816f61b0 ffff880006527e38 ffffffff81585453
> ffff880000000010 ffff880006527e48 ffff880006527de8 000000007fac344c
> Call Trace:
> [<ffffffff815865be>] dump_stack+0x4c/0x6e
> [<ffffffff81585453>] panic+0xd0/0x203
> [<ffffffff818fa621>] mount_block_root+0x297/0x2a6
> [<ffffffff818fa684>] mount_root+0x54/0x58
> [<ffffffff818fa7c0>] prepare_namespace+0x138/0x171
> [<ffffffff818fa22e>] kernel_init_freeable+0x1ec/0x215
> [<ffffffff8157cbb0>] ? rest_init+0x90/0x90
> [<ffffffff8157cbbe>] kernel_init+0xe/0xf0
> [<ffffffff8157c2a2>] ret_fromt_fork+0x42/0x70
> [<ffffffff8157cbb0>] ? rest_init+0x90/0x90
> Kernel Offset: disabled
> ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-
> block(0,0)
>
>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

Re: Kernel panics (in QEMU) when booting with GRUB2 + OVMF + QEMU.

[Mark Lee]

[-- Attachment #1: Type: text/plain, Size: 290 bytes --]

On Saturday, August 15, 2015 09:29:55 AM Andrei Borzenkov wrote:
> Once kernel is loaded, GRUB task is over. Why do you think it is GRUB issue?

This seems to only occur when using GRUB + OVMF + QEMU. Transferring test.img 
to a disk and booting it in hardware UEFI is fine.

Regards,
Mark

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 213 bytes --]

Re: Kernel panics (in QEMU) when booting with GRUB2 + OVMF + QEMU.

[Jordan Uggla]

On Fri, Aug 14, 2015 at 11:29 PM, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
> On 14.08.2015 09:45, Mark Lee wrote:
>>
>> To all,
>>
>> I've encountered a bug when using GRUB2 + OVMF + QEMU on Arch Linux 64
>> bit.
>> The issue can be reproduced when running the latest git code from grub as
>> well.
>>
>> ISSUE: Kernel panics (in QEMU) when booting with GRUB2 + OVMF + QEMU.
>
>
> Once kernel is loaded, GRUB task is over. Why do you think it is GRUB issue?

I worked with Mark in #grub and there were symptoms that Mark has not
described here that lead me to believe that there is either a grub or
OVMF bug, and that is why I asked him to post here.

First Mark demonstrated that the same image, with the same version of
grub and same grub.cfg, was succeeding when booted via BIOS with qemu,
but not when booted via OVMF with qemu, and that the particular error
message before the kernel panic was indicative of the initramfs not
having been loaded. Having largely ruled out misconfiguration I asked
Mark to try running the "linux" and "initrd" commands manually at the
grub shell to see if there were any error messages being displayed.

Mark did this and replied that while the linux command succeeded, the
initrd command failed, and in a way that clearly indicates a bug
somewhere. Specifically, while Mark entered something like "initrd
/boot/initramfs-foo-bar", and everything he had typed was echoed back
to him on screen as expected, the error message he received was (if I
recall correctly) something about the file "/boot/initra" not being
found. In other words, the error message indicated that somewhere the
file name entered had been truncated. I suspect that this happened
when grub was interpreting the grub.cfg as well, and thus the linux
command succeeded but the initramfs command failed, and while grub
still tried to boot, the missing initramfs resulted in the kernel
panic.This, along with other symptoms from previous tries, seems to
indicate to me that there is memory corruption somewhere, either due
to a bug in grub or in OVMF.

I don't have access to the IRC logs of our conversation today (but I
will have access to them Monday) so I'm going by my memory alone.
Hopefully Mark can give the exact error messages he received from
trying to enter the initramfs command directly, or possibly even
reproduce that test and post screenshots or serial output logs of it.
I have not had a chance to try to reproduce this myself, and I don't
know when I will have time to do so.

-- 
Jordan Uggla (Jordan_U on irc.freenode.net)

Re: Kernel panics (in QEMU) when booting with GRUB2 + OVMF + QEMU.

[Mark Lee]

[-- Attachment #1: Type: text/plain, Size: 1292 bytes --]

On Saturday, August 15, 2015 08:31:42 PM Jordan Uggla wrote:
> Specifically, while Mark entered something like "initrd
> /boot/initramfs-foo-bar", and everything he had typed was echoed back
> to him on screen as expected, the error message he received was (if I
> recall correctly) something about the file "/boot/initra" not being
> found. In other words, the error message indicated that somewhere the
> file name entered had been truncated. I suspect that this happened
> when grub was interpreting the grub.cfg as well, and thus the linux
> command succeeded but the initramfs command failed, and while grub
> still tried to boot, the missing initramfs resulted in the kernel
> panic.
That was a personal error. I was typing "/vmlinuz-linux ..." instead of "linux 
/vmlinuz-linux ..." . The same went for the initrd. 

> This, along with other symptoms from previous tries, seems to
> indicate to me that there is memory corruption somewhere, either due
> to a bug in grub or in OVMF.
I have reproduced the error yet again. It is the same error as I listed in the 
first post. Find pictures of the qemu OVMF error : 
https://drive.google.com/file/d/0Bx2cO4TT8XjeMTNLTHJRZzhiSHM/view?usp=sharing
https://drive.google.com/file/d/0Bx2cO4TT8XjeM2dpS1RLQXJMdkE/view?usp=sharing

Regards,
Mark

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 213 bytes --]

Re: Kernel panics (in QEMU) when booting with GRUB2 + OVMF + QEMU.

[Andrei Borzenkov]

On Sun, Aug 16, 2015 at 8:25 AM, Mark Lee <mark@markelee.com> wrote:
> On Saturday, August 15, 2015 08:31:42 PM Jordan Uggla wrote:
>> Specifically, while Mark entered something like "initrd
>> /boot/initramfs-foo-bar", and everything he had typed was echoed back
>> to him on screen as expected, the error message he received was (if I
>> recall correctly) something about the file "/boot/initra" not being
>> found. In other words, the error message indicated that somewhere the
>> file name entered had been truncated. I suspect that this happened
>> when grub was interpreting the grub.cfg as well, and thus the linux
>> command succeeded but the initramfs command failed, and while grub
>> still tried to boot, the missing initramfs resulted in the kernel
>> panic.
> That was a personal error. I was typing "/vmlinuz-linux ..." instead of "linux
> /vmlinuz-linux ..." . The same went for the initrd.
>
>> This, along with other symptoms from previous tries, seems to
>> indicate to me that there is memory corruption somewhere, either due
>> to a bug in grub or in OVMF.
> I have reproduced the error yet again. It is the same error as I listed in the
> first post.

Error itself simply says that kernel cannot find root fs. It does not
say anything about why it happened.

>               Find pictures of the qemu OVMF error :
> https://drive.google.com/file/d/0Bx2cO4TT8XjeMTNLTHJRZzhiSHM/view?usp=sharing

You have incorrect kernel command line. It should be root=/dev/sda3
not root/dev/sda3.

> https://drive.google.com/file/d/0Bx2cO4TT8XjeM2dpS1RLQXJMdkE/view?usp=sharing
>
> Regards,
> Mark
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

Re: Kernel panics (in QEMU) when booting with GRUB2 + OVMF + QEMU.

[Mark Lee]

[-- Attachment #1: Type: text/plain, Size: 718 bytes --]

On Monday, August 17, 2015 05:22:43 PM Andrei Borzenkov wrote:
> Error itself simply says that kernel cannot find root fs. It does not
> say anything about why it happened.
Yes. It boots on hardware (dd if=test.img of=<usb device>). Just  not on qemu 
+ ovmf.

> >               Find pictures of the qemu OVMF error :
> > https://drive.google.com/file/d/0Bx2cO4TT8XjeMTNLTHJRZzhiSHM/view?usp=shar
> > ing
> You have incorrect kernel command line. It should be root=/dev/sda3
> not root/dev/sda3.
> 
I apologize. I've fixed it. Find the errors at :
https://drive.google.com/file/d/0Bx2cO4TT8XjeTnY4a0luclpNUVk/view?usp=sharing
https://drive.google.com/file/d/0Bx2cO4TT8XjeRzFVRGZ1NF9ZZW8/view?usp=sharing

Regards,
Mark

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 213 bytes --]

Re: Kernel panics (in QEMU) when booting with GRUB2 + OVMF + QEMU.

[Vladimir 'phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 1181 bytes --]

Le 17 août 2015 5:26 PM, "Mark Lee" <mark@markelee.com> a écrit :
>
> On Monday, August 17, 2015 05:22:43 PM Andrei Borzenkov wrote:
> > Error itself simply says that kernel cannot find root fs. It does not
> > say anything about why it happened.
> Yes. It boots on hardware (dd if=test.img of=<usb device>). Just  not on
qemu
> + ovmf.
>
> > >               Find pictures of the qemu OVMF error :
> > >
https://drive.google.com/file/d/0Bx2cO4TT8XjeMTNLTHJRZzhiSHM/view?usp=shar
> > > ing
> > You have incorrect kernel command line. It should be root=/dev/sda3
> > not root/dev/sda3.
> >
> I apologize. I've fixed it. Find the errors at :
>
https://drive.google.com/file/d/0Bx2cO4TT8XjeTnY4a0luclpNUVk/view?usp=sharing
>
https://drive.google.com/file/d/0Bx2cO4TT8XjeRzFVRGZ1NF9ZZW8/view?usp=sharing
>
We need longer log to see what happened. I suggest capturing it on serial
port (after adding relevant kernel parameters). My bet is on your initrd
missing drivers for qemu disk
> Regards,
> Mark
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

[-- Attachment #2: Type: text/html, Size: 1862 bytes --]