Re: KVM git hangs with if=virtio (works under kvm 0.12.3)

[Stefan Hajnoczi <stefanha@gmail.com>]

2010/7/7 Gleb Natapov <gleb@redhat.com>:
> On Wed, Jul 07, 2010 at 10:03:31AM +0100, Stefan Hajnoczi wrote:
>> 2010/7/5 Gleb Natapov <gleb@redhat.com>:
>> > On Mon, Jul 05, 2010 at 01:36:08PM +0100, Stefan Hajnoczi wrote:
>> >> 2010/7/5 Gleb Natapov <gleb@redhat.com>:
>> >> > On Mon, Jul 05, 2010 at 01:11:25PM +0100, Stefan Hajnoczi wrote:
>> >> >> On Fri, Jul 2, 2010 at 9:07 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> >> >> > On Fri, Jul 2, 2010 at 4:31 AM, ewheeler <kvm@ew.ewheeler.org> wrote:
>> >> >> >> Hello all,
>> >> >> >>
>> >> >> >> I'm booting a CentOS kernel under today's KVM git and it hangs after
>> >> >> >> initializing the serial port when the drive if=virtio, but not when
>> >> >> >> drive if=ide.  Look close---this is not a "forgot to add virtio_blk"
>> >> >> >> problem.  If I use 0.12.3 from Ubuntu 10.04 it works properly.
>> >> >> >>
>> >> >> >> Reproduction:
>> >> >> >>
>> >> >> >> Using kvm 0.12.3 on ubuntu 10.04 (1:84+dfsg-0ubuntu16+0.12.3+noroms
>> >> >> >> +0ubuntu9) it will work properly:
>> >> >> >>
>> >> >> >>  qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
>> >> >> >>                        -kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus
>> >> >> >>
>> >> >> >> As expected, the kernel panics unable to mount root (good-boot.png).
>> >> >> >> This makes sense, as "dummy-disk-image" is 1MB of 0x00 bytes.
>> >> >> >>
>> >> >> >> ---However---if I use today's git (2010-07-01) of kvm:
>> >> >> >>
>> >> >> >>   /usr/local/kvm-git/bin/qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
>> >> >> >>        -kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus
>> >> >> >>
>> >> >> >> This hangs just after initializing the Serial device (obtained by adding
>> >> >> >> -serial stdio -append console=ttyS0):
>> >> >> >>
>> >> >> >> Note that this only happens with the disk interface set to virtio
>> >> >> >> (if=virtio).  It works fine for ide (if=ide).
>> >> >> >>
>> >> >> >>
>> >> >> >> Am I doing something wrong here?
>> >> >> >> Is anyone else having this problem?
>> >> >> >
>> >> >> > I have seen this issue with a RHEL 5.5 guest running under
>> >> >> > qemu-kvm.git.  It boots a new guest fine but hangs as you described
>> >> >> > with the RHEL 5.5 kernel.  I have not investigated.
>> >> >>
>> >> >> This issue is affected by extboot, a feature that enables booting from
>> >> >> virtio-blk devices.  I have just sent a patch to the KVM mailing list
>> >> >> to restore extboot functionality which has been broken in
>> >> >> qemu-kvm.git.  That patch can be used to work around this issue by
>> >> >> using "-drive ...,boot=on" but it doesn't explain why the RHEL 5.5
>> >> >> kernel hangs during serial initialization when extboot is not present.
>> >> >>
>> >> > Hang that happens during guest boot (after bootloader started the
>> >> > kernel) cannot be worked around by extboot. extboot is also not needed
>> >> > with latest qemu git to boot from virtio disks since the support for
>> >> > that is in the bios now.
>> >>
>> >> I agree that something else is going on here and needs to be
>> >> investigated, but I do think that extboot can indirectly affect the
>> >> guest boot.
>> >>
>> >> With extboot the virtio-blk PCI adapter is not touched by the
>> >> firmware/bootloader.  Is it possible that a virtio-blk interrupt is
>> >> raised and not acknowledged before entering Linux.  When Linux brings
>> >> up the serial port it gets swamped with interrupts?  That's just a
>> >> guess.
>> >>
>> > That is possible when bios is actually used to boot the guest, but bug
>> > reporter uses -kernel option so no bios boot code should run at all.
>> > Virtio is initialized anyway, but this will happen with boot=on too.
>>
>> Okay, I got to the bottom of this.  Here's the story, see bottom of the mail for
>> the solution and workarounds:
>>
> Great. Thanks Stefan. I was sure that fix you found is in qemu.git
> already. It is good to know that Linux actually uses int13 once. Didn't
> know that. To be absolutely sure no interrupt will stuck we should probably
> clear interrupt after each virtio read in bios by reading status register.

Yes, this is the approach I took when rewriting gPXE's virtio-net
driver recently:

http://git.etherboot.org/?p=people/stefanha/gpxe.git;a=blob;f=src/drivers/net/virtio-net.c;h=50c580049c3663f0d1332d51e30c8a59b5663b68;hb=d58bf6fc8db4a55075e370c7e1d436a570400f55#l299

Are you spinning a patch or should I submit one?

>> It turns out that -kernel does involve the BIOS.  KVM pulls apart the bzImage
>> and makes it available via the fw_cfg interface.  The linuxboot.bin option ROM
>> is executed by the BIOS inside the VM to actually jump into the kernel.  This
>> means the BIOS does POST and sets itself up; the kernel's real-mode boot
>> code is going to use BIOS interrupts.
>>
>> So now the VM has booted, BIOS finished POST and executed linuxboot.bin,
>> linuxboot.bin transferred control to Linux.  Then, in Linux arch/x86/boot/edd.c
>> a disk read to sector=0 bytes=512 is made using INT 13h.  Since this disk
>> read comes from the Linux kernel, it happens regardless of -kernel or not.
>>
>> The disk read is serviced by the BIOS.  Older versions of SeaBIOS leave the
>> interrupt raised here, so then the kernel hangs in serial initialization later.
>>
>> However, there is a simple workaround:
>>
>>   x86_64-softmmu/qemu-system-x86_64 -m 512 -drive
>> file=~/rhel5u5.img,if=virtio -kernel /boot/vmlinuz-2.6.32-5-amd64
>> -append edd=skipmbr
>>
>> When the edd=skipmbr kernel parameter is used, the kernel will not perform
>> the disk read and the interrupt will never get stuck.  The VM boots
>> successfully.
>>
>> The edd=skipmbr workaround is not enough when booting from disk and not
>> -kernel.  The BIOS and bootloader will invoke INT 13h and the only way
>> around that is to use extboot.bin with boot=on as I originally described.
>>
>> The root cause was fixed in SeaBIOS commit:
>>
>>   commit 4030db0d2c5a79de2a1b5c31514503e4ff2a3cd1
>>   Author: Gleb Natapov <gleb@redhat.com>
>>   Date:   Mon May 17 16:27:27 2010 +0300
>>
>>       fix two issues with virtio-blk
>>
>>       1. Check if blk_size is valid in virtio_blk config.
>>       2. Disable interrupt otherwise interrupt may stuck
>>          with some guests.
>>
>> If you build SeaBIOS from source and launch KVM with -bios
>> path/to/seabios/out/bios.bin, RHEL5.5 will boot without hanging at serial
>> initialization time.
>>
>> QEMU and KVM need to grab a later SeaBIOS build that contains 4030db0.
>>
>> Stefan
>>
>> PS: If you test this on qemu.git, you may find that the kernel doesn't hang
>> at serial initialization, despite the bug being present in the BIOS.  I'm not
>> sure how QEMU gets away with it but I wonder if the interrupt configuration
>> is different.
>
> --
>                        Gleb.
>

Stefan