From mboxrd@z Thu Jan 1 00:00:00 1970 From: Gleb Natapov Subject: Re: KVM git hangs with if=virtio (works under kvm 0.12.3) Date: Wed, 7 Jul 2010 12:13:58 +0300 Message-ID: <20100707091358.GY4689@redhat.com> References: <1278041518.10459.48.camel@geektop> <20100705121759.GP4689@redhat.com> <20100705124112.GR4689@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: ewheeler , kvm@vger.kernel.org, Anthony Liguori , Avi Kivity To: Stefan Hajnoczi Return-path: Received: from mx1.redhat.com ([209.132.183.28]:54589 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751746Ab0GGJOE convert rfc822-to-8bit (ORCPT ); Wed, 7 Jul 2010 05:14:04 -0400 Content-Disposition: inline In-Reply-To: Sender: kvm-owner@vger.kernel.org List-ID: On Wed, Jul 07, 2010 at 10:03:31AM +0100, Stefan Hajnoczi wrote: > 2010/7/5 Gleb Natapov : > > On Mon, Jul 05, 2010 at 01:36:08PM +0100, Stefan Hajnoczi wrote: > >> 2010/7/5 Gleb Natapov : > >> > On Mon, Jul 05, 2010 at 01:11:25PM +0100, Stefan Hajnoczi wrote: > >> >> On Fri, Jul 2, 2010 at 9:07 AM, Stefan Hajnoczi wrote: > >> >> > On Fri, Jul 2, 2010 at 4:31 AM, ewheeler wrote: > >> >> >> Hello all, > >> >> >> > >> >> >> I'm booting a CentOS kernel under today's KVM git and it han= gs after > >> >> >> initializing the serial port when the drive if=3Dvirtio, but= not when > >> >> >> drive if=3Dide. =9ALook close---this is not a "forgot to add= virtio_blk" > >> >> >> problem. =9AIf I use 0.12.3 from Ubuntu 10.04 it works prope= rly. > >> >> >> > >> >> >> Reproduction: > >> >> >> > >> >> >> Using kvm 0.12.3 on ubuntu 10.04 (1:84+dfsg-0ubuntu16+0.12.3= +noroms > >> >> >> +0ubuntu9) it will work properly: > >> >> >> > >> >> >> =9Aqemu-system-x86_64 -drive file=3Ddummy-disk-image,if=3Dvi= rtio \ > >> >> >> =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A-kernel vmlin= uz-2.6.18-194.3.1.el5.centos.plus > >> >> >> > >> >> >> As expected, the kernel panics unable to mount root (good-bo= ot.png). > >> >> >> This makes sense, as "dummy-disk-image" is 1MB of 0x00 bytes= =2E > >> >> >> > >> >> >> ---However---if I use today's git (2010-07-01) of kvm: > >> >> >> > >> >> >> =9A /usr/local/kvm-git/bin/qemu-system-x86_64 -drive file=3D= dummy-disk-image,if=3Dvirtio \ > >> >> >> =9A =9A =9A =9A-kernel vmlinuz-2.6.18-194.3.1.el5.centos.plu= s > >> >> >> > >> >> >> This hangs just after initializing the Serial device (obtain= ed by adding > >> >> >> -serial stdio -append console=3DttyS0): > >> >> >> > >> >> >> Note that this only happens with the disk interface set to v= irtio > >> >> >> (if=3Dvirtio). =9AIt works fine for ide (if=3Dide). > >> >> >> > >> >> >> > >> >> >> Am I doing something wrong here? > >> >> >> Is anyone else having this problem? > >> >> > > >> >> > I have seen this issue with a RHEL 5.5 guest running under > >> >> > qemu-kvm.git. =9AIt boots a new guest fine but hangs as you d= escribed > >> >> > with the RHEL 5.5 kernel. =9AI have not investigated. > >> >> > >> >> This issue is affected by extboot, a feature that enables booti= ng from > >> >> virtio-blk devices. =9AI have just sent a patch to the KVM mail= ing list > >> >> to restore extboot functionality which has been broken in > >> >> qemu-kvm.git. =9AThat patch can be used to work around this iss= ue by > >> >> using "-drive ...,boot=3Don" but it doesn't explain why the RHE= L 5.5 > >> >> kernel hangs during serial initialization when extboot is not p= resent. > >> >> > >> > Hang that happens during guest boot (after bootloader started th= e > >> > kernel) cannot be worked around by extboot. extboot is also not = needed > >> > with latest qemu git to boot from virtio disks since the support= for > >> > that is in the bios now. > >> > >> I agree that something else is going on here and needs to be > >> investigated, but I do think that extboot can indirectly affect th= e > >> guest boot. > >> > >> With extboot the virtio-blk PCI adapter is not touched by the > >> firmware/bootloader. =9AIs it possible that a virtio-blk interrupt= is > >> raised and not acknowledged before entering Linux. =9AWhen Linux b= rings > >> up the serial port it gets swamped with interrupts? =9AThat's just= a > >> guess. > >> > > That is possible when bios is actually used to boot the guest, but = bug > > reporter uses -kernel option so no bios boot code should run at all= =2E > > Virtio is initialized anyway, but this will happen with boot=3Don t= oo. >=20 > Okay, I got to the bottom of this. Here's the story, see bottom of t= he mail for > the solution and workarounds: >=20 Great. Thanks Stefan. I was sure that fix you found is in qemu.git already. It is good to know that Linux actually uses int13 once. Didn't know that. To be absolutely sure no interrupt will stuck we should prob= ably clear interrupt after each virtio read in bios by reading status regist= er. > It turns out that -kernel does involve the BIOS. KVM pulls apart the= bzImage > and makes it available via the fw_cfg interface. The linuxboot.bin o= ption ROM > is executed by the BIOS inside the VM to actually jump into the kerne= l. This > means the BIOS does POST and sets itself up; the kernel's real-mode b= oot > code is going to use BIOS interrupts. >=20 > So now the VM has booted, BIOS finished POST and executed linuxboot.b= in, > linuxboot.bin transferred control to Linux. Then, in Linux arch/x86/= boot/edd.c > a disk read to sector=3D0 bytes=3D512 is made using INT 13h. Since t= his disk > read comes from the Linux kernel, it happens regardless of -kernel or= not. >=20 > The disk read is serviced by the BIOS. Older versions of SeaBIOS lea= ve the > interrupt raised here, so then the kernel hangs in serial initializat= ion later. >=20 > However, there is a simple workaround: >=20 > x86_64-softmmu/qemu-system-x86_64 -m 512 -drive > file=3D~/rhel5u5.img,if=3Dvirtio -kernel /boot/vmlinuz-2.6.32-5-amd64 > -append edd=3Dskipmbr >=20 > When the edd=3Dskipmbr kernel parameter is used, the kernel will not = perform > the disk read and the interrupt will never get stuck. The VM boots > successfully. >=20 > The edd=3Dskipmbr workaround is not enough when booting from disk and= not > -kernel. The BIOS and bootloader will invoke INT 13h and the only wa= y > around that is to use extboot.bin with boot=3Don as I originally desc= ribed. >=20 > The root cause was fixed in SeaBIOS commit: >=20 > commit 4030db0d2c5a79de2a1b5c31514503e4ff2a3cd1 > Author: Gleb Natapov > Date: Mon May 17 16:27:27 2010 +0300 >=20 > fix two issues with virtio-blk >=20 > 1. Check if blk_size is valid in virtio_blk config. > 2. Disable interrupt otherwise interrupt may stuck > with some guests. >=20 > If you build SeaBIOS from source and launch KVM with -bios > path/to/seabios/out/bios.bin, RHEL5.5 will boot without hanging at se= rial > initialization time. >=20 > QEMU and KVM need to grab a later SeaBIOS build that contains 4030db0= =2E >=20 > Stefan >=20 > PS: If you test this on qemu.git, you may find that the kernel doesn'= t hang > at serial initialization, despite the bug being present in the BIOS. = I'm not > sure how QEMU gets away with it but I wonder if the interrupt configu= ration > is different. -- Gleb.