[Qemu-devel] Hang when using 9p mounts after last Seabios update

[Qemu-devel] Hang when using 9p mounts after last Seabios update

[Daniel P. Berrange]

I'm doing some experimentation with application sandboxing and running
QEMU with the following command line:

  exec $QEMU \
    -m 64 \
    -nographic \
    -nodefconfig \
    -nodefaults \
    -kernel ./kernel-2.6.40.4-5.fc15.x86_64 \
    -initrd ./initrd-2.6.40.4-5.fc15.x86_64 \
    -append 'init=/bin/sh console=ttyS0 debug' \
    -fsdev local,security_model=passthrough,id=fsdev-fs0,path=/ \
    -device virtio-9p-pci,id=fs0,fsdev=fsdev-fs0,mount_tag=org.virttools.sandbox:/ \
    -serial stdio

On 0.14, 0.15 releaes, this all works just fine. On current GIT master,
the guest OS will hang during boot.

The hang always appears to occur when attempting to mount the *last*
virtio9p filesystem exported to the guest.

ie, in the above command line it hangs when mounting the root. If I
passthrough 2 more directories, eg 

    -fsdev local,security_model=passthrough,id=fsdev-fs0,path=/custom-home \
    -device virtio-9p-pci,id=fs0,fsdev=fsdev-fs0,mount_tag=org.virttools.sandbox:/home \
    -fsdev local,security_model=passthrough,id=fsdev-fs0,path=/custom-tmp \
    -device virtio-9p-pci,id=fs0,fsdev=fsdev-fs0,mount_tag=org.virttools.sandbox:/tmp \

then it will hang on the 3rd mount


[    0.798208] virtio-pci 0000:00:05.0: irq 45 for MSI/MSI-X
virt-sandbox-init: insmod /lib/modules/kernel/drivers/net/virtio_net.ko
virt-sandbox-init: insmod /lib/modules/kernel/drivers/virtio/virtio_balloon.ko
virt-sandbox-init: mounting new root on /tmproot
virt-sandbox-init: org.virttools.sandbox:/ -> /tmproot
virt-sandbox-init: chroot
virt-sandbox-init: /dev (tmpfs)
virt-sandbox-init: /dev/pts (devpts)
virt-sandbox-init: /root (tmpfs)
virt-sandbox-init: /sys (sysfs)
virt-sandbox-init: /proc (proc)
virt-sandbox-init: /selinux (selinuxfs)
virt-sandbox-init: org.virttools.sandbox:/home -> /home
virt-sandbox-init: org.virttools.sandbox:/tmp -> /tmp
[    1.036163] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2
..hang...


Now the really wierd thing.... if I add certain extra PCI device onto
the QEMU command line after all of the 9p filesystem devices, then
the hang goes away. Specifically adding '-vga std' or '-device e1000'
stops it hanging.  Adding '-device virtio-balloon' or '-device i6300esb'
does not prevent the hang.


I bisected the problem in QEMU master to this change which pulls in a
new SeaBios:

  commit d67c3f2cd92aed2247bfa8a9da61a902b7b2ff09
  Author: Gerd Hoffmann <kraxel@redhat.com>
  Date:   Wed Aug 10 17:34:13 2011 +0200

    seabios: update to master
    
    commit 8e301472e324b6d6496d8b4ffc66863e99d7a505
    
    user visible changes in seabios:
      * ahci is enabled by default (and thus in this build).
      * bootorder support for ahci.
      * two-pass pci allocator (orders bars by size for better packing).
    
    Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>


I further bisected SeaBios between their 0.6.2 release and the changeset
mentioned above, and got to this changeset in SeaBios:


  commit 01a5c8813b2e709809c07c5d7fab9d1c3ddb4989
  Author: Gerd Hoffmann <kraxel@redhat.com>
  Date:   Mon Jul 11 09:20:29 2011 +0200

    pci: activate two-pass pci initialization code
    
    This patch actually enables the two-pass pci initialization and
    deactivates the old pci initialization bits.
    
    Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>


Before this changeset everything works fine, after it, i get a hang on
9p mounts.


To reproduce this you will need my custom initrd for mounting 9p filesystems
as the root FS. You can get that here:

  http://berrange.com/~dan/qemu-serial-hang-demo.tar.gz

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] Hang when using 9p mounts after last Seabios update

[Kevin O'Connor]

On Thu, Sep 22, 2011 at 12:45:11PM +0100, Daniel P. Berrange wrote:
> I'm doing some experimentation with application sandboxing and running
> QEMU with the following command line:
> 
>   exec $QEMU \
>     -m 64 \
>     -nographic \
>     -nodefconfig \
>     -nodefaults \
>     -kernel ./kernel-2.6.40.4-5.fc15.x86_64 \
>     -initrd ./initrd-2.6.40.4-5.fc15.x86_64 \
>     -append 'init=/bin/sh console=ttyS0 debug' \
>     -fsdev local,security_model=passthrough,id=fsdev-fs0,path=/ \
>     -device virtio-9p-pci,id=fs0,fsdev=fsdev-fs0,mount_tag=org.virttools.sandbox:/ \
>     -serial stdio
> 
> On 0.14, 0.15 releaes, this all works just fine. On current GIT master,
> the guest OS will hang during boot.
[...]

> I further bisected SeaBios between their 0.6.2 release and the changeset
> mentioned above, and got to this changeset in SeaBios:
> 
> 
>   commit 01a5c8813b2e709809c07c5d7fab9d1c3ddb4989
>   Author: Gerd Hoffmann <kraxel@redhat.com>
>   Date:   Mon Jul 11 09:20:29 2011 +0200

Gerd, any thoughts on what could cause this?

-Kevin

Re: [Qemu-devel] Hang when using 9p mounts after last Seabios update

[Kevin O'Connor]

On Thu, Sep 22, 2011 at 12:45:11PM +0100, Daniel P. Berrange wrote:
> On 0.14, 0.15 releaes, this all works just fine. On current GIT master,
> the guest OS will hang during boot.
[...]
> To reproduce this you will need my custom initrd for mounting 9p filesystems
> as the root FS. You can get that here:
> 
>   http://berrange.com/~dan/qemu-serial-hang-demo.tar.gz

Thanks for the detailed report.

I've confirmed the issue and tracked it down.  The current SeaBIOS
code gets confused during alignment checking if there are no prefmem
regions found.

The patch below should fix the issue.

Richard, can you also check to see if this seabios patch fixes your
issue?

-Kevin


Author: Kevin O'Connor <kevin@koconnor.net>
Date:   Sat Oct 1 12:35:32 2011 -0400

    Fix alignment bug in pci_bios_init_root_regions().
    
    If there are no memory allocations for a given type then the "max" bar
    size is zero.  However, ALIGN_DOWN does not handle an alignment of
    zero properly.  Catch and handle the zero case.
    
    Signed-off-by: Kevin O'Connor <kevin@koconnor.net>

diff --git a/src/pciinit.c b/src/pciinit.c
index a857da0..0d8758e 100644
--- a/src/pciinit.c
+++ b/src/pciinit.c
@@ -536,7 +536,7 @@ static void pci_bios_init_bus_bases(struct pci_bus *bus)
     }
 }
 
-#define ROOT_BASE(top, sum, align) ALIGN_DOWN((top)-(sum),(align))
+#define ROOT_BASE(top, sum, max) ALIGN_DOWN((top)-(sum),(max) ?: 1)
 
 static int pci_bios_init_root_regions(u32 start, u32 end)
 {

Re: [Qemu-devel] Hang when using 9p mounts after last Seabios update

[Richard W.M. Jones]

On Sat, Oct 01, 2011 at 12:50:43PM -0400, Kevin O'Connor wrote:
> On Thu, Sep 22, 2011 at 12:45:11PM +0100, Daniel P. Berrange wrote:
> > On 0.14, 0.15 releaes, this all works just fine. On current GIT master,
> > the guest OS will hang during boot.
> [...]
> > To reproduce this you will need my custom initrd for mounting 9p filesystems
> > as the root FS. You can get that here:
> > 
> >   http://berrange.com/~dan/qemu-serial-hang-demo.tar.gz
> 
> Thanks for the detailed report.
> 
> I've confirmed the issue and tracked it down.  The current SeaBIOS
> code gets confused during alignment checking if there are no prefmem
> regions found.
> 
> The patch below should fix the issue.
> 
> Richard, can you also check to see if this seabios patch fixes your
> issue?

Yes, I can confirm that this fixes the issue.

What I did to test this:

 - tried my boot test, and it failed as before

 - git clone git://git.linuxtogo.org/home/kevin/seabios.git

 - noticed that this patch is already applied to git

 - make clean; make

 - cp out/bios.bin ../qemu/pc-bios/bios.bin

 - repeated my boot test, and it was successful

Thanks!

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v

Re: [Qemu-devel] Hang when using 9p mounts after last Seabios update

[Daniel P. Berrange]

On Sat, Oct 01, 2011 at 12:50:43PM -0400, Kevin O'Connor wrote:
> On Thu, Sep 22, 2011 at 12:45:11PM +0100, Daniel P. Berrange wrote:
> > On 0.14, 0.15 releaes, this all works just fine. On current GIT master,
> > the guest OS will hang during boot.
> [...]
> > To reproduce this you will need my custom initrd for mounting 9p filesystems
> > as the root FS. You can get that here:
> > 
> >   http://berrange.com/~dan/qemu-serial-hang-demo.tar.gz
> 
> Thanks for the detailed report.
> 
> I've confirmed the issue and tracked it down.  The current SeaBIOS
> code gets confused during alignment checking if there are no prefmem
> regions found.
> 
> The patch below should fix the issue.

Thanks, I have tested Seabios 1.6.3 which includes that patch, and
can confirm that it does fix the hang I saw.

Anthony/Gerd: we can get QEMU master updated to Seabios 1.6.3 before
the 1.0 release ?

> Author: Kevin O'Connor <kevin@koconnor.net>
> Date:   Sat Oct 1 12:35:32 2011 -0400
> 
>     Fix alignment bug in pci_bios_init_root_regions().
>     
>     If there are no memory allocations for a given type then the "max" bar
>     size is zero.  However, ALIGN_DOWN does not handle an alignment of
>     zero properly.  Catch and handle the zero case.
>     
>     Signed-off-by: Kevin O'Connor <kevin@koconnor.net>
> 
> diff --git a/src/pciinit.c b/src/pciinit.c
> index a857da0..0d8758e 100644
> --- a/src/pciinit.c
> +++ b/src/pciinit.c
> @@ -536,7 +536,7 @@ static void pci_bios_init_bus_bases(struct pci_bus *bus)
>      }
>  }
>  
> -#define ROOT_BASE(top, sum, align) ALIGN_DOWN((top)-(sum),(align))
> +#define ROOT_BASE(top, sum, max) ALIGN_DOWN((top)-(sum),(max) ?: 1)
>  
>  static int pci_bios_init_root_regions(u32 start, u32 end)
>  {

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] Hang when using 9p mounts after last Seabios update

[Chris Krumme]

On 10/11/2011 03:30 AM, Daniel P. Berrange wrote:
> On Sat, Oct 01, 2011 at 12:50:43PM -0400, Kevin O'Connor wrote:
>> On Thu, Sep 22, 2011 at 12:45:11PM +0100, Daniel P. Berrange wrote:
>>> On 0.14, 0.15 releaes, this all works just fine. On current GIT master,
>>> the guest OS will hang during boot.
>> [...]
>>> To reproduce this you will need my custom initrd for mounting 9p filesystems
>>> as the root FS. You can get that here:
>>>
>>>    http://berrange.com/~dan/qemu-serial-hang-demo.tar.gz
>> Thanks for the detailed report.
>>
>> I've confirmed the issue and tracked it down.  The current SeaBIOS
>> code gets confused during alignment checking if there are no prefmem
>> regions found.
>>
>> The patch below should fix the issue.
> Thanks, I have tested Seabios 1.6.3 which includes that patch, and
> can confirm that it does fix the hang I saw.
>
> Anthony/Gerd: we can get QEMU master updated to Seabios 1.6.3 before
> the 1.0 release ?
>
>> Author: Kevin O'Connor<kevin@koconnor.net>
>> Date:   Sat Oct 1 12:35:32 2011 -0400
>>
>>      Fix alignment bug in pci_bios_init_root_regions().
>>
>>      If there are no memory allocations for a given type then the "max" bar
>>      size is zero.  However, ALIGN_DOWN does not handle an alignment of
>>      zero properly.  Catch and handle the zero case.
>>
>>      Signed-off-by: Kevin O'Connor<kevin@koconnor.net>
>>
>> diff --git a/src/pciinit.c b/src/pciinit.c
>> index a857da0..0d8758e 100644
>> --- a/src/pciinit.c
>> +++ b/src/pciinit.c
>> @@ -536,7 +536,7 @@ static void pci_bios_init_bus_bases(struct pci_bus *bus)
>>       }
>>   }
>>
>> -#define ROOT_BASE(top, sum, align) ALIGN_DOWN((top)-(sum),(align))
>> +#define ROOT_BASE(top, sum, max) ALIGN_DOWN((top)-(sum),(max) ?: 1)

Hello,

This is adding a GNUism, can this be:  (max)?(max):1

Thanks

Chris

>>
>>   static int pci_bios_init_root_regions(u32 start, u32 end)
>>   {
> Regards,
> Daniel