hang after seabios

hang after seabios

[Zetan Drableg]

Hi,
Locally I have a supermicro server running OEL 6.5 with KVM
 can do virt-sysprep and libguestfs-test-tool no problem.

Linux 2.6.39-400.215.6.el6uek.x86_64
qemu-kvm-0.12.1.2-2.415.el6_5.10.x86_64
seabios-0.6.1.2-28.el6.x86_64

However I have a server in a datacenter (Sun X4-2) running the same
versions, and libguestfs-test-tool hangs when launching KVM.

virt-sysprep also hangs the same way when trying to access a disk image, so
I'm using libguestfs-test-tool as my example:


   [root@kvm]# libguestfs-test-tool
        ************************************************************
        *                    IMPORTANT NOTICE
        *
        * When reporting bugs, include the COMPLETE, UNEDITED
        * output below in your bug report.
        *
        ************************************************************
   LIBGUESTFS_APPEND=edd=off
   PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr
   /sbin:/usr/bin:/root/bin
   SELinux: Enforcing
   library version: 1.20.11rhel=6,release=2.el6
   guestfs_get_append: edd=off
   guestfs_get_attach_method: appliance
   guestfs_get_autosync: 1
   guestfs_get_cachedir: /var/tmp
   guestfs_get_direct: 0
   guestfs_get_memsize: 500
   guestfs_get_network: 0
   guestfs_get_path: /usr/lib64/guestfs
   guestfs_get_pgroup: 0
   guestfs_get_qemu: /usr/libexec/qemu-kvm
   guestfs_get_recovery_proc: 1
   guestfs_get_selinux: 0
   guestfs_get_smp: 1
   guestfs_get_tmpdir: /tmp
   guestfs_get_trace: 0
   guestfs_get_verbose: 1
   host_cpu: x86_64
   Launching appliance, timeout set to 600 seconds.
   libguestfs: launch: attach-method=appliance
   libguestfs: launch: tmpdir=/tmp/libguestfspx9994
   libguestfs: launch: umask=0077
   libguestfs: launch: euid=0
   libguestfs: command: run: febootstrap-supermin-helper
   libguestfs: command: run: \ --verbose
   libguestfs: command: run: \ -f checksum
   libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
   libguestfs: command: run: \ x86_64
   supermin helper [00000ms] whitelist = (not specified), host_cpu =
   x86_64, kernel = (null), initrd = (null), appliance = (null)
   supermin helper [00000ms] inputs[0] = /usr/lib64/guestfs/supermin.d
   checking modpath /lib/modules/2.6.32-279.el6.x86_64 is a directory
   picked vmlinuz-2.6.32-279.el6.x86_64 because modpath /lib/modules/2.6.32-
   279.el6.x86_64 exists
   checking modpath /lib/modules/2.6.39-200.24.1.el6uek.x86_64 is a
   directory
   picked vmlinuz-2.6.39-200.24.1.el6uek.x86_64 because modpath /lib/modules
   /2.6.39-200.24.1.el6uek.x86_64 exists
   supermin helper [00000ms] finished creating kernel
   supermin helper [00000ms] visiting /usr/lib64/guestfs/supermin.d
   supermin helper [00000ms] visiting /usr/lib64/guestfs/supermin.d/base.img
   supermin helper [00000ms] visiting /usr/lib64/guestfs/supermin.d/
   daemon.img
   supermin helper [00000ms] visiting /usr/lib64/guestfs/supermin.d/
   hostfiles
   supermin helper [00020ms] visiting /usr/lib64/guestfs/supermin.d/init.img
   supermin helper [00020ms] visiting /usr/lib64/guestfs/supermin.d/
   udev-rules.img
   supermin helper [00020ms] adding kernel modules
   supermin helper [00051ms] finished creating appliance
   libguestfs: checksum of existing appliance:
   4805d2b09b84366bd753e62706693476b59c3971f4c1808739426b92f8baa3bf
   libguestfs: [00054ms] begin testing qemu features
   libguestfs: command: run: /usr/libexec/qemu-kvm
   libguestfs: command: run: \ -nographic
   libguestfs: command: run: \ -help
   libguestfs: command: run: /usr/libexec/qemu-kvm
   libguestfs: command: run: \ -nographic
   libguestfs: command: run: \ -version
   libguestfs: qemu version 0.12
   libguestfs: command: run: /usr/libexec/qemu-kvm
   libguestfs: command: run: \ -nographic
   libguestfs: command: run: \ -machine accel=kvm:tcg
   libguestfs: command: run: \ -device ?
   libguestfs: [00182ms] finished testing qemu features
   libguestfs: accept_from_daemon: 0x2266e00 g->state = 1
   [00183ms] /usr/libexec/qemu-kvm \
       -global virtio-blk-pci.scsi=off \
       -nodefconfig \
       -nodefaults \
       -nographic \
       -machine accel=kvm:tcg \
       -cpu host,+kvmclock \
       -m 500 \
       -no-reboot \
       -kernel /var/tmp/.guestfs-0/kernel.47903 \
       -initrd /var/tmp/.guestfs-0/initrd.47903 \
       -device virtio-scsi-pci,id=scsi \
       -drive file=/tmp/libguestfs-test-tool-sda-Iakpwe,cache=none,format
   =raw,id=hd0,if=none \
       -device scsi-hd,drive=hd0 \
       -drive file=/var/tmp/.guestfs-0/root.47903,snapshot=on,id=appliance,
   if=none,cache=unsafe \
       -device scsi-hd,drive=appliance \
       -device virtio-serial \
       -serial stdio \
       -device sga \
       -chardev socket,path=/tmp/libguestfspx9994/guestfsd.sock,id=channel0
   \
       -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0
   \
       -append 'panic=1 console=ttyS0 udevtimeout=600 no_timer_check
   acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0
   guestfs_verbose=1 TERM=xterm edd=off'
   \x1b[1;256r\x1b[256;256H\x1b[6n
   Google, Inc.
   Serial Graphics Adapter 10/14/11
   SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $ (mockbuild@
   ca-build44.us.oracle.com) Fri Oct 14 20:04:36 UTC 2011
   Term: 80x24
   4 0
   SeaBIOS (version seabios-0.6.1.2-28.el6)
   \x1b[2J

At this point it hangs forever.
When I strace virt-sysprep it hangs in the same place.

read(7, "\r\nGoogle, Inc.\r\nSerial Graphics "..., 8192) = 177
select(8, [5 7], NULL, NULL, NULL)      = 1 (in [7])
nanosleep({0, 1000000}, NULL)           = 0
read(7, ": 80x24\r\n4 0\r\n\33[2J\r*SeaBIOS *(vers"..., 8192) = 62
select(8, [5 7], NULL, NULL, NULL)      = 1 (in [7])
nanosleep({0, 1000000}, NULL)           = 0
read(7, "\r*Probing EDD (edd=off to disable*"..., 8192) = 37
select(8, [5 7], NULL, NULL, NULL)      = 1 (in [7])
nanosleep({0, 1000000}, NULL)           = 0
read(7, "ok\r\r\n\33[2J", 8192)         = 9
select(8, [5 7], NULL, NULL, NULL

I updates seabios to 1.7.5 and saw no change.
No SELinux deny audit records.

Can someone help me figure out where the problem lies?

Thanks
-Zetan503
04ff711131--

Re: hang after seabios

[Richard W.M. Jones]

On Wed, Jul 30, 2014 at 03:58:43PM -0700, Zetan Drableg wrote:
>    [00183ms] /usr/libexec/qemu-kvm \
>        -global virtio-blk-pci.scsi=off \
>        -nodefconfig \
>        -nodefaults \
>        -nographic \
>        -machine accel=kvm:tcg \
>        -cpu host,+kvmclock \
>        -m 500 \
>        -no-reboot \
>        -kernel /var/tmp/.guestfs-0/kernel.47903 \
>        -initrd /var/tmp/.guestfs-0/initrd.47903 \
>        -device virtio-scsi-pci,id=scsi \
>        -drive file=/tmp/libguestfs-test-tool-sda-Iakpwe,cache=none,format
>    =raw,id=hd0,if=none \
>        -device scsi-hd,drive=hd0 \
>        -drive file=/var/tmp/.guestfs-0/root.47903,snapshot=on,id=appliance,
>    if=none,cache=unsafe \
>        -device scsi-hd,drive=appliance \
>        -device virtio-serial \
>        -serial stdio \
>        -device sga \
>        -chardev socket,path=/tmp/libguestfspx9994/guestfsd.sock,id=channel0
>    \
>        -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0
>    \
>        -append 'panic=1 console=ttyS0 udevtimeout=600 no_timer_check
>    acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0
>    guestfs_verbose=1 TERM=xterm edd=off'
>    \x1b[1;256r\x1b[256;256H\x1b[6n
>    Google, Inc.
>    Serial Graphics Adapter 10/14/11
>    SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $ (mockbuild@
>    ca-build44.us.oracle.com) Fri Oct 14 20:04:36 UTC 2011
>    Term: 80x24
>    4 0
>    SeaBIOS (version seabios-0.6.1.2-28.el6)
>    \x1b[2J
> 
> At this point it hangs forever.

These hangs can be tricky to diagnose.

There are a couple of things you can do however:

(1) strace qemu to find out what it is doing, or:

(2) [harder, but much more informative] gdb into the guest to find out
where the guest hangs, or:

(3) Take the command line above, and cut it down to try to isolate the
problematic options.  I would concentrate on the following options as
being most likely to cause trouble:

 -cpu
 -machine
 -kernel
 -device sga (remove it)

For (1) and (2) you can use a qemu wrapper to modify the qemu command
that the test tool runs.

See also:

http://libguestfs.org/guestfs.3.html#qemu-wrappers

http://rwmj.wordpress.com/2011/10/12/tip-debugging-the-early-boot-process-with-qemu-and-gdb/#content

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

Re: hang after seabios

[Zetan Drableg]

Hi Richard thanks for the info.
I took the strace approach and ran into this looping over and over again.
Is it failing to get time?

timer_gettime(0x8, {it_interval={0, 0}, it_value={0, 0}}) = 0
timer_settime(0x8, 0, {it_interval={0, 0}, it_value={0, 250000}}, NULL) = 0
timer_gettime(0x8, {it_interval={0, 0}, it_value={0, 204443}}) = 0
select(16, [0 6 9 13 15], [], [], {1, 0}) = 2 (in [6 13], left {0, 999998})
read(13, "\1\0\0\0\0\0\0\0", 4096)      = 8
read(13, 0x7fffa2ed3f70, 4096)          = -1 EAGAIN (Resource
temporarily unavailable)
read(6, "\0", 512)                      = 1
read(6, 0x7fffa2ed4d70, 512)            = -1 EAGAIN (Resource
temporarily unavailable)
select(16, [0 6 9 13 15], [], [], {1, 0}) = 1 (in [15], left {0, 999998})
read(15, "\16\0\0\0\0\0\0\0\376\377\377\377\0\0\0\0\0\0\0\0\0\0\0\0\10\0\0\0\0\0\0\0"...,
128) = 128
rt_sigaction(SIGALRM, NULL, {0x7f6d8b8d17d0, ~[KILL STOP RTMIN RT_1],
SA_RESTORER, 0x7f6d8b211710}, 8) = 0
write(7, "\0", 1)                       = 1
write(14, "\1\0\0\0\0\0\0\0", 8)        = 8
read(15, 0x7fffa2ed4ee0, 128)           = -1 EAGAIN (Resource
temporarily unavailable)
timer_gettime(0x8, {it_interval={0, 0}, it_value={0, 0}}) = 0
timer_settime(0x8, 0, {it_interval={0, 0}, it_value={0, 988758000}}, NULL) = 0
select(16, [0 6 9 13 15], [], [], {1, 0}) = 2 (in [6 13], left {0, 999998})
read(13, "\1\0\0\0\0\0\0\0", 4096)      = 8
read(13, 0x7fffa2ed3f70, 4096)          = -1 EAGAIN (Resource
temporarily unavailable)
read(6, "\0", 512)                      = 1
read(6, 0x7fffa2ed4d70, 512)            = -1 EAGAIN (Resource
temporarily unavailable)

It looks a lot like the bug you filed here.
https://bugzilla.redhat.com/show_bug.cgi?id=553689

There are quite a few hits on this trace but nothing real specific.

-Zetan



On Thu, Jul 31, 2014 at 4:31 AM, Richard W.M. Jones <rjones@redhat.com> wrote:
> On Wed, Jul 30, 2014 at 03:58:43PM -0700, Zetan Drableg wrote:
>>    [00183ms] /usr/libexec/qemu-kvm \
>>        -global virtio-blk-pci.scsi=off \
>>        -nodefconfig \
>>        -nodefaults \
>>        -nographic \
>>        -machine accel=kvm:tcg \
>>        -cpu host,+kvmclock \
>>        -m 500 \
>>        -no-reboot \
>>        -kernel /var/tmp/.guestfs-0/kernel.47903 \
>>        -initrd /var/tmp/.guestfs-0/initrd.47903 \
>>        -device virtio-scsi-pci,id=scsi \
>>        -drive file=/tmp/libguestfs-test-tool-sda-Iakpwe,cache=none,format
>>    =raw,id=hd0,if=none \
>>        -device scsi-hd,drive=hd0 \
>>        -drive file=/var/tmp/.guestfs-0/root.47903,snapshot=on,id=appliance,
>>    if=none,cache=unsafe \
>>        -device scsi-hd,drive=appliance \
>>        -device virtio-serial \
>>        -serial stdio \
>>        -device sga \
>>        -chardev socket,path=/tmp/libguestfspx9994/guestfsd.sock,id=channel0
>>    \
>>        -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0
>>    \
>>        -append 'panic=1 console=ttyS0 udevtimeout=600 no_timer_check
>>    acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0
>>    guestfs_verbose=1 TERM=xterm edd=off'
>>    \x1b[1;256r\x1b[256;256H\x1b[6n
>>    Google, Inc.
>>    Serial Graphics Adapter 10/14/11
>>    SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $ (mockbuild@
>>    ca-build44.us.oracle.com) Fri Oct 14 20:04:36 UTC 2011
>>    Term: 80x24
>>    4 0
>>    SeaBIOS (version seabios-0.6.1.2-28.el6)
>>    \x1b[2J
>>
>> At this point it hangs forever.
>
> These hangs can be tricky to diagnose.
>
> There are a couple of things you can do however:
>
> (1) strace qemu to find out what it is doing, or:
>
> (2) [harder, but much more informative] gdb into the guest to find out
> where the guest hangs, or:
>
> (3) Take the command line above, and cut it down to try to isolate the
> problematic options.  I would concentrate on the following options as
> being most likely to cause trouble:
>
>  -cpu
>  -machine
>  -kernel
>  -device sga (remove it)
>
> For (1) and (2) you can use a qemu wrapper to modify the qemu command
> that the test tool runs.
>
> See also:
>
> http://libguestfs.org/guestfs.3.html#qemu-wrappers
>
> http://rwmj.wordpress.com/2011/10/12/tip-debugging-the-early-boot-process-with-qemu-and-gdb/#content
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-builder quickly builds VMs from scratch
> http://libguestfs.org/virt-builder.1.html

Re: hang after seabios

[Richard W.M. Jones]

On Thu, Jul 31, 2014 at 12:12:26PM -0700, Zetan Drableg wrote:
> Hi Richard thanks for the info.
> I took the strace approach and ran into this looping over and over again.
> Is it failing to get time?
> 
> timer_gettime(0x8, {it_interval={0, 0}, it_value={0, 0}}) = 0
> timer_settime(0x8, 0, {it_interval={0, 0}, it_value={0, 250000}}, NULL) = 0
> timer_gettime(0x8, {it_interval={0, 0}, it_value={0, 204443}}) = 0
> select(16, [0 6 9 13 15], [], [], {1, 0}) = 2 (in [6 13], left {0, 999998})
> read(13, "\1\0\0\0\0\0\0\0", 4096)      = 8
> read(13, 0x7fffa2ed3f70, 4096)          = -1 EAGAIN (Resource
> temporarily unavailable)
> read(6, "\0", 512)                      = 1
> read(6, 0x7fffa2ed4d70, 512)            = -1 EAGAIN (Resource
> temporarily unavailable)
> select(16, [0 6 9 13 15], [], [], {1, 0}) = 1 (in [15], left {0, 999998})
> read(15, "\16\0\0\0\0\0\0\0\376\377\377\377\0\0\0\0\0\0\0\0\0\0\0\0\10\0\0\0\0\0\0\0"...,
> 128) = 128
> rt_sigaction(SIGALRM, NULL, {0x7f6d8b8d17d0, ~[KILL STOP RTMIN RT_1],
> SA_RESTORER, 0x7f6d8b211710}, 8) = 0
> write(7, "\0", 1)                       = 1
> write(14, "\1\0\0\0\0\0\0\0", 8)        = 8
> read(15, 0x7fffa2ed4ee0, 128)           = -1 EAGAIN (Resource
> temporarily unavailable)
> timer_gettime(0x8, {it_interval={0, 0}, it_value={0, 0}}) = 0
> timer_settime(0x8, 0, {it_interval={0, 0}, it_value={0, 988758000}}, NULL) = 0
> select(16, [0 6 9 13 15], [], [], {1, 0}) = 2 (in [6 13], left {0, 999998})
> read(13, "\1\0\0\0\0\0\0\0", 4096)      = 8
> read(13, 0x7fffa2ed3f70, 4096)          = -1 EAGAIN (Resource
> temporarily unavailable)
> read(6, "\0", 512)                      = 1
> read(6, 0x7fffa2ed4d70, 512)            = -1 EAGAIN (Resource
> temporarily unavailable)
> 
> It looks a lot like the bug you filed here.
> https://bugzilla.redhat.com/show_bug.cgi?id=553689

I'm fairly sure this is just qemu running as normal.  It's not a
duplicate of that ancient bug, because you can see from the messages
that SeaBIOS is running.

You need to `gdb' into the guest to see where the emulation got to.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top