Re: Hangs

chris
2008-12-18T18:05:50Z

On Wed, Dec 03, 2008 at 09:49:18AM -0800, chris@versecorp.net wrote:
> On Wed, Dec 03, 2008 at 12:44:54PM +0200, Avi Kivity wrote:
> > chris@versecorp.net wrote:
> > >
> > >I have a way to reproduce my instance of the problem easily now. I was
> > >trying
> > >to build a new kernel on my guest, and found that depmod hangs guests
> > >every time.
> > > In my case, I only have an amd processor - I don't have an intel
> > >host to try it on, right now, but it happens on Ubuntu 8.04
> > >and Ubuntu 8.10 guests, both using kvm-79 and the version of kvm that ships
> > >with ubuntu 8.10.
> > >
> >
> > What's your guest, how is qemu launched (command line)?
> >
> > --
> > error compiling committee.c: too many arguments to function
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe kvm" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> The guest is Ubuntu 8.10 server (64-bit version). I also have the same
> problems with Ubuntu 8.04LTS server.
>
> Here's the command line:
>
> sudo /usr/local/bin/qemu-system-x86_64 \
> -no-kvm-irqchip \
> -daemonize \
> -hda Imgs/sam_home.img \
> -m 512 \
> -cdrom ISOs/ubuntu-8.10-server-amd64.iso \
> -parallel /dev/lp0 \
> -vnc :1 \
> -net nic,macaddr=DE:AD:BE:EF:01:01,model=e1000 \
> -net tap,ifname=tap1,script=/home/chris/kvm/qemu-ifup.sh \
> >>& Logs/sam_run.log
>
> Earlier in the mail chain, Marcelo had me run vmstat when it was hung,
> and it was all zeros. He also asked for a stack trace on the qemu and it
> showed two threads:
>
> (gdb) info threads
> 2 Thread 0x414f1950 (LWP 422) 0x00007f36f07a03e1 in sigtimedwait ()
> from /lib/libc.so.6
> 1 Thread 0x7f36f1f306e0 (LWP 414) 0x00007f36f084b482 in select ()
> from /lib/libc.so.6
> (gdb) thread 1
> [Switching to thread 1 (Thread 0x7f36f1f306e0 (LWP 414))]#0 0x00007f36f084b482
> +in select () from /lib/libc.so.6
> (gdb) bt
> #0 0x00007f36f084b482 in select () from /lib/libc.so.6
> #1 0x00000000004094cb in main_loop_wait (timeout=0)
> at /home/chris/pkgs/kvm/kvm-79/qemu/vl.c:4719
> #2 0x000000000050a7ea in kvm_main_loop ()
> at /home/chris/pkgs/kvm/kvm-79/qemu/qemu-kvm.c:619
> #3 0x000000000040fafc in main (argc=<value optimized out>,
> argv=0x7ffff9f41948) at /home/chris/pkgs/kvm/kvm-79/qemu/vl.c:4871
> (gdb) thread 2
> [Switching to thread 2 (Thread 0x414f1950 (LWP 422))]#0 0x00007f36f07a03e1 in
> +sigtimedwait () from /lib/libc.so.6
> (gdb) bt
> #0 0x00007f36f07a03e1 in sigtimedwait () from /lib/libc.so.6
> #1 0x000000000050a560 in kvm_main_loop_wait (env=0xc319e0, timeout=0)
> at /home/chris/pkgs/kvm/kvm-79/qemu/qemu-kvm.c:284
> #2 0x000000000050aaf7 in ap_main_loop (_env=<value optimized out>)
> at /home/chris/pkgs/kvm/kvm-79/qemu/qemu-kvm.c:425
> #3 0x00007f36f11ba3ea in start_thread () from /lib/libpthread.so.0
> #4 0x00007f36f0852c6d in clone () from /lib/libc.so.6
> #5 0x0000000000000000 in ?? ()
>
> If I can provide any other debug info I'm happy to. I'm beginning to suspect
> you'll be able to easily reproduce it if you run Ubuntu 8.10 as a guest on
> amd processors.
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

FYI, I'm seeing hangs on kvm-81, although the circumstances seem to have
changes somewhat. Now, when the guest hangs, the qemu-system-x86 process
in the host runs 100% busy. And now I do see some kvm_stat counters
increasing, not just all 0s:

kvm_stat -l
-----------------
efer_relo exits fpu_reloa halt_exit halt_wake host_stat hypercall insn_emul insn_emul invlpg io_exits irq_exits irq_injec irq_windo kvm_reque largepage mmio_exit mmu_cache mmu_flood mmu_pde_z mmu_pte_u mmu_pte_w mmu_recyc mmu_shado mmu_unsyn mmu_unsyn nmi_injec nmi_windo pf_fixed pf_guest remote_tl request_n signal_ex tlb_flush
0 277 0 0 0 29 0 0 0 0 0 277 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 289 0 0 0 35 0 0 0 0 0 290 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 277 0 0 0 27 0 0 0 0 0 277 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 290 0 0 0 33 0 0 0 0 0 289 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 290 0 0 0 34 0 0 0 0 0 290 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 288 0 0 0 30 0 0 0 0 0 288 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 296 0 0 0 36 0 0 0 0 0 297 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 292 0 0 0 33 0 0 0 0 0 291 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 275 0 0 0 26 0 0 0 0 0 275 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 289 0 0 0 31 0 0 0 0 0 290 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 309 0 0 0 41 0 0 0 0 0 308 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 289 0 0 0 29 0 0 0 0 0 289 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

GDB on it shows:
---------------
(gdb) info threads
2 Thread 0x41fad950 (LWP 6514) 0x00007f498e8e4a17 in ioctl ()
from /lib/libc.so.6
1 Thread 0x7f498ffc36e0 (LWP 6506) 0x00007f498e8e5482 in select ()
from /lib/libc.so.6

(gdb) thread 1
[Switching to thread 1 (Thread 0x7f498ffc36e0 (LWP 6506))]#0 0x00007f498e8e5482 in select () from /lib/libc.so.6
(gdb) bt
#0 0x00007f498e8e5482 in select () from /lib/libc.so.6
#1 0x0000000000409b8b in main_loop_wait (timeout=0)
at /home/chris/pkgs/kvm/kvm-81/qemu/vl.c:3623
#2 0x000000000051653a in kvm_main_loop ()
at /home/chris/pkgs/kvm/kvm-81/qemu/qemu-kvm.c:597
#3 0x000000000040e0bc in main (argc=<value optimized out>,
argv=0x7fff97fda878) at /home/chris/pkgs/kvm/kvm-81/qemu/vl.c:3785

(gdb) thread 2
[Switching to thread 2 (Thread 0x41fad950 (LWP 6514))]#0 0x00007f498e8e4a17 in ioctl () from /lib/libc.so.6
(gdb) bt
#0 0x00007f498e8e4a17 in ioctl () from /lib/libc.so.6
#1 0x0000000000543bba in kvm_run (kvm=0xbe0040, vcpu=0, env=0xc5a1d0)
at libkvm.c:881
#2 0x0000000000516649 in kvm_cpu_exec (env=<value optimized out>)
at /home/chris/pkgs/kvm/kvm-81/qemu/qemu-kvm.c:207
#3 0x0000000000516948 in ap_main_loop (_env=<value optimized out>)
at /home/chris/pkgs/kvm/kvm-81/qemu/qemu-kvm.c:414
#4 0x00007f498f2543ea in start_thread () from /lib/libpthread.so.0
#5 0x00007f498e8ecc6d in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()

Thanks,
Chris