Re: Hangs

[chris@versecorp.net]

On Wed, Nov 19, 2008 at 02:43:42PM -0800, chris@versecorp.net wrote:
> Thanks for the responses,
> 
> I'm not sure if my problem is the same as Roland's, but it definitely sounds
> plausible.  I had been running ntpdate in the host to synchronize time every hour (in a cron job), so it sounds as if we could be seeing the same issue.
> 

Actually, with ntpdate taken out of crontab, I'm still seeing periodic hangs, so it's either a different problem or I'm hitting it in a different manner.

OK, I installed kvm-79 and kernel 2.6.27.6, and here's the the kvm-stat output 
with 1 guest hung and 3 more operational:

 efer_relo      exits  fpu_reloa  halt_exit  halt_wake  host_stat  hypercall  insn_emul  insn_emul     invlpg   io_exits  irq_exits  irq_windo  largepage  mmio_exit  mmu_cache  mmu_flood  mmu_pde_z  mmu_pte_u  mmu_pte_w  mmu_recyc  mmu_shado  nmi_windo   pf_fixed   pf_guest  remote_tl  request_i  signal_ex  tlb_flush
         0        333         24         32          0        331          2        212          0          0         78          4          0          0        188          0          0          0          2          2          0          0          0          1          1          0          0          0         12
         0        360          3         30          0        331          0        290          0          0          0          4          0          0        269          0          0          0          5          5          0          0          0         35          5          0          0          0         15
         0        287          2         30          0        307          0        202          0          0         52          2          0          0        194          0          0          0          0          0          0          0          0          0          0          0          0          0          4
         0        389         20         29          0        405          0        277          0          0         78          3          0          0        267          0          0          0          0          0          0          0          0          0          0          0          0          0          6
         0        307          4         32          0        315          0        219          0          0         52          3          0          0        198          0          0          0          0          0          0          0          0          0          0          0          0          0         11
         0        327          2         35          0        346          2        285          0          0          0          4          0          0        274          0          0          0          2          2          0          0          0          1          1          0          0          0          7
         0        334         22         31          0        342          0        217          0          0         78          4          0          0        201          0          0          0          0          0          0          0          0          0          0          0          0          0          8
         0        311          3         28          0        324          0        280          0          0          0          2          0          0        265          0          0          0          0          0          0          0          0          0          0          0          0          0          9
         0        292          2         32          0        313          0        204          0          0         52          3          0          0        196          0          0          0          0          0          0          0          0          0          0          0          0          0          4
         0        791         23         46          0        780         10        352          0          0        364          6          0          0        320          0          0          0         10         10          0          0          0          5          5          0          0          0         20
         0        251          3         30          0        259          2        214          0          0          0          4          0          0        198          0          0          0          2          2          0          0          0          1          1          0          0          0         10
         0        313          2         31          0        330          0        278          0          0          0          4          0          0        266          0          0          0          0          0          0          0          0          0          0          0          0          0          6
         0        330         22         30          0        339          0        215          0          0         78          3          0          0        199          0          0          0          0          0          0          0          0          0          0          0          0          0          8

If I shut down the 3 operational guests leaving just the hung guest, the
kvm-stat output is all 0s:

 efer_relo      exits  fpu_reloa  halt_exit  halt_wake  host_stat  hypercall  insn_emul  insn_emul     invlpg   io_exits  irq_exits  irq_windo  largepage  mmio_exit  mmu_cache  mmu_flood  mmu_pde_z  mmu_pte_u  mmu_pte_w  mmu_recyc  mmu_shado  nmi_windo   pf_fixed   pf_guest  remote_tl  request_i  signal_ex  tlb_flush
         0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0
         0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0
         0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0
         0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0
         0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0


The hung guest in this case was run with this command:

sudo /usr/local/bin/qemu-system-x86_64             \
     -daemonize                                    \
     -no-kvm-irqchip                               \
     -hda Imgs/ndev_root.img                       \
     -m 1024                                       \
     -cdrom ISOs/ubuntu-8.10-server-amd64.iso      \
     -vnc :4                                       \
     -net nic,macaddr=DE:AD:BE:EF:04:04,model=e1000 \
     -net tap,ifname=tap4,script=/home/chris/kvm/qemu-ifup.sh \
     >>& Logs/ndev_run.log


I should also mention that when the guest is hung, I can still switch to the monitor with ctrl-alt 2.   So, at least it's a little bit alive.

I've also noticed that the behavior with the hung guest is slightly different on kvm-79 than it was earlier.  When the guest hangs, the kvm process in the host doesn't spin at 100% busy any longer - the guest is just unresponsive at both the network and VNC console.

Also, I've noticed that if I reset the guest from the monitor, the guest will boot up again, and I can get through to it on the network, but strangely, the mouse and keyboard will still be hung at the VNC console (except that I can still switch back and forth to the monitor).  

Hope some of this helps, let me know if you need to me to provide any other troubleshooting info.

Chris