* Unknown KVM internal error on 3.2.1 @ 2012-02-25 15:35 Doug Goldstein 2012-03-07 10:41 ` Avi Kivity 2014-04-10 8:24 ` LEo 0 siblings, 2 replies; 10+ messages in thread From: Doug Goldstein @ 2012-02-25 15:35 UTC (permalink / raw) To: KVM mailing list Hello, I'm seeing the following failure when running on 3.2.1: LC_ALL=C PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin HOME=/ USER=root QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -S -M pc-0.13 -cpu qemu32 -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1 -name bb-2.6.18-238.el5.i686,process=qemu:bb-2.6.18-238.el5.i686 -uuid 60a8accb-35c3-8d3a-f785-b38dc733bed2 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/bb-2.6.18-238.el5.i686.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -drive file=/dev/disk/by-path/ip-192.168.200.20:3260-iscsi-iqn.2011-07.com.company.lab.san-1:2.6.18-238.el5-i686-lun-0,if=none,id=drive-virtio-disk0,format=raw,werror=stop,rerror=stop -device virtio-blk-pci,bus=pci.0,multifunction=on,addr=0x4.0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=16,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:eb:ff:46,bus=pci.0,multifunction=on,addr=0x3.0x0 -chardev file,id=charserial0,path=/var/log/libvirt/qemu/serial/bb-2.6.18-238.el5-i686.log -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc 0.0.0.0:4,password -vga cirrus -device i6300esb,id=watchdog0,bus=pci.0,multifunction=on,addr=0x6.0x0 -watchdog-action reset -device virtio-balloon-pci,id=balloon0,bus=pci.0,multifunction=on,addr=0x5.0x0 Domain id=39 is tainted: high-privileges KVM internal error. Suberror: 1 emulation failure EAX=8004003b EBX=38d54633 ECX=c0460a7e EDX=8005003b ESI=e49329a8 EDI=f7c98d60 EBP=00000286 ESP=f7fecf68 EIP=f91d1778 EFL=00000282 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =007b 00000000 ffffffff 00c0f300 DPL=3 DS [-WA] CS =0060 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA] SS =0068 00000000 ffffffff 00c09300 DPL=0 DS [-WA] DS =007b 00000000 ffffffff 00c0f300 DPL=3 DS [-WA] FS =0000 00000000 ffffffff 00000000 GS =0000 b7f526c0 ffffffff 00000000 LDT=0088 c074a020 00000027 00008200 DPL=0 LDT TR =0080 c180a7c4 00002073 00008b00 DPL=0 TSS32-busy GDT= f7c9f000 000000ff IDT= c06fa000 000007ff CR0=8005003b CR2=0046b044 CR3=3100d000 CR4=000006d0 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? uname: Linux roids 3.2.1-gentoo-r2 #1 SMP Thu Feb 2 04:55:45 CST 2012 x86_64 AMD Opteron(TM) Processor 6272 AuthenticAMD GNU/Linux The Gentoo patchset for this kernel is just a stock 3.2.1 kernel with the following patches applied: http://sources.gentoo.org/cgi-bin/viewvc.cgi/linux-patches/genpatches-2.6/trunk/3.2/2600_Input-ALPS-synaptics-touchpad.patch?revision=2047&view=markup http://sources.gentoo.org/cgi-bin/viewvc.cgi/linux-patches/genpatches-2.6/trunk/3.2/4200_fbcondecor-0.9.6.patch?view=markup cpu: processor : 63 vendor_id : AuthenticAMD cpu family : 21 model : 1 model name : AMD Opteron(TM) Processor 6272 stepping : 2 microcode : 0x6000613 cpu MHz : 2099.875 cache size : 2048 KB physical id : 2 siblings : 16 core id : 7 cpu cores : 8 apicid : 79 initial apicid : 79 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 nodeid_msr topoext perfctr_core arat cpb npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold bogomips : 4200.08 TLB size : 1536 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: ts ttp tm 100mhzsteps hwpstate [9] The machine only runs 16 VMs presently. RHEL 5.0 through 5.7 in i686 and x86_64 variants. All are configured identically. Additionally there are 2 VMs with 6GB of memory running RHEL 5.4 and 5.5 i686, but otherwise configured identically. The only VMs that trigger this internal error are the RHEL 5.6 and RHEL 5.7 i686 guests. So far there seems to be no formula to reproduce, the VMs can be completely idle or they can be in the middle of a GCC compile for the issue to trigger. The problem did not occur on the same hardware running 2.6.38. If you can think of any further information necessary, please let me know and I'll get it. -- Doug Goldstein ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Unknown KVM internal error on 3.2.1 2012-02-25 15:35 Unknown KVM internal error on 3.2.1 Doug Goldstein @ 2012-03-07 10:41 ` Avi Kivity 2012-03-07 11:00 ` Gleb Natapov 2013-01-30 0:21 ` Doug Goldstein 2014-04-10 8:24 ` LEo 1 sibling, 2 replies; 10+ messages in thread From: Avi Kivity @ 2012-03-07 10:41 UTC (permalink / raw) To: Doug Goldstein; +Cc: KVM mailing list On 02/25/2012 05:35 PM, Doug Goldstein wrote: > Hello, I'm seeing the following failure when running on 3.2.1: <snip funky qemu commandline> > KVM internal error. Suberror: 1 > emulation failure > EAX=8004003b EBX=38d54633 ECX=c0460a7e EDX=8005003b > ESI=e49329a8 EDI=f7c98d60 EBP=00000286 ESP=f7fecf68 > EIP=f91d1778 EFL=00000282 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0 > ES =007b 00000000 ffffffff 00c0f300 DPL=3 DS [-WA] > CS =0060 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA] > SS =0068 00000000 ffffffff 00c09300 DPL=0 DS [-WA] > DS =007b 00000000 ffffffff 00c0f300 DPL=3 DS [-WA] > FS =0000 00000000 ffffffff 00000000 > GS =0000 b7f526c0 ffffffff 00000000 > LDT=0088 c074a020 00000027 00008200 DPL=0 LDT > TR =0080 c180a7c4 00002073 00008b00 DPL=0 TSS32-busy > GDT= f7c9f000 000000ff > IDT= c06fa000 000007ff > CR0=8005003b CR2=0046b044 CR3=3100d000 CR4=000006d0 > DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 > DR3=0000000000000000 > DR6=00000000ffff0ff0 DR7=0000000000000400 > EFER=0000000000000000 > Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? > ?? ?? ?? ?? ?? ?? What are all these ?? doing here? Usually they indicate the bad code, but here they don't, this is strange. Please issue the qemu monitor command x/20i 0xf91d1778 on the guest that failed (after restarting it). The value comes from the EIP register in the dump above, so you can use it for other guests. -- error compiling committee.c: too many arguments to function ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Unknown KVM internal error on 3.2.1 2012-03-07 10:41 ` Avi Kivity @ 2012-03-07 11:00 ` Gleb Natapov 2012-03-07 14:19 ` Avi Kivity 2013-01-30 0:21 ` Doug Goldstein 1 sibling, 1 reply; 10+ messages in thread From: Gleb Natapov @ 2012-03-07 11:00 UTC (permalink / raw) To: Avi Kivity; +Cc: Doug Goldstein, KVM mailing list On Wed, Mar 07, 2012 at 12:41:16PM +0200, Avi Kivity wrote: > On 02/25/2012 05:35 PM, Doug Goldstein wrote: > > Hello, I'm seeing the following failure when running on 3.2.1: > > <snip funky qemu commandline> > > > KVM internal error. Suberror: 1 > > emulation failure > > EAX=8004003b EBX=38d54633 ECX=c0460a7e EDX=8005003b > > ESI=e49329a8 EDI=f7c98d60 EBP=00000286 ESP=f7fecf68 > > EIP=f91d1778 EFL=00000282 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0 > > ES =007b 00000000 ffffffff 00c0f300 DPL=3 DS [-WA] > > CS =0060 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA] > > SS =0068 00000000 ffffffff 00c09300 DPL=0 DS [-WA] > > DS =007b 00000000 ffffffff 00c0f300 DPL=3 DS [-WA] > > FS =0000 00000000 ffffffff 00000000 > > GS =0000 b7f526c0 ffffffff 00000000 > > LDT=0088 c074a020 00000027 00008200 DPL=0 LDT > > TR =0080 c180a7c4 00002073 00008b00 DPL=0 TSS32-busy > > GDT= f7c9f000 000000ff > > IDT= c06fa000 000007ff > > CR0=8005003b CR2=0046b044 CR3=3100d000 CR4=000006d0 > > DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 > > DR3=0000000000000000 > > DR6=00000000ffff0ff0 DR7=0000000000000400 > > EFER=0000000000000000 > > Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? > > ?? ?? ?? ?? ?? ?? > > What are all these ?? doing here? Usually they indicate the bad code, > but here they don't, this is strange. > I think it tries to execute code from mmio. > Please issue the qemu monitor command > > x/20i 0xf91d1778 > > on the guest that failed (after restarting it). The value comes from > the EIP register in the dump above, so you can use it for other guests. > > > -- > error compiling committee.c: too many arguments to function > > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Gleb. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Unknown KVM internal error on 3.2.1 2012-03-07 11:00 ` Gleb Natapov @ 2012-03-07 14:19 ` Avi Kivity 2012-03-20 20:40 ` Doug Goldstein 2012-03-26 20:12 ` Doug Goldstein 0 siblings, 2 replies; 10+ messages in thread From: Avi Kivity @ 2012-03-07 14:19 UTC (permalink / raw) To: Gleb Natapov; +Cc: Doug Goldstein, KVM mailing list On 03/07/2012 01:00 PM, Gleb Natapov wrote: > > > > > KVM internal error. Suberror: 1 > > > emulation failure > > > EAX=8004003b EBX=38d54633 ECX=c0460a7e EDX=8005003b > > > ESI=e49329a8 EDI=f7c98d60 EBP=00000286 ESP=f7fecf68 > > > EIP=f91d1778 EFL=00000282 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0 > > > ES =007b 00000000 ffffffff 00c0f300 DPL=3 DS [-WA] > > > CS =0060 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA] > > > SS =0068 00000000 ffffffff 00c09300 DPL=0 DS [-WA] > > > DS =007b 00000000 ffffffff 00c0f300 DPL=3 DS [-WA] > > > FS =0000 00000000 ffffffff 00000000 > > > GS =0000 b7f526c0 ffffffff 00000000 > > > LDT=0088 c074a020 00000027 00008200 DPL=0 LDT > > > TR =0080 c180a7c4 00002073 00008b00 DPL=0 TSS32-busy > > > GDT= f7c9f000 000000ff > > > IDT= c06fa000 000007ff > > > CR0=8005003b CR2=0046b044 CR3=3100d000 CR4=000006d0 > > > DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 > > > DR3=0000000000000000 > > > DR6=00000000ffff0ff0 DR7=0000000000000400 > > > EFER=0000000000000000 > > > Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> > > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? > > > ?? ?? ?? ?? ?? ?? > > > > What are all these ?? doing here? Usually they indicate the bad code, > > but here they don't, this is strange. > > > I think it tries to execute code from mmio. > Likely. But let's be sure. When it happens again, please keep the guest alive so we can examine it via qemu monitor commands. -- error compiling committee.c: too many arguments to function ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Unknown KVM internal error on 3.2.1 2012-03-07 14:19 ` Avi Kivity @ 2012-03-20 20:40 ` Doug Goldstein 2012-03-26 20:12 ` Doug Goldstein 1 sibling, 0 replies; 10+ messages in thread From: Doug Goldstein @ 2012-03-20 20:40 UTC (permalink / raw) To: Avi Kivity; +Cc: Gleb Natapov, KVM mailing list, libvir-list On Wed, Mar 7, 2012 at 8:19 AM, Avi Kivity <avi@redhat.com> wrote: > On 03/07/2012 01:00 PM, Gleb Natapov wrote: >> > >> > > KVM internal error. Suberror: 1 >> > > emulation failure >> > > EAX=8004003b EBX=38d54633 ECX=c0460a7e EDX=8005003b >> > > ESI=e49329a8 EDI=f7c98d60 EBP=00000286 ESP=f7fecf68 >> > > EIP=f91d1778 EFL=00000282 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0 >> > > ES =007b 00000000 ffffffff 00c0f300 DPL=3 DS [-WA] >> > > CS =0060 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA] >> > > SS =0068 00000000 ffffffff 00c09300 DPL=0 DS [-WA] >> > > DS =007b 00000000 ffffffff 00c0f300 DPL=3 DS [-WA] >> > > FS =0000 00000000 ffffffff 00000000 >> > > GS =0000 b7f526c0 ffffffff 00000000 >> > > LDT=0088 c074a020 00000027 00008200 DPL=0 LDT >> > > TR =0080 c180a7c4 00002073 00008b00 DPL=0 TSS32-busy >> > > GDT= f7c9f000 000000ff >> > > IDT= c06fa000 000007ff >> > > CR0=8005003b CR2=0046b044 CR3=3100d000 CR4=000006d0 >> > > DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 >> > > DR3=0000000000000000 >> > > DR6=00000000ffff0ff0 DR7=0000000000000400 >> > > EFER=0000000000000000 >> > > Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> >> > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? >> > > ?? ?? ?? ?? ?? ?? >> > >> > What are all these ?? doing here? Usually they indicate the bad code, >> > but here they don't, this is strange. >> > >> I think it tries to execute code from mmio. >> > > Likely. But let's be sure. > > When it happens again, please keep the guest alive so we can examine it > via qemu monitor commands. > > -- > error compiling committee.c: too many arguments to function > Shortly after I sent the original e-mail I told libvirt to use the host CPU. As discussed in another thread since libvirt uses -nodefconfig, it doesn't really have the correct CPU. I believe the original issue is from the fact that I had an AMD K10 based processor with the errata that was previously causing some issues when it wasn't respected. The default qemu64 processor that libvirt had the guest use did not take into account this errata and causes this issue. I have reverted these CPU settings and will get the monitor information as soon as it occurs again. Providing all the background just to be sure. The host is: processor : 63 vendor_id : AuthenticAMD cpu family : 21 model : 1 model name : AMD Opteron(TM) Processor 6272 stepping : 2 microcode : 0x6000613 cpu MHz : 2099.875 cache size : 2048 KB physical id : 2 siblings : 16 core id : 7 cpu cores : 8 apicid : 79 initial apicid : 79 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 nodeid_msr topoext perfctr_core arat cpb npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold bogomips : 4200.08 TLB size : 1536 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: ts ttp tm 100mhzsteps hwpstate [9] The guest is: processor : 1 vendor_id : AuthenticAMD cpu family : 6 model : 15 model name : Intel(R) Core(TM)2 Duo CPU T7700 @ 2.40GHz stepping : 11 cpu MHz : 2100.397 cache size : 512 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb lm pni cx16 popcnt lahf_lm cmp_legacy svm cr8legacy abm sse4a misalignsse 3dnowprefetch bogomips : 4200.01 The QEMU command line contains the following: -cpu core2duo,+wdt,+skinit,+osvw,+3dnowprefetch,+misalignsse,+sse4a,+abm,+cr8legacy,+extapic,+svm,+cmp_legacy,+lahf_lm,+rdtscp,+pdpe1gb,+fxsr_opt,+mmxext,+aes,+popcnt,+sse4.2,+sse4.1,+cx16,+ht While libvirt's XML contains: <cpu match='exact'> <model>Opteron_G3</model> <vendor>AMD</vendor> <feature policy='require' name='aes'/> <feature policy='require' name='skinit'/> <feature policy='require' name='vme'/> <feature policy='require' name='mmxext'/> <feature policy='require' name='fxsr_opt'/> <feature policy='require' name='cr8legacy'/> <feature policy='require' name='ht'/> <feature policy='require' name='3dnowprefetch'/> <feature policy='require' name='ssse3'/> <feature policy='require' name='wdt'/> <feature policy='require' name='extapic'/> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='osvw'/> <feature policy='require' name='sse4.1'/> <feature policy='require' name='cmp_legacy'/> <feature policy='require' name='sse4.2'/> </cpu> What the guest used to be with qemu64 CPU: processor : 1 vendor_id : AuthenticAMD cpu family : 6 model : 3 model name : QEMU Virtual CPU version 0.15.1 stepping : 3 cpu MHz : 2100.026 fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 4 wp : yes flags : fpu de pse tsc msr pae mce cx8 apic pge cmov pat mmx fxsr sse sse2 pni popcnt bogomips : 4199.33 I also find it odd that the CPU says its 0.15.1 when -M pc-0.13 is explicitly passed in via the command line. -- Doug Goldstein ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Unknown KVM internal error on 3.2.1 2012-03-07 14:19 ` Avi Kivity 2012-03-20 20:40 ` Doug Goldstein @ 2012-03-26 20:12 ` Doug Goldstein 2012-03-27 10:53 ` Avi Kivity 1 sibling, 1 reply; 10+ messages in thread From: Doug Goldstein @ 2012-03-26 20:12 UTC (permalink / raw) To: Avi Kivity; +Cc: Gleb Natapov, KVM mailing list On Wed, Mar 7, 2012 at 8:19 AM, Avi Kivity <avi@redhat.com> wrote: > On 03/07/2012 01:00 PM, Gleb Natapov wrote: >> > >> > > KVM internal error. Suberror: 1 >> > > emulation failure >> > > EAX=8004003b EBX=38d54633 ECX=c0460a7e EDX=8005003b >> > > ESI=e49329a8 EDI=f7c98d60 EBP=00000286 ESP=f7fecf68 >> > > EIP=f91d1778 EFL=00000282 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0 >> > > ES =007b 00000000 ffffffff 00c0f300 DPL=3 DS [-WA] >> > > CS =0060 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA] >> > > SS =0068 00000000 ffffffff 00c09300 DPL=0 DS [-WA] >> > > DS =007b 00000000 ffffffff 00c0f300 DPL=3 DS [-WA] >> > > FS =0000 00000000 ffffffff 00000000 >> > > GS =0000 b7f526c0 ffffffff 00000000 >> > > LDT=0088 c074a020 00000027 00008200 DPL=0 LDT >> > > TR =0080 c180a7c4 00002073 00008b00 DPL=0 TSS32-busy >> > > GDT= f7c9f000 000000ff >> > > IDT= c06fa000 000007ff >> > > CR0=8005003b CR2=0046b044 CR3=3100d000 CR4=000006d0 >> > > DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 >> > > DR3=0000000000000000 >> > > DR6=00000000ffff0ff0 DR7=0000000000000400 >> > > EFER=0000000000000000 >> > > Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> >> > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? >> > > ?? ?? ?? ?? ?? ?? >> > >> > What are all these ?? doing here? Usually they indicate the bad code, >> > but here they don't, this is strange. >> > >> I think it tries to execute code from mmio. >> > > Likely. But let's be sure. > > When it happens again, please keep the guest alive so we can examine it > via qemu monitor commands. > It happened again. Here's the output requested. Before resuming: # virsh qemu-monitor-command bb-rhel-5.3-i686 --hmp x/20i 0xf91b4748 0x00000000f91b4748: lock popf 0x00000000f91b474a: lock popf 0x00000000f91b474c: lock popf 0x00000000f91b474e: lock popf 0x00000000f91b4750: lock popf 0x00000000f91b4752: lock popf 0x00000000f91b4754: lock popf 0x00000000f91b4756: lock popf 0x00000000f91b4758: lock popf 0x00000000f91b475a: lock popf 0x00000000f91b475c: lock popf 0x00000000f91b475e: lock popf 0x00000000f91b4760: lock popf 0x00000000f91b4762: lock popf 0x00000000f91b4764: lock popf 0x00000000f91b4766: lock popf 0x00000000f91b4768: lock popf 0x00000000f91b476a: lock popf 0x00000000f91b476c: lock popf 0x00000000f91b476e: lock popf After resuming and immediately running the requested command: # virsh qemu-monitor-command bb-rhel-5.3-i686 --hmp x/20i 0xf91b4748 0x00000000f91b4748: mov %eax,%cr0 0x00000000f91b474b: mov %ebx,(%ecx) 0x00000000f91b474d: mov %edx,%cr0 0x00000000f91b4750: mov (%esi),%esi 0x00000000f91b4752: mov (%esi),%eax 0x00000000f91b4754: prefetchnta (%eax) 0x00000000f91b4757: nop 0x00000000f91b4758: mov 0xf91db9f8,%eax 0x00000000f91b475d: add $0x8,%eax 0x00000000f91b4760: cmp %eax,%esi 0x00000000f91b4762: jne 0xf91b4738 0x00000000f91b4764: lock incl 0xf91da3b0 0x00000000f91b476b: pop %ebx 0x00000000f91b476c: mov $0x5,%edx 0x00000000f91b4771: pop %esi 0x00000000f91b4772: mov $0xf91da380,%eax 0x00000000f91b4777: jmp 0xc0432047 0x00000000f91b477c: push %ebp 0x00000000f91b477d: push %edi 0x00000000f91b477e: push %esi LC_ALL=C PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin HOME=/ USER=root QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -S -M pc-0.13 -cpu qemu32 -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1 -name bb-rhel-5.3-i686,process=qemu:bb-rhel-5.3-i686 -uuid bc053d3a-ece8-8f25-92ea-7ff197f72ac5 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/bb-rhel-5.3-i686.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -drive file=/dev/disk/by-path/ip-192.168.200.20:3260-iscsi-iqn.2011-07.com.pikewerks.lab.san-1:bb-rhel-5.3-i686-lun-0,if=none,id=drive-ide0-0-0,format=raw,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=15,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:af:90:fa,bus=pci.0,multifunction=on,addr=0x3.0x0 -chardev file,id=charserial0,path=/var/log/libvirt/qemu/serial/bb-rhel-5.3-i686.log -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc 0.0.0.0:19,password -vga cirrus -device AC97,id=sound0,bus=pci.0,multifunction=on,addr=0x4.0x0 -device i6300esb,id=watchdog0,bus=pci.0,multifunction=on,addr=0x6.0x0 -watchdog-action reset -device virtio-balloon-pci,id=balloon0,bus=pci.0,multifunction=on,addr=0x5.0x0 Domain id=30 is tainted: high-privileges KVM internal error. Suberror: 1 emulation failure EAX=8004003b EBX=00685b04 ECX=f8b1767e EDX=8005003b ESI=ce687888 EDI=f7caff60 EBP=00000286 ESP=f7feff68 EIP=f91b4748 EFL=00000282 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =007b 00000000 ffffffff 00c0f300 DPL=3 DS [-WA] CS =0060 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA] SS =0068 00000000 ffffffff 00c09300 DPL=0 DS [-WA] DS =007b 00000000 ffffffff 00c0f300 DPL=3 DS [-WA] FS =0000 00000000 ffffffff 00000000 GS =0000 b7ff06c0 ffffffff 00000000 LDT=0088 c073f020 00000027 00008200 DPL=0 LDT TR =0080 c1804000 00002073 00008b00 DPL=0 TSS32-busy GDT= c1812000 000000ff IDT= c06ee000 000007ff CR0=8005003b CR2=091020c8 CR3=37f5f000 CR4=000006d0 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? -- Doug Goldstein ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Unknown KVM internal error on 3.2.1 2012-03-26 20:12 ` Doug Goldstein @ 2012-03-27 10:53 ` Avi Kivity 2012-03-27 20:00 ` Doug Goldstein 0 siblings, 1 reply; 10+ messages in thread From: Avi Kivity @ 2012-03-27 10:53 UTC (permalink / raw) To: Doug Goldstein; +Cc: Gleb Natapov, KVM mailing list On 03/26/2012 10:12 PM, Doug Goldstein wrote: > > > > When it happens again, please keep the guest alive so we can examine it > > via qemu monitor commands. > > > > It happened again. Here's the output requested. > > Before resuming: > > # virsh qemu-monitor-command bb-rhel-5.3-i686 --hmp x/20i 0xf91b4748 > 0x00000000f91b4748: lock popf > 0x00000000f91b474a: lock popf > 0x00000000f91b474c: lock popf > 0x00000000f91b474e: lock popf > 0x00000000f91b4750: lock popf > 0x00000000f91b4752: lock popf > 0x00000000f91b4754: lock popf > 0x00000000f91b4756: lock popf > 0x00000000f91b4758: lock popf > 0x00000000f91b475a: lock popf > 0x00000000f91b475c: lock popf > 0x00000000f91b475e: lock popf > 0x00000000f91b4760: lock popf > 0x00000000f91b4762: lock popf > 0x00000000f91b4764: lock popf > 0x00000000f91b4766: lock popf > 0x00000000f91b4768: lock popf > 0x00000000f91b476a: lock popf > 0x00000000f91b476c: lock popf > 0x00000000f91b476e: lock popf > > After resuming and immediately running the requested command: Did the guest run properly after resuming? > > # virsh qemu-monitor-command bb-rhel-5.3-i686 --hmp x/20i 0xf91b4748 > 0x00000000f91b4748: mov %eax,%cr0 > 0x00000000f91b474b: mov %ebx,(%ecx) > 0x00000000f91b474d: mov %edx,%cr0 > 0x00000000f91b4750: mov (%esi),%esi > 0x00000000f91b4752: mov (%esi),%eax > 0x00000000f91b4754: prefetchnta (%eax) > 0x00000000f91b4757: nop > 0x00000000f91b4758: mov 0xf91db9f8,%eax > 0x00000000f91b475d: add $0x8,%eax > 0x00000000f91b4760: cmp %eax,%esi > 0x00000000f91b4762: jne 0xf91b4738 > 0x00000000f91b4764: lock incl 0xf91da3b0 > 0x00000000f91b476b: pop %ebx > 0x00000000f91b476c: mov $0x5,%edx > 0x00000000f91b4771: pop %esi > 0x00000000f91b4772: mov $0xf91da380,%eax > 0x00000000f91b4777: jmp 0xc0432047 > 0x00000000f91b477c: push %ebp > 0x00000000f91b477d: push %edi > 0x00000000f91b477e: push %esi > This is perfectly sane and expected code (unlike the previous dump). Are you running with transparent hugepages and/or ksm enabled? You might try to disable them. -- error compiling committee.c: too many arguments to function ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Unknown KVM internal error on 3.2.1 2012-03-27 10:53 ` Avi Kivity @ 2012-03-27 20:00 ` Doug Goldstein 0 siblings, 0 replies; 10+ messages in thread From: Doug Goldstein @ 2012-03-27 20:00 UTC (permalink / raw) To: Avi Kivity; +Cc: Gleb Natapov, KVM mailing list On Tue, Mar 27, 2012 at 5:53 AM, Avi Kivity <avi@redhat.com> wrote: > On 03/26/2012 10:12 PM, Doug Goldstein wrote: >> > >> > When it happens again, please keep the guest alive so we can examine it >> > via qemu monitor commands. >> > >> >> It happened again. Here's the output requested. >> >> Before resuming: >> >> # virsh qemu-monitor-command bb-rhel-5.3-i686 --hmp x/20i 0xf91b4748 >> 0x00000000f91b4748: lock popf >> 0x00000000f91b474a: lock popf >> 0x00000000f91b474c: lock popf >> 0x00000000f91b474e: lock popf >> 0x00000000f91b4750: lock popf >> 0x00000000f91b4752: lock popf >> 0x00000000f91b4754: lock popf >> 0x00000000f91b4756: lock popf >> 0x00000000f91b4758: lock popf >> 0x00000000f91b475a: lock popf >> 0x00000000f91b475c: lock popf >> 0x00000000f91b475e: lock popf >> 0x00000000f91b4760: lock popf >> 0x00000000f91b4762: lock popf >> 0x00000000f91b4764: lock popf >> 0x00000000f91b4766: lock popf >> 0x00000000f91b4768: lock popf >> 0x00000000f91b476a: lock popf >> 0x00000000f91b476c: lock popf >> 0x00000000f91b476e: lock popf >> >> After resuming and immediately running the requested command: > > Did the guest run properly after resuming? Yes. They always run fine after resuming. About 50% of the time they'll hit the same issue within a few seconds/minute. And another resume and they're off and running again for weeks. > >> >> # virsh qemu-monitor-command bb-rhel-5.3-i686 --hmp x/20i 0xf91b4748 >> 0x00000000f91b4748: mov %eax,%cr0 >> 0x00000000f91b474b: mov %ebx,(%ecx) >> 0x00000000f91b474d: mov %edx,%cr0 >> 0x00000000f91b4750: mov (%esi),%esi >> 0x00000000f91b4752: mov (%esi),%eax >> 0x00000000f91b4754: prefetchnta (%eax) >> 0x00000000f91b4757: nop >> 0x00000000f91b4758: mov 0xf91db9f8,%eax >> 0x00000000f91b475d: add $0x8,%eax >> 0x00000000f91b4760: cmp %eax,%esi >> 0x00000000f91b4762: jne 0xf91b4738 >> 0x00000000f91b4764: lock incl 0xf91da3b0 >> 0x00000000f91b476b: pop %ebx >> 0x00000000f91b476c: mov $0x5,%edx >> 0x00000000f91b4771: pop %esi >> 0x00000000f91b4772: mov $0xf91da380,%eax >> 0x00000000f91b4777: jmp 0xc0432047 >> 0x00000000f91b477c: push %ebp >> 0x00000000f91b477d: push %edi >> 0x00000000f91b477e: push %esi >> > > This is perfectly sane and expected code (unlike the previous dump). > > Are you running with transparent hugepages and/or ksm enabled? You > might try to disable them. Yes. I'm running with both. I'll try disabling them both and see if this reoccurs. Thanks. -- Doug Goldstein ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Unknown KVM internal error on 3.2.1 2012-03-07 10:41 ` Avi Kivity 2012-03-07 11:00 ` Gleb Natapov @ 2013-01-30 0:21 ` Doug Goldstein 1 sibling, 0 replies; 10+ messages in thread From: Doug Goldstein @ 2013-01-30 0:21 UTC (permalink / raw) To: KVM mailing list On Wed, Mar 7, 2012 at 4:41 AM, Avi Kivity <avi@redhat.com> wrote: > On 02/25/2012 05:35 PM, Doug Goldstein wrote: >> Hello, I'm seeing the following failure when running on 3.2.1: > > <snip funky qemu commandline> > >> KVM internal error. Suberror: 1 >> emulation failure >> EAX=8004003b EBX=38d54633 ECX=c0460a7e EDX=8005003b >> ESI=e49329a8 EDI=f7c98d60 EBP=00000286 ESP=f7fecf68 >> EIP=f91d1778 EFL=00000282 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0 >> ES =007b 00000000 ffffffff 00c0f300 DPL=3 DS [-WA] >> CS =0060 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA] >> SS =0068 00000000 ffffffff 00c09300 DPL=0 DS [-WA] >> DS =007b 00000000 ffffffff 00c0f300 DPL=3 DS [-WA] >> FS =0000 00000000 ffffffff 00000000 >> GS =0000 b7f526c0 ffffffff 00000000 >> LDT=0088 c074a020 00000027 00008200 DPL=0 LDT >> TR =0080 c180a7c4 00002073 00008b00 DPL=0 TSS32-busy >> GDT= f7c9f000 000000ff >> IDT= c06fa000 000007ff >> CR0=8005003b CR2=0046b044 CR3=3100d000 CR4=000006d0 >> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 >> DR3=0000000000000000 >> DR6=00000000ffff0ff0 DR7=0000000000000400 >> EFER=0000000000000000 >> Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> >> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? >> ?? ?? ?? ?? ?? ?? > > What are all these ?? doing here? Usually they indicate the bad code, > but here they don't, this is strange. > > Please issue the qemu monitor command > > x/20i 0xf91d1778 > > on the guest that failed (after restarting it). The value comes from > the EIP register in the dump above, so you can use it for other guests. > > > -- > error compiling committee.c: too many arguments to function > Its been a while so I wanted to retry this. For background I have the following setup. Host: 3.6.11 kernel, qemu-kvm 1.2.0 + qemu 1.2.2 + extra patches (what ships in Fedora 18), AMD Opteron 6272 processor. qemu command line snippet showing CPU configuration: -M pc-1.2 -cpu Opteron_G4,+perfctr_nb,+perfctr_core ,+topoext,+nodeid_msr,+lwp,+wdt,+skinit,+ibs,+osvw,+cr8legacy,+extapic,+cmp_lega cy,+fxsr_opt,+mmxext,+osxsave,+monitor,+ht,+vme -enable-kvm -m 4096 -smp 2,socke ts=1,cores=2,threads=1 libvirt CPU: <cpu mode='host-model'> <model fallback='allow'/> <topology sockets='1' cores='2' threads='1'/> </cpu> The error: KVM internal error. Suberror: 1 emulation failure EAX=8004003b EBX=00000004 ECX=00000004 EDX=8005003b ESI=f0f2b510 EDI=f8b1d67e EBP=00000286 ESP=f7feff58 EIP=f9258392 EFL=00000282 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =007b 00000000 ffffffff 00c0f300 DPL=3 DS [-WA] CS =0060 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA] SS =0068 00000000 ffffffff 00c09300 DPL=0 DS [-WA] DS =007b 00000000 ffffffff 00c0f300 DPL=3 DS [-WA] FS =0000 00000000 ffffffff 00000000 GS =0000 b7f2d6c0 ffffffff 00000000 LDT=0088 c073f020 00000027 00008200 DPL=0 LDT TR =0080 c3010000 00002073 00008b00 DPL=0 TSS32-busy GDT= c301e000 000000ff IDT= c06ee000 000007ff CR0=8005003b CR2=00bbe2c0 CR3=355e7000 CR4=000006d0 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? # virsh qemu-monitor-command bad-domain --hmp x/20i 0xf9258392 0x00000000f9258392: add %al,(%eax) 0x00000000f9258394: add %al,(%eax) 0x00000000f9258396: add %al,(%eax) 0x00000000f9258398: add %al,(%eax) 0x00000000f925839a: add %al,(%eax) 0x00000000f925839c: add %al,(%eax) 0x00000000f925839e: add %al,(%eax) 0x00000000f92583a0: add %al,(%eax) 0x00000000f92583a2: add %al,(%eax) 0x00000000f92583a4: add %al,(%eax) 0x00000000f92583a6: add %al,(%eax) 0x00000000f92583a8: add %al,(%eax) 0x00000000f92583aa: add %al,(%eax) 0x00000000f92583ac: add %al,(%eax) 0x00000000f92583ae: add %al,(%eax) 0x00000000f92583b0: add %al,(%eax) 0x00000000f92583b2: add %al,(%eax) 0x00000000f92583b4: add %al,(%eax) 0x00000000f92583b6: add %al,(%eax) 0x00000000f92583b8: add %al,(%eax) When I attempted to resume the VM, either libvirt of qemu stopped me telling me that resetting the VM is required. But grabbing the memory at EIP resulted in a different value... # virsh qemu-monitor-command bad-domain --hmp x/20i 0xf9258392 0x00000000f9258392: dec %eax 0x00000000f9258393: dec %eax 0x00000000f9258394: dec %eax 0x00000000f9258395: dec %eax 0x00000000f9258396: dec %eax 0x00000000f9258397: dec %eax 0x00000000f9258398: dec %eax 0x00000000f9258399: dec %eax 0x00000000f925839a: dec %eax 0x00000000f925839b: dec %eax 0x00000000f925839c: dec %eax 0x00000000f925839d: dec %eax 0x00000000f925839e: dec %eax 0x00000000f925839f: dec %eax 0x00000000f92583a0: dec %eax 0x00000000f92583a1: dec %eax 0x00000000f92583a2: dec %eax 0x00000000f92583a3: dec %eax 0x00000000f92583a4: dec %eax 0x00000000f92583a5: dec %eax -- Doug Goldstein ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Unknown KVM internal error on 3.2.1 2012-02-25 15:35 Unknown KVM internal error on 3.2.1 Doug Goldstein 2012-03-07 10:41 ` Avi Kivity @ 2014-04-10 8:24 ` LEo 1 sibling, 0 replies; 10+ messages in thread From: LEo @ 2014-04-10 8:24 UTC (permalink / raw) To: kvm me too and help ! #uname -a Linux rhel64pt 3.14.0-rc3+ #1 SMP Thu Mar 27 17:29:24 CST 2014 x86_64 x86_64 x86_64 GNU/Linux # qemu-system-x86_64 -m 2048 -smp 4 -boot order=cd -hda /root/kvm_demo/rhel64.img -cdrom /appstore/iso/rhel-server-6.4-x86_64- dvd.iso -vnc :2 KVM internal error. Suberror: 1 emulation failure RAX=ffffffff81000122 RBX=0000000001f8b000 RCX=0000000001d55000 RDX=0000000001000000 RSI=0000000000093780 RDI=0000000001a8c000 RBP=0000000000000000 RSP=000000000236e140 R8 =0000000001a8c000 R9 =0000000000000001 R10=0000000000000038 R11=0000000000000038 R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000 RIP=ffffffff81000122 RFL=00010006 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 0000000000000000 ffffffff 00c00000 CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA] SS =0000 0000000000000000 ffffffff 00c00000 DS =0000 0000000000000000 ffffffff 00c00000 FS =0000 0000000000000000 ffffffff 00c00000 GS =0000 0000000000000000 ffffffff 00c00000 LDT=0000 0000000000000000 ffffffff 00c00000 TR =0020 0000000000000000 00000fff 00808b00 DPL=0 TSS64-busy GDT= 00000000004d80d8 00000030 IDT= 0000000000000000 00000000 CR0=80000011 CR2=0000000000000000 CR3=000000909238e090 CR4=000000a0 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000500 Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2014-04-10 8:30 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-02-25 15:35 Unknown KVM internal error on 3.2.1 Doug Goldstein 2012-03-07 10:41 ` Avi Kivity 2012-03-07 11:00 ` Gleb Natapov 2012-03-07 14:19 ` Avi Kivity 2012-03-20 20:40 ` Doug Goldstein 2012-03-26 20:12 ` Doug Goldstein 2012-03-27 10:53 ` Avi Kivity 2012-03-27 20:00 ` Doug Goldstein 2013-01-30 0:21 ` Doug Goldstein 2014-04-10 8:24 ` LEo
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox