Unknown KVM internal error on 3.2.1

Unknown KVM internal error on 3.2.1

[Doug Goldstein]

Hello, I'm seeing the following failure when running on 3.2.1:

LC_ALL=C PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
HOME=/ USER=root QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -S -M pc-0.13
-cpu qemu32 -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1
-name bb-2.6.18-238.el5.i686,process=qemu:bb-2.6.18-238.el5.i686 -uuid
60a8accb-35c3-8d3a-f785-b38dc733bed2 -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/bb-2.6.18-238.el5.i686.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -drive
file=/dev/disk/by-path/ip-192.168.200.20:3260-iscsi-iqn.2011-07.com.company.lab.san-1:2.6.18-238.el5-i686-lun-0,if=none,id=drive-virtio-disk0,format=raw,werror=stop,rerror=stop
-device virtio-blk-pci,bus=pci.0,multifunction=on,addr=0x4.0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fd=16,id=hostnet0 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:eb:ff:46,bus=pci.0,multifunction=on,addr=0x3.0x0
-chardev file,id=charserial0,path=/var/log/libvirt/qemu/serial/bb-2.6.18-238.el5-i686.log
-device isa-serial,chardev=charserial0,id=serial0 -usb -vnc
0.0.0.0:4,password -vga cirrus -device
i6300esb,id=watchdog0,bus=pci.0,multifunction=on,addr=0x6.0x0
-watchdog-action reset -device
virtio-balloon-pci,id=balloon0,bus=pci.0,multifunction=on,addr=0x5.0x0
Domain id=39 is tainted: high-privileges
KVM internal error. Suberror: 1
emulation failure
EAX=8004003b EBX=38d54633 ECX=c0460a7e EDX=8005003b
ESI=e49329a8 EDI=f7c98d60 EBP=00000286 ESP=f7fecf68
EIP=f91d1778 EFL=00000282 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =007b 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
CS =0060 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
SS =0068 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =007b 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
FS =0000 00000000 ffffffff 00000000
GS =0000 b7f526c0 ffffffff 00000000
LDT=0088 c074a020 00000027 00008200 DPL=0 LDT
TR =0080 c180a7c4 00002073 00008b00 DPL=0 TSS32-busy
GDT=     f7c9f000 000000ff
IDT=     c06fa000 000007ff
CR0=8005003b CR2=0046b044 CR3=3100d000 CR4=000006d0
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??>
?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
?? ?? ?? ?? ?? ??

uname:
Linux roids 3.2.1-gentoo-r2 #1 SMP Thu Feb 2 04:55:45 CST 2012 x86_64
AMD Opteron(TM) Processor 6272 AuthenticAMD GNU/Linux

The Gentoo patchset for this kernel is just a stock 3.2.1 kernel with
the following patches applied:
http://sources.gentoo.org/cgi-bin/viewvc.cgi/linux-patches/genpatches-2.6/trunk/3.2/2600_Input-ALPS-synaptics-touchpad.patch?revision=2047&view=markup
http://sources.gentoo.org/cgi-bin/viewvc.cgi/linux-patches/genpatches-2.6/trunk/3.2/4200_fbcondecor-0.9.6.patch?view=markup

cpu:
processor	: 63
vendor_id	: AuthenticAMD
cpu family	: 21
model		: 1
model name	: AMD Opteron(TM) Processor 6272
stepping	: 2
microcode	: 0x6000613
cpu MHz		: 2099.875
cache size	: 2048 KB
physical id	: 2
siblings	: 16
core id		: 7
cpu cores	: 8
apicid		: 79
initial apicid	: 79
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt
pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid
amd_dcm aperfmperf pni pclmulqdq monitor ssse3 cx16 sse4_1 sse4_2
popcnt aes xsave avx lahf_lm cmp_legacy svm extapic cr8_legacy abm
sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4
nodeid_msr topoext perfctr_core arat cpb npt lbrv svm_lock nrip_save
tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bogomips	: 4200.08
TLB size	: 1536 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate [9]


The machine only runs 16 VMs presently. RHEL 5.0 through 5.7 in i686
and x86_64 variants. All are configured identically. Additionally
there are 2 VMs with 6GB of memory running RHEL 5.4 and 5.5 i686, but
otherwise configured identically. The only VMs that trigger this
internal error are the RHEL 5.6 and RHEL 5.7 i686 guests. So far there
seems to be no formula to reproduce, the VMs can be completely idle or
they can be in the middle of a GCC compile for the issue to trigger.
The problem did not occur on the same hardware running 2.6.38.

If you can think of any further information necessary, please let me
know and I'll get it.

-- 
Doug Goldstein

Re: Unknown KVM internal error on 3.2.1

[Avi Kivity]

On 02/25/2012 05:35 PM, Doug Goldstein wrote:
> Hello, I'm seeing the following failure when running on 3.2.1:

<snip funky qemu commandline>

> KVM internal error. Suberror: 1
> emulation failure
> EAX=8004003b EBX=38d54633 ECX=c0460a7e EDX=8005003b
> ESI=e49329a8 EDI=f7c98d60 EBP=00000286 ESP=f7fecf68
> EIP=f91d1778 EFL=00000282 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0
> ES =007b 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
> CS =0060 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
> SS =0068 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
> DS =007b 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
> FS =0000 00000000 ffffffff 00000000
> GS =0000 b7f526c0 ffffffff 00000000
> LDT=0088 c074a020 00000027 00008200 DPL=0 LDT
> TR =0080 c180a7c4 00002073 00008b00 DPL=0 TSS32-busy
> GDT=     f7c9f000 000000ff
> IDT=     c06fa000 000007ff
> CR0=8005003b CR2=0046b044 CR3=3100d000 CR4=000006d0
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
> DR3=0000000000000000
> DR6=00000000ffff0ff0 DR7=0000000000000400
> EFER=0000000000000000
> Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??>
> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
> ?? ?? ?? ?? ?? ??

What are all these ?? doing here?  Usually they indicate the bad code,
but here they don't, this is strange.

Please issue the qemu monitor command

  x/20i 0xf91d1778

on the guest that failed (after restarting it).  The value comes from
the EIP register in the dump above, so you can use it for other guests.


-- 
error compiling committee.c: too many arguments to function

Re: Unknown KVM internal error on 3.2.1

[Gleb Natapov]

On Wed, Mar 07, 2012 at 12:41:16PM +0200, Avi Kivity wrote:
> On 02/25/2012 05:35 PM, Doug Goldstein wrote:
> > Hello, I'm seeing the following failure when running on 3.2.1:
> 
> <snip funky qemu commandline>
> 
> > KVM internal error. Suberror: 1
> > emulation failure
> > EAX=8004003b EBX=38d54633 ECX=c0460a7e EDX=8005003b
> > ESI=e49329a8 EDI=f7c98d60 EBP=00000286 ESP=f7fecf68
> > EIP=f91d1778 EFL=00000282 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0
> > ES =007b 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
> > CS =0060 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
> > SS =0068 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
> > DS =007b 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
> > FS =0000 00000000 ffffffff 00000000
> > GS =0000 b7f526c0 ffffffff 00000000
> > LDT=0088 c074a020 00000027 00008200 DPL=0 LDT
> > TR =0080 c180a7c4 00002073 00008b00 DPL=0 TSS32-busy
> > GDT=     f7c9f000 000000ff
> > IDT=     c06fa000 000007ff
> > CR0=8005003b CR2=0046b044 CR3=3100d000 CR4=000006d0
> > DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
> > DR3=0000000000000000
> > DR6=00000000ffff0ff0 DR7=0000000000000400
> > EFER=0000000000000000
> > Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??>
> > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
> > ?? ?? ?? ?? ?? ??
> 
> What are all these ?? doing here?  Usually they indicate the bad code,
> but here they don't, this is strange.
> 
I think it tries to execute code from mmio.

> Please issue the qemu monitor command
> 
>   x/20i 0xf91d1778
> 
> on the guest that failed (after restarting it).  The value comes from
> the EIP register in the dump above, so you can use it for other guests.
> 
> 
> -- 
> error compiling committee.c: too many arguments to function
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
			Gleb.

Re: Unknown KVM internal error on 3.2.1

[Avi Kivity]

On 03/07/2012 01:00 PM, Gleb Natapov wrote:
> > 
> > > KVM internal error. Suberror: 1
> > > emulation failure
> > > EAX=8004003b EBX=38d54633 ECX=c0460a7e EDX=8005003b
> > > ESI=e49329a8 EDI=f7c98d60 EBP=00000286 ESP=f7fecf68
> > > EIP=f91d1778 EFL=00000282 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0
> > > ES =007b 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
> > > CS =0060 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
> > > SS =0068 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
> > > DS =007b 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
> > > FS =0000 00000000 ffffffff 00000000
> > > GS =0000 b7f526c0 ffffffff 00000000
> > > LDT=0088 c074a020 00000027 00008200 DPL=0 LDT
> > > TR =0080 c180a7c4 00002073 00008b00 DPL=0 TSS32-busy
> > > GDT=     f7c9f000 000000ff
> > > IDT=     c06fa000 000007ff
> > > CR0=8005003b CR2=0046b044 CR3=3100d000 CR4=000006d0
> > > DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
> > > DR3=0000000000000000
> > > DR6=00000000ffff0ff0 DR7=0000000000000400
> > > EFER=0000000000000000
> > > Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??>
> > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
> > > ?? ?? ?? ?? ?? ??
> > 
> > What are all these ?? doing here?  Usually they indicate the bad code,
> > but here they don't, this is strange.
> > 
> I think it tries to execute code from mmio.
>

Likely.  But let's be sure.

When it happens again, please keep the guest alive so we can examine it
via qemu monitor commands. 

-- 
error compiling committee.c: too many arguments to function

Re: Unknown KVM internal error on 3.2.1

[Doug Goldstein]

On Wed, Mar 7, 2012 at 8:19 AM, Avi Kivity <avi@redhat.com> wrote:
> On 03/07/2012 01:00 PM, Gleb Natapov wrote:
>> >
>> > > KVM internal error. Suberror: 1
>> > > emulation failure
>> > > EAX=8004003b EBX=38d54633 ECX=c0460a7e EDX=8005003b
>> > > ESI=e49329a8 EDI=f7c98d60 EBP=00000286 ESP=f7fecf68
>> > > EIP=f91d1778 EFL=00000282 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0
>> > > ES =007b 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
>> > > CS =0060 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
>> > > SS =0068 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
>> > > DS =007b 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
>> > > FS =0000 00000000 ffffffff 00000000
>> > > GS =0000 b7f526c0 ffffffff 00000000
>> > > LDT=0088 c074a020 00000027 00008200 DPL=0 LDT
>> > > TR =0080 c180a7c4 00002073 00008b00 DPL=0 TSS32-busy
>> > > GDT=     f7c9f000 000000ff
>> > > IDT=     c06fa000 000007ff
>> > > CR0=8005003b CR2=0046b044 CR3=3100d000 CR4=000006d0
>> > > DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>> > > DR3=0000000000000000
>> > > DR6=00000000ffff0ff0 DR7=0000000000000400
>> > > EFER=0000000000000000
>> > > Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??>
>> > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
>> > > ?? ?? ?? ?? ?? ??
>> >
>> > What are all these ?? doing here?  Usually they indicate the bad code,
>> > but here they don't, this is strange.
>> >
>> I think it tries to execute code from mmio.
>>
>
> Likely.  But let's be sure.
>
> When it happens again, please keep the guest alive so we can examine it
> via qemu monitor commands.
>
> --
> error compiling committee.c: too many arguments to function
>

Shortly after I sent the original e-mail I told libvirt to use the
host CPU. As discussed in another thread since libvirt uses
-nodefconfig, it doesn't really have the correct CPU. I believe the
original issue is from the fact that I had an AMD K10 based processor
with the errata that was previously causing some issues when it wasn't
respected. The default qemu64 processor that libvirt had the guest use
did not take into account this errata and causes this issue.

I have reverted these CPU settings and will get the monitor
information as soon as it occurs again.

Providing all the background just to be sure.


The host is:

processor	: 63
vendor_id	: AuthenticAMD
cpu family	: 21
model		: 1
model name	: AMD Opteron(TM) Processor 6272
stepping	: 2
microcode	: 0x6000613
cpu MHz		: 2099.875
cache size	: 2048 KB
physical id	: 2
siblings	: 16
core id		: 7
cpu cores	: 8
apicid		: 79
initial apicid	: 79
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt
pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid
amd_dcm aperfmperf pni pclmulqdq monitor ssse3 cx16 sse4_1 sse4_2
popcnt aes xsave avx lahf_lm cmp_legacy svm extapic cr8_legacy abm
sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4
nodeid_msr topoext perfctr_core arat cpb npt lbrv svm_lock nrip_save
tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bogomips	: 4200.08
TLB size	: 1536 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate [9]

The guest is:

processor	: 1
vendor_id	: AuthenticAMD
cpu family	: 6
model		: 15
model name	: Intel(R) Core(TM)2 Duo CPU     T7700  @ 2.40GHz
stepping	: 11
cpu MHz		: 2100.397
cache size	: 512 KB
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat
pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb lm
pni cx16 popcnt lahf_lm cmp_legacy svm cr8legacy abm sse4a misalignsse
3dnowprefetch
bogomips	: 4200.01

The QEMU command line contains the following:

-cpu core2duo,+wdt,+skinit,+osvw,+3dnowprefetch,+misalignsse,+sse4a,+abm,+cr8legacy,+extapic,+svm,+cmp_legacy,+lahf_lm,+rdtscp,+pdpe1gb,+fxsr_opt,+mmxext,+aes,+popcnt,+sse4.2,+sse4.1,+cx16,+ht

While libvirt's XML contains:

  <cpu match='exact'>
    <model>Opteron_G3</model>
    <vendor>AMD</vendor>
    <feature policy='require' name='aes'/>
    <feature policy='require' name='skinit'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='mmxext'/>
    <feature policy='require' name='fxsr_opt'/>
    <feature policy='require' name='cr8legacy'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='3dnowprefetch'/>
    <feature policy='require' name='ssse3'/>
    <feature policy='require' name='wdt'/>
    <feature policy='require' name='extapic'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='osvw'/>
    <feature policy='require' name='sse4.1'/>
    <feature policy='require' name='cmp_legacy'/>
    <feature policy='require' name='sse4.2'/>
  </cpu>

What the guest used to be with qemu64 CPU:

processor	: 1
vendor_id	: AuthenticAMD
cpu family	: 6
model		: 3
model name	: QEMU Virtual CPU version 0.15.1
stepping	: 3
cpu MHz		: 2100.026
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 4
wp		: yes
flags		: fpu de pse tsc msr pae mce cx8 apic pge cmov pat mmx fxsr sse
sse2 pni popcnt
bogomips	: 4199.33

I also find it odd that the CPU says its 0.15.1 when -M pc-0.13 is
explicitly passed in via the command line.

-- 
Doug Goldstein

Re: Unknown KVM internal error on 3.2.1

[Doug Goldstein]

On Wed, Mar 7, 2012 at 8:19 AM, Avi Kivity <avi@redhat.com> wrote:
> On 03/07/2012 01:00 PM, Gleb Natapov wrote:
>> >
>> > > KVM internal error. Suberror: 1
>> > > emulation failure
>> > > EAX=8004003b EBX=38d54633 ECX=c0460a7e EDX=8005003b
>> > > ESI=e49329a8 EDI=f7c98d60 EBP=00000286 ESP=f7fecf68
>> > > EIP=f91d1778 EFL=00000282 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0
>> > > ES =007b 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
>> > > CS =0060 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
>> > > SS =0068 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
>> > > DS =007b 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
>> > > FS =0000 00000000 ffffffff 00000000
>> > > GS =0000 b7f526c0 ffffffff 00000000
>> > > LDT=0088 c074a020 00000027 00008200 DPL=0 LDT
>> > > TR =0080 c180a7c4 00002073 00008b00 DPL=0 TSS32-busy
>> > > GDT=     f7c9f000 000000ff
>> > > IDT=     c06fa000 000007ff
>> > > CR0=8005003b CR2=0046b044 CR3=3100d000 CR4=000006d0
>> > > DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>> > > DR3=0000000000000000
>> > > DR6=00000000ffff0ff0 DR7=0000000000000400
>> > > EFER=0000000000000000
>> > > Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??>
>> > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
>> > > ?? ?? ?? ?? ?? ??
>> >
>> > What are all these ?? doing here?  Usually they indicate the bad code,
>> > but here they don't, this is strange.
>> >
>> I think it tries to execute code from mmio.
>>
>
> Likely.  But let's be sure.
>
> When it happens again, please keep the guest alive so we can examine it
> via qemu monitor commands.
>

It happened again. Here's the output requested.

Before resuming:

# virsh qemu-monitor-command bb-rhel-5.3-i686 --hmp x/20i 0xf91b4748
0x00000000f91b4748:  lock popf
0x00000000f91b474a:  lock popf
0x00000000f91b474c:  lock popf
0x00000000f91b474e:  lock popf
0x00000000f91b4750:  lock popf
0x00000000f91b4752:  lock popf
0x00000000f91b4754:  lock popf
0x00000000f91b4756:  lock popf
0x00000000f91b4758:  lock popf
0x00000000f91b475a:  lock popf
0x00000000f91b475c:  lock popf
0x00000000f91b475e:  lock popf
0x00000000f91b4760:  lock popf
0x00000000f91b4762:  lock popf
0x00000000f91b4764:  lock popf
0x00000000f91b4766:  lock popf
0x00000000f91b4768:  lock popf
0x00000000f91b476a:  lock popf
0x00000000f91b476c:  lock popf
0x00000000f91b476e:  lock popf

After resuming and immediately running the requested command:

# virsh qemu-monitor-command bb-rhel-5.3-i686 --hmp x/20i 0xf91b4748
0x00000000f91b4748:  mov    %eax,%cr0
0x00000000f91b474b:  mov    %ebx,(%ecx)
0x00000000f91b474d:  mov    %edx,%cr0
0x00000000f91b4750:  mov    (%esi),%esi
0x00000000f91b4752:  mov    (%esi),%eax
0x00000000f91b4754:  prefetchnta (%eax)
0x00000000f91b4757:  nop
0x00000000f91b4758:  mov    0xf91db9f8,%eax
0x00000000f91b475d:  add    $0x8,%eax
0x00000000f91b4760:  cmp    %eax,%esi
0x00000000f91b4762:  jne    0xf91b4738
0x00000000f91b4764:  lock incl 0xf91da3b0
0x00000000f91b476b:  pop    %ebx
0x00000000f91b476c:  mov    $0x5,%edx
0x00000000f91b4771:  pop    %esi
0x00000000f91b4772:  mov    $0xf91da380,%eax
0x00000000f91b4777:  jmp    0xc0432047
0x00000000f91b477c:  push   %ebp
0x00000000f91b477d:  push   %edi
0x00000000f91b477e:  push   %esi


LC_ALL=C PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
HOME=/ USER=root QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -S -M pc-0.13
-cpu qemu32 -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1
-name bb-rhel-5.3-i686,process=qemu:bb-rhel-5.3-i686 -uuid
bc053d3a-ece8-8f25-92ea-7ff197f72ac5 -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/bb-rhel-5.3-i686.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -drive
file=/dev/disk/by-path/ip-192.168.200.20:3260-iscsi-iqn.2011-07.com.pikewerks.lab.san-1:bb-rhel-5.3-i686-lun-0,if=none,id=drive-ide0-0-0,format=raw,cache=none
-device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
-netdev tap,fd=15,id=hostnet0 -device
rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:af:90:fa,bus=pci.0,multifunction=on,addr=0x3.0x0
-chardev file,id=charserial0,path=/var/log/libvirt/qemu/serial/bb-rhel-5.3-i686.log
-device isa-serial,chardev=charserial0,id=serial0 -usb -vnc
0.0.0.0:19,password -vga cirrus -device
AC97,id=sound0,bus=pci.0,multifunction=on,addr=0x4.0x0 -device
i6300esb,id=watchdog0,bus=pci.0,multifunction=on,addr=0x6.0x0
-watchdog-action reset -device
virtio-balloon-pci,id=balloon0,bus=pci.0,multifunction=on,addr=0x5.0x0
Domain id=30 is tainted: high-privileges
KVM internal error. Suberror: 1
emulation failure
EAX=8004003b EBX=00685b04 ECX=f8b1767e EDX=8005003b
ESI=ce687888 EDI=f7caff60 EBP=00000286 ESP=f7feff68
EIP=f91b4748 EFL=00000282 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =007b 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
CS =0060 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
SS =0068 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =007b 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
FS =0000 00000000 ffffffff 00000000
GS =0000 b7ff06c0 ffffffff 00000000
LDT=0088 c073f020 00000027 00008200 DPL=0 LDT
TR =0080 c1804000 00002073 00008b00 DPL=0 TSS32-busy
GDT=     c1812000 000000ff
IDT=     c06ee000 000007ff
CR0=8005003b CR2=091020c8 CR3=37f5f000 CR4=000006d0
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??>
?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
?? ?? ?? ?? ?? ??


-- 
Doug Goldstein

Re: Unknown KVM internal error on 3.2.1

[Avi Kivity]

On 03/26/2012 10:12 PM, Doug Goldstein wrote:
> >
> > When it happens again, please keep the guest alive so we can examine it
> > via qemu monitor commands.
> >
>
> It happened again. Here's the output requested.
>
> Before resuming:
>
> # virsh qemu-monitor-command bb-rhel-5.3-i686 --hmp x/20i 0xf91b4748
> 0x00000000f91b4748:  lock popf
> 0x00000000f91b474a:  lock popf
> 0x00000000f91b474c:  lock popf
> 0x00000000f91b474e:  lock popf
> 0x00000000f91b4750:  lock popf
> 0x00000000f91b4752:  lock popf
> 0x00000000f91b4754:  lock popf
> 0x00000000f91b4756:  lock popf
> 0x00000000f91b4758:  lock popf
> 0x00000000f91b475a:  lock popf
> 0x00000000f91b475c:  lock popf
> 0x00000000f91b475e:  lock popf
> 0x00000000f91b4760:  lock popf
> 0x00000000f91b4762:  lock popf
> 0x00000000f91b4764:  lock popf
> 0x00000000f91b4766:  lock popf
> 0x00000000f91b4768:  lock popf
> 0x00000000f91b476a:  lock popf
> 0x00000000f91b476c:  lock popf
> 0x00000000f91b476e:  lock popf
>
> After resuming and immediately running the requested command:

Did the guest run properly after resuming?

>
> # virsh qemu-monitor-command bb-rhel-5.3-i686 --hmp x/20i 0xf91b4748
> 0x00000000f91b4748:  mov    %eax,%cr0
> 0x00000000f91b474b:  mov    %ebx,(%ecx)
> 0x00000000f91b474d:  mov    %edx,%cr0
> 0x00000000f91b4750:  mov    (%esi),%esi
> 0x00000000f91b4752:  mov    (%esi),%eax
> 0x00000000f91b4754:  prefetchnta (%eax)
> 0x00000000f91b4757:  nop
> 0x00000000f91b4758:  mov    0xf91db9f8,%eax
> 0x00000000f91b475d:  add    $0x8,%eax
> 0x00000000f91b4760:  cmp    %eax,%esi
> 0x00000000f91b4762:  jne    0xf91b4738
> 0x00000000f91b4764:  lock incl 0xf91da3b0
> 0x00000000f91b476b:  pop    %ebx
> 0x00000000f91b476c:  mov    $0x5,%edx
> 0x00000000f91b4771:  pop    %esi
> 0x00000000f91b4772:  mov    $0xf91da380,%eax
> 0x00000000f91b4777:  jmp    0xc0432047
> 0x00000000f91b477c:  push   %ebp
> 0x00000000f91b477d:  push   %edi
> 0x00000000f91b477e:  push   %esi
>

This is perfectly sane and expected code (unlike the previous dump).

Are you running with transparent hugepages and/or ksm enabled?  You
might try to disable them.

-- 
error compiling committee.c: too many arguments to function

Re: Unknown KVM internal error on 3.2.1

[Doug Goldstein]

On Tue, Mar 27, 2012 at 5:53 AM, Avi Kivity <avi@redhat.com> wrote:
> On 03/26/2012 10:12 PM, Doug Goldstein wrote:
>> >
>> > When it happens again, please keep the guest alive so we can examine it
>> > via qemu monitor commands.
>> >
>>
>> It happened again. Here's the output requested.
>>
>> Before resuming:
>>
>> # virsh qemu-monitor-command bb-rhel-5.3-i686 --hmp x/20i 0xf91b4748
>> 0x00000000f91b4748:  lock popf
>> 0x00000000f91b474a:  lock popf
>> 0x00000000f91b474c:  lock popf
>> 0x00000000f91b474e:  lock popf
>> 0x00000000f91b4750:  lock popf
>> 0x00000000f91b4752:  lock popf
>> 0x00000000f91b4754:  lock popf
>> 0x00000000f91b4756:  lock popf
>> 0x00000000f91b4758:  lock popf
>> 0x00000000f91b475a:  lock popf
>> 0x00000000f91b475c:  lock popf
>> 0x00000000f91b475e:  lock popf
>> 0x00000000f91b4760:  lock popf
>> 0x00000000f91b4762:  lock popf
>> 0x00000000f91b4764:  lock popf
>> 0x00000000f91b4766:  lock popf
>> 0x00000000f91b4768:  lock popf
>> 0x00000000f91b476a:  lock popf
>> 0x00000000f91b476c:  lock popf
>> 0x00000000f91b476e:  lock popf
>>
>> After resuming and immediately running the requested command:
>
> Did the guest run properly after resuming?

Yes. They always run fine after resuming. About 50% of the time
they'll hit the same issue within a few seconds/minute. And another
resume and they're off and running again for weeks.
>
>>
>> # virsh qemu-monitor-command bb-rhel-5.3-i686 --hmp x/20i 0xf91b4748
>> 0x00000000f91b4748:  mov    %eax,%cr0
>> 0x00000000f91b474b:  mov    %ebx,(%ecx)
>> 0x00000000f91b474d:  mov    %edx,%cr0
>> 0x00000000f91b4750:  mov    (%esi),%esi
>> 0x00000000f91b4752:  mov    (%esi),%eax
>> 0x00000000f91b4754:  prefetchnta (%eax)
>> 0x00000000f91b4757:  nop
>> 0x00000000f91b4758:  mov    0xf91db9f8,%eax
>> 0x00000000f91b475d:  add    $0x8,%eax
>> 0x00000000f91b4760:  cmp    %eax,%esi
>> 0x00000000f91b4762:  jne    0xf91b4738
>> 0x00000000f91b4764:  lock incl 0xf91da3b0
>> 0x00000000f91b476b:  pop    %ebx
>> 0x00000000f91b476c:  mov    $0x5,%edx
>> 0x00000000f91b4771:  pop    %esi
>> 0x00000000f91b4772:  mov    $0xf91da380,%eax
>> 0x00000000f91b4777:  jmp    0xc0432047
>> 0x00000000f91b477c:  push   %ebp
>> 0x00000000f91b477d:  push   %edi
>> 0x00000000f91b477e:  push   %esi
>>
>
> This is perfectly sane and expected code (unlike the previous dump).
>
> Are you running with transparent hugepages and/or ksm enabled?  You
> might try to disable them.

Yes. I'm running with both. I'll try disabling them both and see if
this reoccurs.

Thanks.

-- 
Doug Goldstein

Re: Unknown KVM internal error on 3.2.1

[Doug Goldstein]

On Wed, Mar 7, 2012 at 4:41 AM, Avi Kivity <avi@redhat.com> wrote:
> On 02/25/2012 05:35 PM, Doug Goldstein wrote:
>> Hello, I'm seeing the following failure when running on 3.2.1:
>
> <snip funky qemu commandline>
>
>> KVM internal error. Suberror: 1
>> emulation failure
>> EAX=8004003b EBX=38d54633 ECX=c0460a7e EDX=8005003b
>> ESI=e49329a8 EDI=f7c98d60 EBP=00000286 ESP=f7fecf68
>> EIP=f91d1778 EFL=00000282 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0
>> ES =007b 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
>> CS =0060 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
>> SS =0068 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
>> DS =007b 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
>> FS =0000 00000000 ffffffff 00000000
>> GS =0000 b7f526c0 ffffffff 00000000
>> LDT=0088 c074a020 00000027 00008200 DPL=0 LDT
>> TR =0080 c180a7c4 00002073 00008b00 DPL=0 TSS32-busy
>> GDT=     f7c9f000 000000ff
>> IDT=     c06fa000 000007ff
>> CR0=8005003b CR2=0046b044 CR3=3100d000 CR4=000006d0
>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>> DR3=0000000000000000
>> DR6=00000000ffff0ff0 DR7=0000000000000400
>> EFER=0000000000000000
>> Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??>
>> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
>> ?? ?? ?? ?? ?? ??
>
> What are all these ?? doing here?  Usually they indicate the bad code,
> but here they don't, this is strange.
>
> Please issue the qemu monitor command
>
>   x/20i 0xf91d1778
>
> on the guest that failed (after restarting it).  The value comes from
> the EIP register in the dump above, so you can use it for other guests.
>
>
> --
> error compiling committee.c: too many arguments to function
>

Its been a while so I wanted to retry this. For background I have the
following setup.

Host: 3.6.11 kernel, qemu-kvm 1.2.0 + qemu 1.2.2 + extra patches (what
ships in Fedora 18), AMD Opteron 6272 processor.

qemu command line snippet showing CPU configuration: -M pc-1.2 -cpu
Opteron_G4,+perfctr_nb,+perfctr_core
,+topoext,+nodeid_msr,+lwp,+wdt,+skinit,+ibs,+osvw,+cr8legacy,+extapic,+cmp_lega
cy,+fxsr_opt,+mmxext,+osxsave,+monitor,+ht,+vme -enable-kvm -m 4096 -smp 2,socke
ts=1,cores=2,threads=1

libvirt CPU:
  <cpu mode='host-model'>
    <model fallback='allow'/>
    <topology sockets='1' cores='2' threads='1'/>
  </cpu>

The error:

KVM internal error. Suberror: 1
emulation failure
EAX=8004003b EBX=00000004 ECX=00000004 EDX=8005003b
ESI=f0f2b510 EDI=f8b1d67e EBP=00000286 ESP=f7feff58
EIP=f9258392 EFL=00000282 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =007b 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
CS =0060 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
SS =0068 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =007b 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
FS =0000 00000000 ffffffff 00000000
GS =0000 b7f2d6c0 ffffffff 00000000
LDT=0088 c073f020 00000027 00008200 DPL=0 LDT
TR =0080 c3010000 00002073 00008b00 DPL=0 TSS32-busy
GDT=     c301e000 000000ff
IDT=     c06ee000 000007ff
CR0=8005003b CR2=00bbe2c0 CR3=355e7000 CR4=000006d0
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??>
?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
?? ?? ?? ?? ?? ??


# virsh qemu-monitor-command bad-domain --hmp x/20i 0xf9258392
0x00000000f9258392:  add    %al,(%eax)
0x00000000f9258394:  add    %al,(%eax)
0x00000000f9258396:  add    %al,(%eax)
0x00000000f9258398:  add    %al,(%eax)
0x00000000f925839a:  add    %al,(%eax)
0x00000000f925839c:  add    %al,(%eax)
0x00000000f925839e:  add    %al,(%eax)
0x00000000f92583a0:  add    %al,(%eax)
0x00000000f92583a2:  add    %al,(%eax)
0x00000000f92583a4:  add    %al,(%eax)
0x00000000f92583a6:  add    %al,(%eax)
0x00000000f92583a8:  add    %al,(%eax)
0x00000000f92583aa:  add    %al,(%eax)
0x00000000f92583ac:  add    %al,(%eax)
0x00000000f92583ae:  add    %al,(%eax)
0x00000000f92583b0:  add    %al,(%eax)
0x00000000f92583b2:  add    %al,(%eax)
0x00000000f92583b4:  add    %al,(%eax)
0x00000000f92583b6:  add    %al,(%eax)
0x00000000f92583b8:  add    %al,(%eax)

When I attempted to resume the VM, either libvirt of qemu stopped me
telling me that resetting the VM is required. But grabbing the memory
at EIP resulted in a different value...

# virsh qemu-monitor-command bad-domain --hmp x/20i 0xf9258392
0x00000000f9258392:  dec    %eax
0x00000000f9258393:  dec    %eax
0x00000000f9258394:  dec    %eax
0x00000000f9258395:  dec    %eax
0x00000000f9258396:  dec    %eax
0x00000000f9258397:  dec    %eax
0x00000000f9258398:  dec    %eax
0x00000000f9258399:  dec    %eax
0x00000000f925839a:  dec    %eax
0x00000000f925839b:  dec    %eax
0x00000000f925839c:  dec    %eax
0x00000000f925839d:  dec    %eax
0x00000000f925839e:  dec    %eax
0x00000000f925839f:  dec    %eax
0x00000000f92583a0:  dec    %eax
0x00000000f92583a1:  dec    %eax
0x00000000f92583a2:  dec    %eax
0x00000000f92583a3:  dec    %eax
0x00000000f92583a4:  dec    %eax
0x00000000f92583a5:  dec    %eax

-- 
Doug Goldstein

Re: Unknown KVM internal error on 3.2.1

[LEo]

me too and help !

#uname -a
Linux rhel64pt 3.14.0-rc3+ #1 SMP Thu Mar 27 17:29:24 CST 2014 x86_64 x86_64 
x86_64 GNU/Linux

# qemu-system-x86_64 -m 2048 -smp 4 -boot order=cd -hda 
/root/kvm_demo/rhel64.img -cdrom /appstore/iso/rhel-server-6.4-x86_64-
dvd.iso -vnc :2
KVM internal error. Suberror: 1
emulation failure
RAX=ffffffff81000122 RBX=0000000001f8b000 RCX=0000000001d55000 
RDX=0000000001000000
RSI=0000000000093780 RDI=0000000001a8c000 RBP=0000000000000000 
RSP=000000000236e140
R8 =0000000001a8c000 R9 =0000000000000001 R10=0000000000000038 
R11=0000000000000038
R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 
R15=0000000000000000
RIP=ffffffff81000122 RFL=00010006 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 0000000000000000 ffffffff 00c00000
CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0000 0000000000000000 ffffffff 00c00000
DS =0000 0000000000000000 ffffffff 00c00000
FS =0000 0000000000000000 ffffffff 00c00000
GS =0000 0000000000000000 ffffffff 00c00000
LDT=0000 0000000000000000 ffffffff 00c00000
TR =0020 0000000000000000 00000fff 00808b00 DPL=0 TSS64-busy
GDT=     00000000004d80d8 00000030
IDT=     0000000000000000 00000000
CR0=80000011 CR2=0000000000000000 CR3=000000909238e090 CR4=000000a0
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 
DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000500
Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> ?? ?? 
?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 
?? ??