Kernel GPF in vmx_save_host_state()

Kernel GPF in vmx_save_host_state()

[Benjamin Gilbert]

I accidentally tried to run a 64-bit guest on a 32-bit host.  Even 
though this isn't supported, it shouldn't crash my kernel.  :-)

CPU: Intel Core 2 Quad
KVM: kernel kvm-84-640-g967f619, userspace kvm-84-304-g2ced1d8.  Also 
occurs with vanilla kvm-84.
Host kernel: Vanilla 2.6.28
Host arch: i386
Guest: Debian Etch x86_64, distributor 2.6.18 kernel
Commandline: qemu-system-x86_64 -hda debian-etch-x86_64 -m 512
Occurs with: -no-kvm-irqchip, -no-kvm-pit
Does not occur with: -no-kvm

As soon as the guest kernel starts, the host produces:

general protection fault: 0000 [#1] PREEMPT SMP 

last sysfs file: /sys/kernel/uevent_seqnum 

Dumping ftrace buffer: 

    (ftrace buffer empty) 

Modules linked in: kvm_intel kvm 

 

Pid: 6570, comm: qemu-system-x86 Not tainted (2.6.28-686 #4) Precision 
WorkStat
EIP: 0060:[<f8cf3d97>] EFLAGS: 00010246 CPU: 2 

EIP is at vmx_save_host_state+0x193/0x1a6 [kvm_intel] 

EAX: 00000100 EBX: 00000000 ECX: c0000080 EDX: 00000000 

ESI: c0000080 EDI: 00000000 EBP: f462ae80 ESP: f462ae58 

  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 

Process qemu-system-x86 (pid: 6570, ti=f462a000 task=f43fe460 
task.ti=f462a000)
Stack: 

  c0406379 f7c5467d 00000100 00000000 ee020020 00000000 00000000 
ee020020
  00000001 00000000 f462aeb0 f7c58b4d f47b3000 ee020020 00000000 
c0406469
  c0403ede 7ffbfeff fffffffe 0000ae80 f43a8730 00000000 f462af18 
f7c5467d
Call Trace: 

  [<c0406379>] ? _spin_unlock+0x2c/0x41 

  [<f7c5467d>] ? kvm_vcpu_ioctl+0xf4/0x40f [kvm] 

  [<f7c58b4d>] ? kvm_arch_vcpu_ioctl_run+0x444/0x918 [kvm] 

  [<c0406469>] ? _spin_unlock_irqrestore+0x59/0x5d 

  [<c0403ede>] ? preempt_schedule+0x30/0x3f 

  [<f7c5467d>] ? kvm_vcpu_ioctl+0xf4/0x40f [kvm] 

  [<c013ae80>] ? up_read+0x1b/0x2f 

  [<c0148144>] ? futex_wake+0xd0/0xdb 

  [<c0148e4c>] ? do_futex+0x81/0x6c9 

  [<f7c54589>] ? kvm_vcpu_ioctl+0x0/0x40f [kvm] 

  [<c018e345>] ? vfs_ioctl+0x27/0x6c 

  [<c018e7ec>] ? do_vfs_ioctl+0x394/0x3d8 

  [<c0184c88>] ? fget_light+0xc8/0xe4 

  [<c018e84c>] ? sys_ioctl+0x1c/0x5f 

  [<c011e06f>] ? sub_preempt_count+0x9d/0xab 

  [<c018e875>] ? sys_ioctl+0x45/0x5f 

  [<c0102e25>] ? sysenter_do_call+0x12/0x35 

Code: ec 81 e1 01 08 00 00 31 db 89 f2 09 ca 89 55 e0 89 f8 09 d8 89 45 
e4 be 8
EIP: [<f8cf3d97>] vmx_save_host_state+0x193/0x1a6 [kvm_intel] SS:ESP 
0068:f462a8
---[ end trace b07f1e77e8b208d3 ]--- 


--Benjamin Gilbert

Re: Kernel GPF in vmx_save_host_state()

[Avi Kivity]

Benjamin Gilbert wrote:
> I accidentally tried to run a 64-bit guest on a 32-bit host.  Even 
> though this isn't supported, it shouldn't crash my kernel.  :-)
>
> CPU: Intel Core 2 Quad
> KVM: kernel kvm-84-640-g967f619, userspace kvm-84-304-g2ced1d8.  Also 
> occurs with vanilla kvm-84.
> Host kernel: Vanilla 2.6.28
> Host arch: i386
> Guest: Debian Etch x86_64, distributor 2.6.18 kernel
> Commandline: qemu-system-x86_64 -hda debian-etch-x86_64 -m 512
> Occurs with: -no-kvm-irqchip, -no-kvm-pit
> Does not occur with: -no-kvm
>
> As soon as the guest kernel starts, the host produces:
>
> general protection fault: 0000 [#1] PREEMPT SMP
> last sysfs file: /sys/kernel/uevent_seqnum
> Dumping ftrace buffer:
>    (ftrace buffer empty)
> Modules linked in: kvm_intel kvm
>
>
> Pid: 6570, comm: qemu-system-x86 Not tainted (2.6.28-686 #4) Precision 
> WorkStat
> EIP: 0060:[<f8cf3d97>] EFLAGS: 00010246 CPU: 2
> EIP is at vmx_save_host_state+0x193/0x1a6 [kvm_intel]
> EAX: 00000100 EBX: 00000000 ECX: c0000080 EDX: 00000000
>

 From ECX, it looks like it's trying to access EFER and enable LM.

Please provide your /proc/cpuinfo.


> Code: ec 81 e1 01 08 00 00 31 db 89 f2 09 ca 89 55 e0 89 f8 09 d8 89 
> 45 e4 be 8

And this line, in full.

-- 
error compiling committee.c: too many arguments to function

Re: Kernel GPF in vmx_save_host_state()

[Benjamin Gilbert]

Avi Kivity wrote:
>  From ECX, it looks like it's trying to access EFER and enable LM.
> 
> Please provide your /proc/cpuinfo.

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Core(TM)2 Quad CPU           @ 2.66GHz
stepping	: 7
cpu MHz		: 2660.345
cache size	: 4096 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm 
constant_tsc arch_perfmon pebs bts pni dtes64 monitor ds_cpl vmx est tm2 
ssse3 cx16 xtpr pdcm lahf_lm tpr_shadow
bogomips	: 5320.69
clflush size	: 64
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Core(TM)2 Quad CPU           @ 2.66GHz
stepping	: 7
cpu MHz		: 2660.345
cache size	: 4096 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 1
initial apicid	: 1
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm 
constant_tsc arch_perfmon pebs bts pni dtes64 monitor ds_cpl vmx est tm2 
ssse3 cx16 xtpr pdcm lahf_lm tpr_shadow
bogomips	: 5320.23
clflush size	: 64
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Core(TM)2 Quad CPU           @ 2.66GHz
stepping	: 7
cpu MHz		: 2660.345
cache size	: 4096 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm 
constant_tsc arch_perfmon pebs bts pni dtes64 monitor ds_cpl vmx est tm2 
ssse3 cx16 xtpr pdcm lahf_lm tpr_shadow
bogomips	: 5320.28
clflush size	: 64
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Core(TM)2 Quad CPU           @ 2.66GHz
stepping	: 7
cpu MHz		: 2660.345
cache size	: 4096 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 3
initial apicid	: 3
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm 
constant_tsc arch_perfmon pebs bts pni dtes64 monitor ds_cpl vmx est tm2 
ssse3 cx16 xtpr pdcm lahf_lm tpr_shadow
bogomips	: 5320.24
clflush size	: 64
power management:

>> Code: ec 81 e1 01 08 00 00 31 db 89 f2 09 ca 89 55 e0 89 f8 09 d8 89 
>> 45 e4 be 8
> 
> And this line, in full.

Whoops.

Code: ec 81 e1 01 08 00 00 31 db 89 f2 09 ca 89 55 e0 89 f8 09 d8 89 45 
e4 be 80 00 00 c0 8b 55 e0 8b 4d e4 89 ca 31 c9 89 f1 8b 45 e0 <0f> 30 
8b 5d e8 ff 83 ec 00 00 00 83 c4 1c 5b 5e 5f 5d c3 55 89

--Benjamin Gilbert

Re: Kernel GPF in vmx_save_host_state()

[Amit Shah]

On (Tue) Mar 17 2009 [19:24:44], Benjamin Gilbert wrote:
> I accidentally tried to run a 64-bit guest on a 32-bit host.  Even  
> though this isn't supported, it shouldn't crash my kernel.  :-)

Glauber had a patch that fixes this; it needs a refresh now though:

http://marc.info/?l=qemu-devel&m=123324996718919&w=2

Amit

Re: Kernel GPF in vmx_save_host_state()

[Avi Kivity]

Benjamin Gilbert wrote:
>
> Code: ec 81 e1 01 08 00 00 31 db 89 f2 09 ca 89 55 e0 89 f8 09 d8 89 
> 45 e4 be 80 00 00 c0 8b 55 e0 8b 4d e4 89 ca 31 c9 89 f1 8b 45 e0 <0f> 
> 30 8b 5d e8 ff 83 ec 00 00 00 83 c4 1c 5b 5e 5f 5d c3 55 89
>

Well, that's certainly the wrmsr instruction.  But I don't see how this 
can happen.

Can you patch set_efer() in x86.c to print the value of the efer 
argument and of efer_reserved_bits?

-- 
error compiling committee.c: too many arguments to function

Re: Kernel GPF in vmx_save_host_state()

[Avi Kivity]

Amit Shah wrote:
> On (Tue) Mar 17 2009 [19:24:44], Benjamin Gilbert wrote:
>   
>> I accidentally tried to run a 64-bit guest on a 32-bit host.  Even  
>> though this isn't supported, it shouldn't crash my kernel.  :-)
>>     
>
> Glauber had a patch that fixes this; it needs a refresh now though:
>
> http://marc.info/?l=qemu-devel&m=123324996718919&w=2
>   

That won't fix the kernel crash.


-- 
error compiling committee.c: too many arguments to function

Re: Kernel GPF in vmx_save_host_state()

[Amit Shah]

On (Thu) Mar 19 2009 [11:55:57], Avi Kivity wrote:
> Amit Shah wrote:
>> On (Tue) Mar 17 2009 [19:24:44], Benjamin Gilbert wrote:
>>   
>>> I accidentally tried to run a 64-bit guest on a 32-bit host.  Even   
>>> though this isn't supported, it shouldn't crash my kernel.  :-)
>>>     
>>
>> Glauber had a patch that fixes this; it needs a refresh now though:
>>
>> http://marc.info/?l=qemu-devel&m=123324996718919&w=2
>>   
>
> That won't fix the kernel crash.

Right.

What action should the kernel take, though? If lm is advertised and we
don't let the guest enter long mode -- exit to userspace and close the
VM?

Re: Kernel GPF in vmx_save_host_state()

[Avi Kivity]

Amit Shah wrote:
> What action should the kernel take, though? If lm is advertised and we
> don't let the guest enter long mode -- exit to userspace and close the
> VM?
>   

Since it should never happen, it doesn't really matter.  #GP will be the 
natural action (setting reserved bit); userspace lied to the guest, and 
we can't recover from that.

-- 
error compiling committee.c: too many arguments to function

Re: Kernel GPF in vmx_save_host_state()

[Benjamin Gilbert]

Avi Kivity wrote:
> Well, that's certainly the wrmsr instruction.  But I don't see how this 
> can happen.
> 
> Can you patch set_efer() in x86.c to print the value of the efer 
> argument and of efer_reserved_bits?

Yes, but apparently set_efer() is never called.  To verify, I patched 
kvm_set_msr_common() to print the msr parameter:

kvm_set_msr_common: 0xc0010117
kvm_set_msr_common: 0x250
kvm_set_msr_common: 0x258
kvm_set_msr_common: 0x259
kvm_set_msr_common: 0x268
kvm_set_msr_common: 0x269
kvm_set_msr_common: 0x26a
kvm_set_msr_common: 0x26b
kvm_set_msr_common: 0x26c
kvm_set_msr_common: 0x26d
kvm_set_msr_common: 0x26e
kvm_set_msr_common: 0x26f
kvm_set_msr_common: 0x200
kvm_set_msr_common: 0x201
kvm_set_msr_common: 0x2ff
general protection fault: 0000 [#1] PREEMPT SMP
[...]

--Benjamin Gilbert

Re: Kernel GPF in vmx_save_host_state()

[Avi Kivity]

Benjamin Gilbert wrote:
> Avi Kivity wrote:
>> Well, that's certainly the wrmsr instruction.  But I don't see how 
>> this can happen.
>>
>> Can you patch set_efer() in x86.c to print the value of the efer 
>> argument and of efer_reserved_bits?
>
> Yes, but apparently set_efer() is never called.  To verify, I patched 
> kvm_set_msr_common() to print the msr parameter:
>

On 32-bit, we might actually reach the default: label of the switch in 
vmx_set_msr().  Can you add a printk() there? print both msr_index, and, 
if msr is not NULL, msr->index and msr->data.

-- 
error compiling committee.c: too many arguments to function

Re: Kernel GPF in vmx_save_host_state()

[Benjamin Gilbert]

Avi Kivity wrote:
> On 32-bit, we might actually reach the default: label of the switch in 
> vmx_set_msr().  Can you add a printk() there? print both msr_index, and, 
> if msr is not NULL, msr->index and msr->data.

Sure:

vmx_set_msr: msr_index 0xc0000081 msr->index 0xc0000081 msr->data 0x0
vmx_set_msr: msr_index 0xc0010117
vmx_set_msr: msr_index 0x250
vmx_set_msr: msr_index 0x258
vmx_set_msr: msr_index 0x259
vmx_set_msr: msr_index 0x268
vmx_set_msr: msr_index 0x269
vmx_set_msr: msr_index 0x26a
vmx_set_msr: msr_index 0x26b
vmx_set_msr: msr_index 0x26c
vmx_set_msr: msr_index 0x26d
vmx_set_msr: msr_index 0x26e
vmx_set_msr: msr_index 0x26f
vmx_set_msr: msr_index 0x200
vmx_set_msr: msr_index 0x201
vmx_set_msr: msr_index 0x2ff
vmx_set_msr: msr_index 0xc0000080 msr->index 0xc0000080 msr->data 0x100
general protection fault: 0000 [#1] PREEMPT SMP
[...]

The printk is after msr->data is set to the value of the data parameter.

--Benjamin Gilbert

Re: Kernel GPF in vmx_save_host_state()

[Avi Kivity]

Benjamin Gilbert wrote:
> vmx_set_msr: msr_index 0xc0000080 msr->index 0xc0000080 msr->data 0x100

How did that get in there?!

Please add a dump_stack() after that printk().


-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: Kernel GPF in vmx_save_host_state()

[Benjamin Gilbert]

Avi Kivity wrote:
> Benjamin Gilbert wrote:
>> vmx_set_msr: msr_index 0xc0000080 msr->index 0xc0000080 msr->data 0x100
> 
> How did that get in there?!
> 
> Please add a dump_stack() after that printk().

Pid: 2381, comm: qemu-system-x86 Not tainted 2.6.28-686 #4
Call Trace:
  [<f8cf2fdc>] vmx_set_msr+0x150/0x178 [kvm_intel]
  [<f8cf325a>] handle_wrmsr+0x71/0x9d [kvm_intel]
  [<f8cf4fb0>] kvm_handle_exit+0x1c8/0x1e5 [kvm_intel]
  [<f7c58e34>] kvm_arch_vcpu_ioctl_run+0x6f2/0x918 [kvm]
  [<c0406469>] ? _spin_unlock_irqrestore+0x59/0x5d
  [<c0403ede>] ? preempt_schedule+0x30/0x3f
  [<f7c5467d>] kvm_vcpu_ioctl+0xf4/0x40f [kvm]
  [<c013ae80>] ? up_read+0x1b/0x2f
  [<c0148144>] ? futex_wake+0xd0/0xdb
  [<c0148e4c>] ? do_futex+0x81/0x6c9
  [<f7c54589>] ? kvm_vcpu_ioctl+0x0/0x40f [kvm]
  [<c018e345>] vfs_ioctl+0x27/0x6c
  [<c018e7ec>] do_vfs_ioctl+0x394/0x3d8
  [<c0184c88>] ? fget_light+0xc8/0xe4
  [<c018e84c>] ? sys_ioctl+0x1c/0x5f
  [<c011e06f>] ? sub_preempt_count+0x9d/0xab
  [<c018e875>] sys_ioctl+0x45/0x5f
  [<c0102e25>] sysenter_do_call+0x12/0x35

--Benjamin Gilbert

Re: Kernel GPF in vmx_save_host_state()

[Avi Kivity]

[-- Attachment #1: Type: text/plain, Size: 544 bytes --]

Benjamin Gilbert wrote:
>>> vmx_set_msr: msr_index 0xc0000080 msr->index 0xc0000080 msr->data 0x100
>>
>> How did that get in there?!
>>
>> Please add a dump_stack() after that printk().
>
> Pid: 2381, comm: qemu-system-x86 Not tainted 2.6.28-686 #4
> Call Trace:
>  [<f8cf2fdc>] vmx_set_msr+0x150/0x178 [kvm_intel]
>  [<f8cf325a>] handle_wrmsr+0x71/0x9d [kvm_intel]

Duh, I noted this hole in a previous email.

Attached patch should fix.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.


[-- Attachment #2: dont-allow-uninhibited-efer-access-on-i386.patch --]
[-- Type: text/x-patch, Size: 485 bytes --]

diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
index 88ef094..da6461d 100644
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -942,11 +942,11 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 data)
 	int ret = 0;
 
 	switch (msr_index) {
-#ifdef CONFIG_X86_64
 	case MSR_EFER:
 		vmx_load_host_state(vmx);
 		ret = kvm_set_msr_common(vcpu, msr_index, data);
 		break;
+#ifdef CONFIG_X86_64
 	case MSR_FS_BASE:
 		vmcs_writel(GUEST_FS_BASE, data);
 		break;

Re: Kernel GPF in vmx_save_host_state()

[Benjamin Gilbert]

Avi Kivity wrote:
> Duh, I noted this hole in a previous email.
> 
> Attached patch should fix.

It does, thanks.

--Benjamin Gilbert