kvm_amd BUG: unable to handle kernel NULL pointer dereference at 00000014

kvm_amd BUG: unable to handle kernel NULL pointer dereference at 00000014

[IVAN ANGELOV]

Hello,
This provided dmesg message and kernel behavior appear when trying to
run qemu-kvm with kvm_amd module. Without kvm_amd qemu-kvm runs fine
but a slower. I managed to see that this happens with 2.6.38-rc6 ,
2.6.38-rc7 vanilla kernels compiled using kernel-package. OS ubuntu
natty. Using the standard toolchain and gcc from ubuntu: gcc version
4.5.2 (Ubuntu/Linaro 4.5.2-3ubuntu3)
I reverted to 2.6.37.2 linux kernel, compiled with the very same tools
and machine I use qemu-kvm with kvm_amd module without any problems.
If I can provide some extra info about that please let me know.
This issue also appears with the kernel provided by the Ubuntu
distribution: 2.6.38-5-generic-pae #32-Ubuntu SMP Tue Feb 22 17:48:56
UTC 2011 i686 athlon i386 GNU/Linux , I suspect it is somehow related
to the 2.6.38 kernel series.
cpuinfo - phenom ii x4 955 mildly overclocked - 4 fields like this is
the whole cpuinfo.
cat /proc/cpuinfo
processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 16
model		: 4
model name	: AMD Phenom(tm) II X4 955 Processor
stepping	: 3
cpu MHz		: 3680.416
cache size	: 512 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt
pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc nonstop_tsc extd_apicid
pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm
sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt npt lbrv svm_lock
nrip_save
bogomips	: 7360.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

This is what dmesg shows:

101.358689] BUG: unable to handle kernel NULL pointer dereference at 00000014
[  101.358785] IP: [<f89d4ad2>] x86_decode_insn+0x12/0x9b0 [kvm]
[  101.358868] *pdpt = 0000000032008001 *pde = 0000000000000000
[  101.359005] Oops: 0000 [#1] SMP
[  101.359081] last sysfs file:
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
[  101.359178] Modules linked in: kvm_amd kvm binfmt_misc
snd_hda_codec_via snd_hda_intel snd_hda_codec snd_hwdep snd_pcm
snd_seq_midi snd_rawmidi snd_seq_midi_event radeon ttm snd_seq
snd_timer snd_seq_device asus_atk0110 drm_kms_helper sp5100_tco drm
snd i2c_algo_bit joydev soundcore snd_page_alloc i2c_piix4 k10temp lp
shpchp parport usbhid hid ahci pata_atiixp libahci r8169
[  101.359766]
[  101.359786] Pid: 1109, comm: qemu-system-x86 Not tainted
2.6.38-5-generic-pae #32-Ubuntu System manufacturer System Product
Name/M4A88T-M
[  101.359974] EIP: 0060:[<f89d4ad2>] EFLAGS: 00210292 CPU: 2
[  101.360084] EIP is at x86_decode_insn+0x12/0x9b0 [kvm]
[  101.360166] EAX: eec79454 EBX: eec79454 ECX: 00000000 EDX: 00000000
[  101.360240] ESI: f89e76e0 EDI: 00000000 EBP: edca3de0 ESP: edca3d8c
[  101.360316]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[  101.360395] Process qemu-system-x86 (pid: 1109, ti=edca2000
task=f2121940 task.ti=edca2000)
[  101.360499] Stack:
[  101.360536]  edca3dd0 edca3dcc edca3dc0 f89b6969 000f0000 00000000
0000ffff 010bf000
[  101.360671]  00010000 00000000 f8657020 eec78000 00000000 edca3de0
f89bf5a5 eec79484
[  101.360804]  00000000 00000000 eec78000 eec79454 00000000 edca3e00
f89bfa22 f8657020
[  101.360934] Call Trace:
[  101.360973]  [<f89b6969>] ? kvm_get_cs_db_l_bits+0x29/0x50 [kvm]
[  101.361063]  [<f89bf5a5>] ? init_emulate_ctxt+0x45/0x130 [kvm]
[  101.361152]  [<f89bfa22>] x86_emulate_instruction+0x162/0x450 [kvm]
[  101.361229]  [<f864df53>] emulate_on_interception+0x23/0x30 [kvm_amd]
[  101.361318]  [<f8650c9f>] cr_interception+0x7f/0x210 [kvm_amd]
[  101.361392]  [<f865313f>] handle_exit+0x15f/0x405 [kvm_amd]
[  101.361458]  [<c10eb6d2>] ? unlock_page+0x42/0x50
[  101.361513]  [<c1107bd8>] ? __do_fault+0x418/0x520
[  101.361575]  [<f89be5fd>] ? kvm_get_cr8+0x2d/0x30 [kvm]
[  101.361636]  [<f864e296>] ? svm_vcpu_run+0x1b6/0x370 [kvm_amd]
[  101.361709]  [<f89c4692>] vcpu_enter_guest+0x1a2/0x440 [kvm]
[  101.361780]  [<f89c4ebf>] __vcpu_run+0x12f/0x280 [kvm]
[  101.361842]  [<c106992a>] ? sigprocmask+0x7a/0x100
[  101.361904]  [<f89c50d1>] kvm_arch_vcpu_ioctl_run+0xc1/0x1b0 [kvm]
[  101.361980]  [<f89b0fea>] kvm_vcpu_ioctl+0x49a/0x5f0 [kvm]
[  101.362046]  [<c1287494>] ? _copy_from_user+0x44/0x70
[  101.362104]  [<c106bfb2>] ? sys_rt_sigtimedwait+0x102/0x210
[  101.362170]  [<c107595c>] ? common_timer_set+0xfc/0x160
[  101.362232]  [<c153151f>] ? _raw_spin_lock_irqsave+0x2f/0x50
[  101.362302]  [<f89b0b50>] ? kvm_vcpu_ioctl+0x0/0x5f0 [kvm]
[  101.362367]  [<c1141deb>] do_vfs_ioctl+0x7b/0x2e0
[  101.362421]  [<c1287502>] ? copy_to_user+0x42/0x60
[  101.362477]  [<c11420d7>] sys_ioctl+0x87/0x90
[  101.362528]  [<c100ab9f>] sysenter_do_call+0x12/0x28
[  101.362587] Code: 30 89 c2 e9 63 ff ff ff 90 8b 4d e0 89 f2 ff 51
30 89 c2 eb c1 8d 74 26 00 55 89 e5 57 56 53 83 ec 48 3e 8d 74 26 00
89 c3 8b 33 <65> a1 14 00 00 00 89 45 f0 31 c0 8d 45 d0 89 45 c8 31 c0
89 75
[  101.362628] EIP: [<f89d4ad2>] x86_decode_insn+0x12/0x9b0 [kvm]
SS:ESP 0068:edca3d8c
[  101.362628] CR2: 0000000000000014
[  101.394484] ---[ end trace 4bab68acb7870e5d ]---
[  119.293605] usb 4-2: USB disconnect, address 2
[  120.084016] usb 4-2: new low speed USB device using ohci_hcd and address 3
[  120.259034] input: SIGMACH1P U+P Mouse as
/devices/pci0000:00/0000:00:12.1/usb4/4-2/4-2:1.0/input/input5
[  120.259103] generic-usb 0003:1C4F:0003.0003: input,hidraw1: USB HID
v1.10 Mouse [SIGMACH1P U+P Mouse] on usb-0000:00:12.1-2/input0

Best Regards, Ivan

Re: kvm_amd BUG: unable to handle kernel NULL pointer dereference at 00000014

[Avi Kivity]

On 03/04/2011 12:34 AM, IVAN ANGELOV wrote:
> Hello,
> This provided dmesg message and kernel behavior appear when trying to
> run qemu-kvm with kvm_amd module. Without kvm_amd qemu-kvm runs fine
> but a slower. I managed to see that this happens with 2.6.38-rc6 ,
> 2.6.38-rc7 vanilla kernels compiled using kernel-package. OS ubuntu
> natty. Using the standard toolchain and gcc from ubuntu: gcc version
> 4.5.2 (Ubuntu/Linaro 4.5.2-3ubuntu3)
> I reverted to 2.6.37.2 linux kernel, compiled with the very same tools
> and machine I use qemu-kvm with kvm_amd module without any problems.
> If I can provide some extra info about that please let me know.
> This issue also appears with the kernel provided by the Ubuntu
> distribution: 2.6.38-5-generic-pae #32-Ubuntu SMP Tue Feb 22 17:48:56
> UTC 2011 i686 athlon i386 GNU/Linux , I suspect it is somehow related
> to the 2.6.38 kernel series.
> cpuinfo - phenom ii x4 955 mildly overclocked - 4 fields like this is
> the whole cpuinfo.


    c:    65 a1 14 00 00 00        mov    %gs:0x14,%eax

faults, gsbase == NULL.

But arch/x86/include/asm/percpu.h says:

#ifdef CONFIG_X86_64
#define __percpu_seg        gs
#define __percpu_mov_op        movq
#else
#define __percpu_seg        fs
#define __percpu_mov_op        movl
#endif


So we should be using %fs, not %gs.

-- 
error compiling committee.c: too many arguments to functionb

Re: kvm_amd BUG: unable to handle kernel NULL pointer dereference at 00000014

[Roedel, Joerg]

(CC'ing x86 Maintainers)

On Sun, Mar 06, 2011 at 05:23:12AM -0500, Avi Kivity wrote:
> On 03/04/2011 12:34 AM, IVAN ANGELOV wrote:
> > Hello,
> > This provided dmesg message and kernel behavior appear when trying to
> > run qemu-kvm with kvm_amd module. Without kvm_amd qemu-kvm runs fine
> > but a slower. I managed to see that this happens with 2.6.38-rc6 ,
> > 2.6.38-rc7 vanilla kernels compiled using kernel-package. OS ubuntu
> > natty. Using the standard toolchain and gcc from ubuntu: gcc version
> > 4.5.2 (Ubuntu/Linaro 4.5.2-3ubuntu3)
> > I reverted to 2.6.37.2 linux kernel, compiled with the very same tools
> > and machine I use qemu-kvm with kvm_amd module without any problems.
> > If I can provide some extra info about that please let me know.
> > This issue also appears with the kernel provided by the Ubuntu
> > distribution: 2.6.38-5-generic-pae #32-Ubuntu SMP Tue Feb 22 17:48:56
> > UTC 2011 i686 athlon i386 GNU/Linux , I suspect it is somehow related
> > to the 2.6.38 kernel series.
> > cpuinfo - phenom ii x4 955 mildly overclocked - 4 fields like this is
> > the whole cpuinfo.
> 
> 
>     c:    65 a1 14 00 00 00        mov    %gs:0x14,%eax
> 
> faults, gsbase == NULL.
> 
> But arch/x86/include/asm/percpu.h says:
> 
> #ifdef CONFIG_X86_64
> #define __percpu_seg        gs
> #define __percpu_mov_op        movq
> #else
> #define __percpu_seg        fs
> #define __percpu_mov_op        movl
> #endif
> 
> 
> So we should be using %fs, not %gs.

There is no access to per_cpu variables at the start of x86_decode_insn.
I did a bit of investigation and it turns out that the faulting
instruction is inserted into the code by the gcc because the
CONFIG_CC_STACKPROTECTOR is enabled.
The user tested this is Ubuntu 11.04 alpha-something i386 and this
distro uses gcc 4.5.2. So CC_STACKPROTECTOR seems to be harmful with
this gcc version but I am not sure whether this counts as a gcc bug.

	Joerg

-- 
AMD Operating System Research Center

Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632

Re: kvm_amd BUG: unable to handle kernel NULL pointer dereference at 00000014

[Avi Kivity]

On 03/07/2011 02:11 PM, Roedel, Joerg wrote:
> (CC'ing x86 Maintainers)
>
> On Sun, Mar 06, 2011 at 05:23:12AM -0500, Avi Kivity wrote:
> >  On 03/04/2011 12:34 AM, IVAN ANGELOV wrote:
> >  >  Hello,
> >  >  This provided dmesg message and kernel behavior appear when trying to
> >  >  run qemu-kvm with kvm_amd module. Without kvm_amd qemu-kvm runs fine
> >  >  but a slower. I managed to see that this happens with 2.6.38-rc6 ,
> >  >  2.6.38-rc7 vanilla kernels compiled using kernel-package. OS ubuntu
> >  >  natty. Using the standard toolchain and gcc from ubuntu: gcc version
> >  >  4.5.2 (Ubuntu/Linaro 4.5.2-3ubuntu3)
> >  >  I reverted to 2.6.37.2 linux kernel, compiled with the very same tools
> >  >  and machine I use qemu-kvm with kvm_amd module without any problems.
> >  >  If I can provide some extra info about that please let me know.
> >  >  This issue also appears with the kernel provided by the Ubuntu
> >  >  distribution: 2.6.38-5-generic-pae #32-Ubuntu SMP Tue Feb 22 17:48:56
> >  >  UTC 2011 i686 athlon i386 GNU/Linux , I suspect it is somehow related
> >  >  to the 2.6.38 kernel series.
> >  >  cpuinfo - phenom ii x4 955 mildly overclocked - 4 fields like this is
> >  >  the whole cpuinfo.
> >
> >
> >      c:    65 a1 14 00 00 00        mov    %gs:0x14,%eax
> >
> >  faults, gsbase == NULL.
> >
> >  But arch/x86/include/asm/percpu.h says:
> >
> >  #ifdef CONFIG_X86_64
> >  #define __percpu_seg        gs
> >  #define __percpu_mov_op        movq
> >  #else
> >  #define __percpu_seg        fs
> >  #define __percpu_mov_op        movl
> >  #endif
> >
> >
> >  So we should be using %fs, not %gs.
>
> There is no access to per_cpu variables at the start of x86_decode_insn.
> I did a bit of investigation and it turns out that the faulting
> instruction is inserted into the code by the gcc because the
> CONFIG_CC_STACKPROTECTOR is enabled.
> The user tested this is Ubuntu 11.04 alpha-something i386 and this
> distro uses gcc 4.5.2. So CC_STACKPROTECTOR seems to be harmful with
> this gcc version but I am not sure whether this counts as a gcc bug.

Ah, looks like %gs is the expected segment on i386 with 
-fstack-protector.  So we must disable lazy gs reload in that scenario.

-- 
error compiling committee.c: too many arguments to function

Re: kvm_amd BUG: unable to handle kernel NULL pointer dereference at 00000014

[Roedel, Joerg]

On Mon, Mar 07, 2011 at 07:50:14AM -0500, Avi Kivity wrote:
> On 03/07/2011 02:11 PM, Roedel, Joerg wrote:

> > There is no access to per_cpu variables at the start of x86_decode_insn.
> > I did a bit of investigation and it turns out that the faulting
> > instruction is inserted into the code by the gcc because the
> > CONFIG_CC_STACKPROTECTOR is enabled.
> > The user tested this is Ubuntu 11.04 alpha-something i386 and this
> > distro uses gcc 4.5.2. So CC_STACKPROTECTOR seems to be harmful with
> > this gcc version but I am not sure whether this counts as a gcc bug.
> 
> Ah, looks like %gs is the expected segment on i386 with 
> -fstack-protector.  So we must disable lazy gs reload in that scenario.

According to the comments in stackprotector.h its the same on amd64 (the
difference is that gcc expects the canary value at a different offset
from %gs).
So we should probably unlazy %gs reload alltogether.

	Joerg

-- 
AMD Operating System Research Center

Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632

Re: kvm_amd BUG: unable to handle kernel NULL pointer dereference at 00000014

[Avi Kivity]

On 03/07/2011 03:16 PM, Roedel, Joerg wrote:
> On Mon, Mar 07, 2011 at 07:50:14AM -0500, Avi Kivity wrote:
> >  On 03/07/2011 02:11 PM, Roedel, Joerg wrote:
>
> >  >  There is no access to per_cpu variables at the start of x86_decode_insn.
> >  >  I did a bit of investigation and it turns out that the faulting
> >  >  instruction is inserted into the code by the gcc because the
> >  >  CONFIG_CC_STACKPROTECTOR is enabled.
> >  >  The user tested this is Ubuntu 11.04 alpha-something i386 and this
> >  >  distro uses gcc 4.5.2. So CC_STACKPROTECTOR seems to be harmful with
> >  >  this gcc version but I am not sure whether this counts as a gcc bug.
> >
> >  Ah, looks like %gs is the expected segment on i386 with
> >  -fstack-protector.  So we must disable lazy gs reload in that scenario.
>
> According to the comments in stackprotector.h its the same on amd64 (the
> difference is that gcc expects the canary value at a different offset
> from %gs).
> So we should probably unlazy %gs reload alltogether.

On x86_64 we don't do lazy %gs reload (lazy %fs instead), so it should 
work as is.

-- 
error compiling committee.c: too many arguments to function

Re: kvm_amd BUG: unable to handle kernel NULL pointer dereference at 00000014

[Roedel, Joerg]

On Mon, Mar 07, 2011 at 08:20:42AM -0500, Avi Kivity wrote:
> On 03/07/2011 03:16 PM, Roedel, Joerg wrote:

> > According to the comments in stackprotector.h its the same on amd64 (the
> > difference is that gcc expects the canary value at a different offset
> > from %gs).
> > So we should probably unlazy %gs reload alltogether.
> 
> On x86_64 we don't do lazy %gs reload (lazy %fs instead), so it should 
> work as is.

Right. I mixed that up with the lazy KERNEL_GS_BASE switching on amd64.

-- 
AMD Operating System Research Center

Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632