applications running in KVM crashing?

applications running in KVM crashing?

[Nikola Ciprich]

Hi,
on two of my KVM machines, some applications started to crash.
I'm not sure, but I think it might be related to some host update
I've made recently. I also can't find any difference to guests, on which
I don't see any such problems.
It's usually sendmail or postgres, crashing with following dmesg message:
postmaster[4347] trap invalid opcode ip:7f8f10fac273 sp:7fff947ecac8 error:0 in libpthread-2.5.so[7f8f10fa2000+16000]
Is it possible this might be KVM related, or should I look elsewhere?
I'll enable core dumps and post more detailed backtrace once I have it...
Thanks a lot in advance!
BR
nik

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

Re: applications running in KVM crashing?

[Avi Kivity]

On 06/09/2010 07:14 AM, Nikola Ciprich wrote:
> Hi,
> on two of my KVM machines, some applications started to crash.
> I'm not sure, but I think it might be related to some host update
> I've made recently. I also can't find any difference to guests, on which
> I don't see any such problems.
> It's usually sendmail or postgres, crashing with following dmesg message:
> postmaster[4347] trap invalid opcode ip:7f8f10fac273 sp:7fff947ecac8 error:0 in libpthread-2.5.so[7f8f10fa2000+16000]
> Is it possible this might be KVM related, or should I look elsewhere?
> I'll enable core dumps and post more detailed backtrace once I have it...
>    

backtrace / disassembly of the failing instructions will help.  Do the 
failures always occur in the same ip or random ips?

-- 
error compiling committee.c: too many arguments to function

Re: applications running in KVM crashing?

[Nikola Ciprich]

> backtrace / disassembly of the failing instructions will help.  Do
> the failures always occur in the same ip or random ips?
Hi Avi,
OK, I'll post is once I have it, but it can take few days...
ips seem to remain the same, for same process (even across reboots):

May 25 02:23:07 sql2 [444620.285403] sendmail[3550] trap invalid opcode ip:7f1d83d22214 sp:7fff78a0aa60 error:0 in libc-2.5.so[7f1d83c88000+14e000]
May 28 04:56:47 sql2 [713039.887997] postmaster[3530] trap invalid opcode ip:7f5fa99d5273 sp:7fff76e6c398 error:0 in libpthread-2.5.so[7f5fa99cb000+16000]
May 28 12:50:11 sql2 [741443.509959] postmaster[9745] trap invalid opcode ip:7f5a14adc273 sp:7fff96fa6d58 error:0 in libpthread-2.5.so[7f5a14ad2000+16000]
May 28 15:28:38 sql2 [750950.480189] postmaster[6452] trap invalid opcode ip:7f5a14adc273 sp:7fff96fa6cd8 error:0 in libpthread-2.5.so[7f5a14ad2000+16000]
Jun  8 11:57:44 sql2 [482644.254679] postmaster[4347] trap invalid opcode ip:7f8f10fac273 sp:7fff947ecac8 error:0 in libpthread-2.5.so[7f8f10fa2000+16000]


> 
> -- 
> error compiling committee.c: too many arguments to function
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

Re: applications running in KVM crashing?

[Nikola Ciprich]

On Wed, Jun 09, 2010 at 02:18:00PM +0200, Nikola Ciprich wrote:
> > backtrace / disassembly of the failing instructions will help.  Do
> > the failures always occur in the same ip or random ips?

Hello Avi,
FINALLY I got another crash, so I can provide proper backtrace.
[1170486.891280] postmaster[3594] trap invalid opcode ip:7f6391472273 sp:7fff48ce2158 error:0 in libpthread-2.5.so[7f6391468000+16000]

bt full:

Core was generated by `/usr/bin/postmaster -p 5432 -D /home/pgsql/data'.
Program terminated with signal 4, Illegal instruction.
#0  0x00007f6391472273 in pthread_rwlock_rdlock () from /lib64/libpthread.so.0
#0  0x00007f6391472273 in pthread_rwlock_rdlock () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f6392bde166 in __dcigettext (domainname=0x0, msgid1=0x7a5bf4 "server process", msgid2=0x0, plural=0, n=0, category=5)
    at dcigettext.c:452
        domain = <value optimized out>
        binding = <value optimized out>
        categoryname = <value optimized out>
        categoryvalue = <value optimized out>
        dirname = <value optimized out>
        xdomainname = <value optimized out>
        single_locale = <value optimized out>
        retval = 0x0
        retlen = <value optimized out>
        saved_errno = <value optimized out>
        search = <value optimized out>
        foundp = 0x0
        msgid_len = 6836378
        localename = <value optimized out>
        domainname_len = <value optimized out>
        tree_lock = {__data = {__lock = 0, __nr_readers = 0, __readers_wakeup = 0, __writer_wakeup = 0, __nr_readers_queued = 0,
            __nr_writers_queued = 0, __writer = 0, __shared = 0, __pad1 = 0, __pad2 = 0, __flags = 0},
          __size = '\000' <repeats 55 times>, __align = 0}
#2  0x00000000005b2bbc in CleanupBackend (postgres_signal_arg=<value optimized out>) at postmaster.c:2490
No locals.
#3  reaper (postgres_signal_arg=<value optimized out>) at postmaster.c:2463
        save_errno = 4
        pid = 3545
        exitstatus = 0
        status = 0
        __func__ = "reaper"
#4  <signal handler called>
No symbol table info available.
#5  0x00007f6392c7ffa3 in __select_nocancel () from /lib64/libc.so.6
No symbol table info available.
#6  0x00000000005b02f7 in ServerLoop () at postmaster.c:1347
        timeout = {tv_sec = 59, tv_usec = 32718}
        rmask = {fds_bits = {56, 0 <repeats 15 times>}}
        selres = <value optimized out>
        readmask = {fds_bits = {56, 0 <repeats 15 times>}}
        nSockets = 6
        now = 1279098771
        last_touch_time = 1279097691
        __func__ = "ServerLoop"
#7  0x00000000005b18ac in PostmasterMain (argc=5, argv=0x1bac9d0) at postmaster.c:1040
        opt = <value optimized out>
        status = <value optimized out>
        userDoption = <value optimized out>
        __func__ = "PostmasterMain"
#8  0x000000000055f56e in main (argc=5, argv=<value optimized out>) at main.c:188
No locals.


I'm not sure if this is exactly what You need, so please feel free to demand further information.
I've got corefile here, so hopefully I can dig more from gdb if You need (and tell me what to do).
have a nice day!
nik

> 
> > 
> > -- 
> > error compiling committee.c: too many arguments to function
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe kvm" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> -- 
> -------------------------------------
> Ing. Nikola CIPRICH
> LinuxBox.cz, s.r.o.
> 28. rijna 168, 709 01 Ostrava
> 
> tel.:   +420 596 603 142
> fax:    +420 596 621 273
> mobil:  +420 777 093 799
> www.linuxbox.cz
> 
> mobil servis: +420 737 238 656
> email servis: servis@linuxbox.cz
> -------------------------------------
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

Re: applications running in KVM crashing?

[Avi Kivity]

On 07/14/2010 12:45 PM, Nikola Ciprich wrote:
> On Wed, Jun 09, 2010 at 02:18:00PM +0200, Nikola Ciprich wrote:
>    
>>> backtrace / disassembly of the failing instructions will help.  Do
>>> the failures always occur in the same ip or random ips?
>>>        
> Hello Avi,
> FINALLY I got another crash, so I can provide proper backtrace.
> [1170486.891280] postmaster[3594] trap invalid opcode ip:7f6391472273 sp:7fff48ce2158 error:0 in libpthread-2.5.so[7f6391468000+16000]
>
>    

disassembly around the failure please.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: applications running in KVM crashing?

[Nikola Ciprich]

> disassembly around the failure please.
hmm, is this what You want?

(gdb) disas 0x00007f6391472273
Dump of assembler code for function pthread_rwlock_rdlock:
0x00007f6391472210 <pthread_rwlock_rdlock+0>:   xor    %r10,%r10
0x00007f6391472213 <pthread_rwlock_rdlock+3>:   mov    $0x1,%esi
0x00007f6391472218 <pthread_rwlock_rdlock+8>:   xor    %eax,%eax
0x00007f639147221a <pthread_rwlock_rdlock+10>:  lock cmpxchg %esi,(%rdi)
0x00007f639147221e <pthread_rwlock_rdlock+14>:  jne    0x7f639147227c <pthread_rwlock_rdlock+108>
0x00007f6391472220 <pthread_rwlock_rdlock+16>:  mov    0x18(%rdi),%eax
0x00007f6391472223 <pthread_rwlock_rdlock+19>:  test   %eax,%eax
0x00007f6391472225 <pthread_rwlock_rdlock+21>:  jne    0x7f6391472286 <pthread_rwlock_rdlock+118>
0x00007f6391472227 <pthread_rwlock_rdlock+23>:  cmpl   $0x0,0x14(%rdi)
0x00007f639147222b <pthread_rwlock_rdlock+27>:  je     0x7f639147226c <pthread_rwlock_rdlock+92>
0x00007f639147222d <pthread_rwlock_rdlock+29>:  cmpl   $0x0,0x30(%rdi)
0x00007f6391472231 <pthread_rwlock_rdlock+33>:  je     0x7f639147226c <pthread_rwlock_rdlock+92>
0x00007f6391472233 <pthread_rwlock_rdlock+35>:  incl   0x10(%rdi)
0x00007f6391472236 <pthread_rwlock_rdlock+38>:  je     0x7f63914722ab <pthread_rwlock_rdlock+155>
0x00007f6391472238 <pthread_rwlock_rdlock+40>:  mov    0x8(%rdi),%edx
0x00007f639147223b <pthread_rwlock_rdlock+43>:  lock decl (%rdi)
0x00007f639147223e <pthread_rwlock_rdlock+46>:  jne    0x7f63914722b5 <pthread_rwlock_rdlock+165>
0x00007f6391472240 <pthread_rwlock_rdlock+48>:  mov    0x1c(%rdi),%esi
0x00007f6391472243 <pthread_rwlock_rdlock+51>:  xor    %fs:0x48,%esi
0x00007f639147224b <pthread_rwlock_rdlock+59>:  add    $0x8,%rdi
0x00007f639147224f <pthread_rwlock_rdlock+63>:  mov    $0xca,%eax
0x00007f6391472254 <pthread_rwlock_rdlock+68>:  syscall
0x00007f6391472256 <pthread_rwlock_rdlock+70>:  sub    $0x8,%rdi
0x00007f639147225a <pthread_rwlock_rdlock+74>:  mov    $0x1,%esi
0x00007f639147225f <pthread_rwlock_rdlock+79>:  xor    %eax,%eax
0x00007f6391472261 <pthread_rwlock_rdlock+81>:  lock cmpxchg %esi,(%rdi)
0x00007f6391472265 <pthread_rwlock_rdlock+85>:  jne    0x7f63914722bf <pthread_rwlock_rdlock+175>
0x00007f6391472267 <pthread_rwlock_rdlock+87>:  decl   0x10(%rdi)
0x00007f639147226a <pthread_rwlock_rdlock+90>:  jmp    0x7f6391472220 <pthread_rwlock_rdlock+16>
0x00007f639147226c <pthread_rwlock_rdlock+92>:  xor    %edx,%edx
0x00007f639147226e <pthread_rwlock_rdlock+94>:  incl   0x4(%rdi)
0x00007f6391472271 <pthread_rwlock_rdlock+97>:  je     0x7f63914722a1 <pthread_rwlock_rdlock+145>
0x00007f6391472273 <pthread_rwlock_rdlock+99>:  lock decl (%rdi)
0x00007f6391472276 <pthread_rwlock_rdlock+102>: jne    0x7f6391472297 <pthread_rwlock_rdlock+135>
0x00007f6391472278 <pthread_rwlock_rdlock+104>: mov    %rdx,%rax
0x00007f639147227b <pthread_rwlock_rdlock+107>: retq
0x00007f639147227c <pthread_rwlock_rdlock+108>: mov    0x1c(%rdi),%esi
0x00007f639147227f <pthread_rwlock_rdlock+111>: callq  0x7f63914754a0 <__lll_lock_wait>
0x00007f6391472284 <pthread_rwlock_rdlock+116>: jmp    0x7f6391472220 <pthread_rwlock_rdlock+16>
0x00007f6391472286 <pthread_rwlock_rdlock+118>: cmp    %fs:0x90,%eax
0x00007f639147228e <pthread_rwlock_rdlock+126>: jne    0x7f6391472233 <pthread_rwlock_rdlock+35>
0x00007f6391472290 <pthread_rwlock_rdlock+128>: mov    $0x23,%edx
0x00007f6391472295 <pthread_rwlock_rdlock+133>: jmp    0x7f6391472273 <pthread_rwlock_rdlock+99>
0x00007f6391472297 <pthread_rwlock_rdlock+135>: mov    0x1c(%rdi),%esi
0x00007f639147229a <pthread_rwlock_rdlock+138>: callq  0x7f63914755e0 <__lll_unlock_wake>
0x00007f639147229f <pthread_rwlock_rdlock+143>: jmp    0x7f6391472278 <pthread_rwlock_rdlock+104>
0x00007f63914722a1 <pthread_rwlock_rdlock+145>: decl   0x4(%rdi)
0x00007f63914722a4 <pthread_rwlock_rdlock+148>: mov    $0xb,%edx
0x00007f63914722a9 <pthread_rwlock_rdlock+153>: jmp    0x7f6391472273 <pthread_rwlock_rdlock+99>
0x00007f63914722ab <pthread_rwlock_rdlock+155>: decl   0x10(%rdi)
0x00007f63914722ae <pthread_rwlock_rdlock+158>: mov    $0xb,%edx
0x00007f63914722b3 <pthread_rwlock_rdlock+163>: jmp    0x7f6391472273 <pthread_rwlock_rdlock+99>
0x00007f63914722b5 <pthread_rwlock_rdlock+165>: mov    0x1c(%rdi),%esi
0x00007f63914722b8 <pthread_rwlock_rdlock+168>: callq  0x7f63914755e0 <__lll_unlock_wake>
0x00007f63914722bd <pthread_rwlock_rdlock+173>: jmp    0x7f6391472240 <pthread_rwlock_rdlock+48>
0x00007f63914722bf <pthread_rwlock_rdlock+175>: mov    0x1c(%rdi),%esi
0x00007f63914722c2 <pthread_rwlock_rdlock+178>: add    $0x0,%rdi
0x00007f63914722c6 <pthread_rwlock_rdlock+182>: callq  0x7f63914754a0 <__lll_lock_wait>
0x00007f63914722cb <pthread_rwlock_rdlock+187>: jmp    0x7f6391472267 <pthread_rwlock_rdlock+87>
End of assembler dump.


> 
> -- 
> I have a truly marvellous patch that fixes the bug which this
> signature is too narrow to contain.
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

Re: applications running in KVM crashing?

[Avi Kivity]

On 07/14/2010 04:41 PM, Nikola Ciprich wrote:
>> disassembly around the failure please.
>>      
> hmm, is this what You want?
>
> (gdb) disas 0x00007f6391472273
> 0x00007f6391472273<pthread_rwlock_rdlock+99>:  lock decl (%rdi)
>    

By coincidence, a patch was sent yesterday fixing lock decl.  Can you 
try it out?

commit cb7eaecb3389c7fa2490ea1bee8f10cfa5df30d4
Author: Gleb Natapov <gleb@redhat.com>
Date:   Tue Jul 13 16:40:23 2010 +0300

     KVM: x86: emulator: inc/dec can have lock prefix

     Mark inc (0xfe/0 0xff/0) and dec (0xfe/1 0xff/1) as lock prefix 
capable.

     Signed-off-by: Gleb Natapov <gleb@redhat.com>
     Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>

Although userspace code shouldn't invoke the emulator in most situations.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: applications running in KVM crashing?

[Nikola Ciprich]

Sure!
good timing, we have service window today, so I can give it a try 
almost immediately.
problem needs weeks to reproduce, so I'll report if I got it again.
Since Gleb too states patch should fix it, I guess we can consider
it closed for now...
Thanks a lot to both of You!
regards
nik

On Wed, Jul 14, 2010 at 05:00:39PM +0300, Avi Kivity wrote:
> On 07/14/2010 04:41 PM, Nikola Ciprich wrote:
> >>disassembly around the failure please.
> >hmm, is this what You want?
> >
> >(gdb) disas 0x00007f6391472273
> >0x00007f6391472273<pthread_rwlock_rdlock+99>:  lock decl (%rdi)
> 
> By coincidence, a patch was sent yesterday fixing lock decl.  Can
> you try it out?
> 
> commit cb7eaecb3389c7fa2490ea1bee8f10cfa5df30d4
> Author: Gleb Natapov <gleb@redhat.com>
> Date:   Tue Jul 13 16:40:23 2010 +0300
> 
>     KVM: x86: emulator: inc/dec can have lock prefix
> 
>     Mark inc (0xfe/0 0xff/0) and dec (0xfe/1 0xff/1) as lock prefix
> capable.
> 
>     Signed-off-by: Gleb Natapov <gleb@redhat.com>
>     Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
> 
> Although userspace code shouldn't invoke the emulator in most situations.
> 
> -- 
> I have a truly marvellous patch that fixes the bug which this
> signature is too narrow to contain.
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

Re: applications running in KVM crashing?

[Gleb Natapov]

On Wed, Jul 14, 2010 at 03:41:32PM +0200, Nikola Ciprich wrote:
> > disassembly around the failure please.
> hmm, is this what You want?
>
What "cat /proc/cpuinfo | grep ^flags | tail -1" shows on this machine?

--
			Gleb.

Re: applications running in KVM crashing?

[Nikola Ciprich]

> What "cat /proc/cpuinfo | grep ^flags | tail -1" shows on this machine?
host:
fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx tm2 ssse3 cx16 xtpr pdcm dca lahf_lm tpr_shadow

guest:
fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good pni cx16 hypervisor lahf_lm

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

Re: applications running in KVM crashing?

[Gleb Natapov]

On Wed, Jul 14, 2010 at 04:16:47PM +0200, Nikola Ciprich wrote:
> > What "cat /proc/cpuinfo | grep ^flags | tail -1" shows on this machine?
> host:
> fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx tm2 ssse3 cx16 xtpr pdcm dca lahf_lm tpr_shadow
> 
Thanks. No ept as expected. The patch Avi referred to should fix this.

--
			Gleb.