From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([2001:4830:134:3::10]:42213)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <lists@wiesinger.com>) id 1YNNJT-0007uq-JZ
	for qemu-devel@nongnu.org; Mon, 16 Feb 2015 10:10:23 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <lists@wiesinger.com>) id 1YNNJO-0006T2-Th
	for qemu-devel@nongnu.org; Mon, 16 Feb 2015 10:10:19 -0500
Received: from vps01.wiesinger.com ([46.36.37.179]:59841)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <lists@wiesinger.com>) id 1YNNJO-00060k-I6
	for qemu-devel@nongnu.org; Mon, 16 Feb 2015 10:10:14 -0500
Message-ID: <54E20812.4090006@wiesinger.com>
Date: Mon, 16 Feb 2015 16:09:06 +0100
From: Gerhard Wiesinger <lists@wiesinger.com>
MIME-Version: 1.0
References: <54AE87C1.2060907@wiesinger.com>	<54AEBD43.2060705@redhat.com>	<54AEC877.9080600@wiesinger.com>	<54AECAF3.3060909@redhat.com>	<54AF047D.8010009@wiesinger.com>	<54B3B2F5.1090405@wiesinger.com>	<54B57C51.7090002@wiesinger.com>	<54B584AB.4090303@redhat.com>	<54B58AC0.5080805@wiesinger.com>	<54B58B18.9060205@redhat.com>	<54B595C7.3080101@wiesinger.com>
	<54B5BF5F.9000805@redhat.com> <54B633CE.3040901@wiesinger.com>
	<54E05659.9050701@wiesinger.com> <54E1FC2B.3030805@redhat.com>
In-Reply-To: <54E1FC2B.3030805@redhat.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Subject: Re: [Qemu-devel] Fedora FC21 - Bug: 100% CPU and hangs in
 gettimeofday(&tp, NULL); forever
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: Paolo Bonzini <pbonzini@redhat.com>, Laine Stump <laine@redhat.com>, qemu-devel@nongnu.org, Cole Robinson <crobinso@redhat.com>, virt@lists.fedoraproject.org

On 16.02.2015 15:18, Paolo Bonzini wrote:
>
> On 15/02/2015 09:18, Gerhard Wiesinger wrote:
>>
>> Can you grab some random backtraces ("thread apply all bt full") with gdb?

Very low load on the machine, so I guess most will be sleeping and hard 
to catch non sleeping ones. See below
For the records: gdb -p 14139 -ex 'thread apply all bt full' -batch
>>
>> What is the libvirt XML or qemu command line?

See below.
>> Kernel (host/guest):  3.18.6-200.fc21.x86_64 #1 SMP
>> qemu-kvm-2.2.0-5.fc21.x86_64
>>
>> Bug 1178975 - endless loop in clock_gettime() on a kvm-based VM
>> https://bugzilla.redhat.com/show_bug.cgi?id=1178975
>> is fixed (didn't occour with the test program posted at
>> https://bugzilla.redhat.com/show_bug.cgi?id=1178975#c28 in 30min,
>> happened before reproduceable in 2min, still running)
>>
>> So I guess there is another problem in the kernel with volatile and gcc
>> optimizations (or maybe in qemu-KVM)
> No, this doesn't look like volatile.

But why 100% on 2 cores?

There were also recent bugfixes with optimizations in gcc-4.9.2-6.fc21 
maybe we hit one of these.
- update from the 4.9 branch
   - PRs c++/54442, c++/64514, c++/64521, c++/64901, c/57653, c/61553, 
c/64766,
     c/64778, c/64824, c/64868, debug/64511, debug/64663, fortran/56867,
     fortran/57023, fortran/60922, fortran/62044, fortran/63733,
     fortran/64230, fortran/64528, fortran/64771, ipa/63970, ipa/64068,
     ipa/64559, libstdc++/64476, libstdc++/64584, libstdc++/64585,
     libstdc++/64646, libstdc++/64649, libstdc++/64680, middle-end/63704,
     middle-end/64391, middle-end/64421, middle-end/64734,
     rtl-optimization/61058, rtl-optimization/63637,
     rtl-optimization/64286, rtl-optimization/64557, target/61413,
     target/63424, target/64358, target/64479, target/64505, target/64513,
     target/64580, target/64795, target/64882, target/64938, target/64979,
     testsuite/64712, tree-optimization/64563

Ciao,
Gerhard

/usr/bin/qemu-system-x86_64 -machine accel=kvm -name myvm -S -machine 
pc-0.15,accel=kvm,usb=off -m 384 -realtime mlock=off -smp 
2,sockets=2,cores=1,threads=1 -uuid b40e77d3-cd86-4d59-9ee4-5756ec88bf99 
-no-user-config -nodefaults -chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvm.monitor,server,nowait 
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc 
-no-shutdown -boot menu=on,strict=on -device 
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device 
lsi,id=scsi0,bus=pci.0,addr=0x6 -device ahci,id=ahci0,bus=pci.0,addr=0x9 
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive 
if=none,id=drive-ide0-1-0,readonly=on,format=raw -device 
ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive 
file=/var/lib/libvirt/images/myvm.img,if=none,id=drive-virtio-disk0,format=qcow2 
-device 
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 
-netdev tap,fd=33,id=hostnet0,vhost=on,vhostfd=34 -device 
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:aa:bb:cc,bus=pci.0,addr=0x8 
-netdev tap,fd=35,id=hostnet1,vhost=on,vhostfd=36 -device 
virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:aa:bb:cd,bus=pci.0,addr=0x7 
-chardev pty,id=charserial0 -device 
isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 
-vnc 127.0.0.1:9 -k de -device 
VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -device 
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on

<domain type='kvm' id='12'>
   <name>myvm</name>
   <uuid>b40e77d3-cd86-4d59-9ee4-5756ec88bf99</uuid>
   <description>Fedora 21</description>
   <memory unit='KiB'>393216</memory>
   <currentMemory unit='KiB'>393216</currentMemory>
   <vcpu placement='static'>2</vcpu>
   <resource>
     <partition>/machine</partition>
   </resource>
   <os>
     <type arch='x86_64' machine='pc-0.15'>hvm</type>
     <bootmenu enable='yes'/>
   </os>
   <features>
     <acpi/>
     <apic/>
     <pae/>
   </features>
   <clock offset='utc'/>
   <on_poweroff>destroy</on_poweroff>
   <on_reboot>restart</on_reboot>
   <on_crash>restart</on_crash>
   <devices>
     <emulator>/usr/bin/qemu-kvm</emulator>
     <disk type='block' device='cdrom'>
       <driver name='qemu' type='raw'/>
       <backingStore/>
       <target dev='hdc' bus='ide'/>
       <readonly/>
       <alias name='ide0-1-0'/>
       <address type='drive' controller='0' bus='1' target='0' unit='0'/>
     </disk>
     <disk type='file' device='disk'>
       <driver name='qemu' type='qcow2'/>
       <source file='/var/lib/libvirt/images/myvm.img'/>
       <backingStore/>
       <target dev='vda' bus='virtio'/>
       <boot order='1'/>
       <alias name='virtio-disk0'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x03' 
function='0x0'/>
     </disk>
     <controller type='ide' index='0'>
       <alias name='ide0'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x1'/>
     </controller>
     <controller type='usb' index='0'>
       <alias name='usb0'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x2'/>
     </controller>
     <controller type='virtio-serial' index='0'>
       <alias name='virtio-serial0'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x04' 
function='0x0'/>
     </controller>
     <controller type='scsi' index='0'>
       <alias name='scsi0'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x06' 
function='0x0'/>
     </controller>
     <controller type='sata' index='0'>
       <alias name='sata0'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x09' 
function='0x0'/>
     </controller>
     <controller type='pci' index='0' model='pci-root'>
       <alias name='pci.0'/>
     </controller>
     <interface type='network'>
       <mac address='52:54:00:aa:bb:cc'/>
       <source network='DMZ' bridge='virbr2'/>
       <target dev='vnet16'/>
       <model type='virtio'/>
       <alias name='net0'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x08' 
function='0x0'/>
     </interface>
     <interface type='bridge'>
       <mac address='52:54:00:aa:bb:cd'/>
       <source bridge='br0'/>
       <target dev='vnet17'/>
       <model type='virtio'/>
       <alias name='net1'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x07' 
function='0x0'/>
     </interface>
     <serial type='pty'>
       <source path='/dev/pts/9'/>
       <target port='0'/>
       <alias name='serial0'/>
     </serial>
     <console type='pty' tty='/dev/pts/9'>
       <source path='/dev/pts/9'/>
       <target type='serial' port='0'/>
       <alias name='serial0'/>
     </console>
     <input type='tablet' bus='usb'>
       <alias name='input0'/>
     </input>
     <input type='mouse' bus='ps2'/>
     <input type='keyboard' bus='ps2'/>
     <graphics type='vnc' port='5909' autoport='yes' listen='127.0.0.1' 
keymap='de'>
       <listen type='address' address='127.0.0.1'/>
     </graphics>
     <video>
       <model type='vga' vram='16384' heads='1'/>
       <alias name='video0'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x02' 
function='0x0'/>
     </video>
     <memballoon model='virtio'>
       <alias name='balloon0'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x05' 
function='0x0'/>
     </memballoon>
   </devices>
</domain>




Thread 11 (Thread 0x7ffe8bdfe700 (LWP 14147)):
#0  0x00007ffe9dca8977 in ioctl () at /lib64/libc.so.6
#1  0x00007ffea8c37c35 in kvm_vcpu_ioctl ()
#2  0x00007ffea8c37cec in kvm_cpu_exec ()
#3  0x00007ffea8c25b02 in qemu_kvm_cpu_thread_fn ()
#4  0x00007ffea771952a in start_thread () at /lib64/libpthread.so.0
#5  0x00007ffe9dcb279d in clone () at /lib64/libc.so.6

Thread 10 (Thread 0x7ffe8b5fd700 (LWP 14148)):
#0  0x00007ffe9dca8977 in ioctl () at /lib64/libc.so.6
#1  0x00007ffea8c37c35 in kvm_vcpu_ioctl ()
#2  0x00007ffea8c37cec in kvm_cpu_exec ()
#3  0x00007ffea8c25b02 in qemu_kvm_cpu_thread_fn ()
#4  0x00007ffea771952a in start_thread () at /lib64/libpthread.so.0
#5  0x00007ffe9dcb279d in clone () at /lib64/libc.so.6

Thread 9 (Thread 0x7ffe893ff700 (LWP 14158)):
#0  0x00007ffea771e590 in pthread_cond_wait@@GLIBC_2.3.2 () at 
/lib64/libpthread.so.0
#1  0x00007ffea8eccd79 in qemu_cond_wait ()
#2  0x00007ffea8e50dd3 in vnc_worker_thread_loop ()
#3  0x00007ffea8e511b8 in vnc_worker_thread ()
#4  0x00007ffea771952a in start_thread () at /lib64/libpthread.so.0
#5  0x00007ffe9dcb279d in clone () at /lib64/libc.so.6

Thread 8 (Thread 0x7ffe4effd700 (LWP 18334)):
#0  0x00007ffea77207f0 in sem_timedwait () at /lib64/libpthread.so.0
#1  0x00007ffea8eccec7 in qemu_sem_timedwait ()
#2  0x00007ffea8e5523c in worker_thread ()
#3  0x00007ffea771952a in start_thread () at /lib64/libpthread.so.0
#4  0x00007ffe9dcb279d in clone () at /lib64/libc.so.6

Thread 7 (Thread 0x7ffe8231f700 (LWP 18335)):
#0  0x00007ffea77207f0 in sem_timedwait () at /lib64/libpthread.so.0
#1  0x00007ffea8eccec7 in qemu_sem_timedwait ()
#2  0x00007ffea8e5523c in worker_thread ()
#3  0x00007ffea771952a in start_thread () at /lib64/libpthread.so.0
#4  0x00007ffe9dcb279d in clone () at /lib64/libc.so.6

Thread 6 (Thread 0x7ffe908c7700 (LWP 22620)):
#0  0x00007ffea77207f0 in sem_timedwait () at /lib64/libpthread.so.0
#1  0x00007ffea8eccec7 in qemu_sem_timedwait ()
#2  0x00007ffea8e5523c in worker_thread ()
#3  0x00007ffea771952a in start_thread () at /lib64/libpthread.so.0
#4  0x00007ffe9dcb279d in clone () at /lib64/libc.so.6

Thread 5 (Thread 0x7ffe2d7fa700 (LWP 12264)):
#0  0x00007ffea77207f0 in sem_timedwait () at /lib64/libpthread.so.0
#1  0x00007ffea8eccec7 in qemu_sem_timedwait ()
#2  0x00007ffea8e5523c in worker_thread ()
#3  0x00007ffea771952a in start_thread () at /lib64/libpthread.so.0
#4  0x00007ffe9dcb279d in clone () at /lib64/libc.so.6

Thread 4 (Thread 0x7ffe83622700 (LWP 17892)):
#0  0x00007ffea77207f0 in sem_timedwait () at /lib64/libpthread.so.0
#1  0x00007ffea8eccec7 in qemu_sem_timedwait ()
#2  0x00007ffea8e5523c in worker_thread ()
#3  0x00007ffea771952a in start_thread () at /lib64/libpthread.so.0
#4  0x00007ffe9dcb279d in clone () at /lib64/libc.so.6

Thread 3 (Thread 0x7ffe4f7fe700 (LWP 17893)):
#0  0x00007ffea77207f0 in sem_timedwait () at /lib64/libpthread.so.0
#1  0x00007ffea8eccec7 in qemu_sem_timedwait ()
#2  0x00007ffea8e5523c in worker_thread ()
#3  0x00007ffea771952a in start_thread () at /lib64/libpthread.so.0
#4  0x00007ffe9dcb279d in clone () at /lib64/libc.so.6

Thread 2 (Thread 0x7ffe82b20700 (LWP 17894)):
---Type <return> to continue, or q <return> to quit---
#0  0x00007ffea77207f0 in sem_timedwait () at /lib64/libpthread.so.0
#1  0x00007ffea8eccec7 in qemu_sem_timedwait ()
#2  0x00007ffea8e5523c in worker_thread ()
#3  0x00007ffea771952a in start_thread () at /lib64/libpthread.so.0
#4  0x00007ffe9dcb279d in clone () at /lib64/libc.so.6

Thread 1 (Thread 0x7ffea8aa5a80 (LWP 14139)):
#0  0x00007ffe9dca72c1 in ppoll () at /lib64/libc.so.6
#1  0x00007ffea8e6289c in qemu_poll_ns ()
#2  0x00007ffea8e62034 in main_loop_wait ()
#3  0x00007ffea8bfdcdd in main ()


[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007ffe9dca72c1 in ppoll () from /lib64/libc.so.6

Thread 8 (Thread 0x7ffe8bdfe700 (LWP 14147)):
#0  0x00007ffe9dca8977 in ioctl () at /lib64/libc.so.6
#1  0x00007ffea8c37c35 in kvm_vcpu_ioctl ()
#2  0x00007ffea8c37cec in kvm_cpu_exec ()
#3  0x00007ffea8c25b02 in qemu_kvm_cpu_thread_fn ()
#4  0x00007ffea771952a in start_thread () at /lib64/libpthread.so.0
#5  0x00007ffe9dcb279d in clone () at /lib64/libc.so.6

Thread 7 (Thread 0x7ffe8b5fd700 (LWP 14148)):
#0  0x00007ffe9dca8977 in ioctl () at /lib64/libc.so.6
#1  0x00007ffea8c37c35 in kvm_vcpu_ioctl ()
#2  0x00007ffea8c37cec in kvm_cpu_exec ()
#3  0x00007ffea8c25b02 in qemu_kvm_cpu_thread_fn ()
#4  0x00007ffea771952a in start_thread () at /lib64/libpthread.so.0
#5  0x00007ffe9dcb279d in clone () at /lib64/libc.so.6

Thread 6 (Thread 0x7ffe893ff700 (LWP 14158)):
#0  0x00007ffea771e590 in pthread_cond_wait@@GLIBC_2.3.2 () at 
/lib64/libpthread.so.0
#1  0x00007ffea8eccd79 in qemu_cond_wait ()
#2  0x00007ffea8e50dd3 in vnc_worker_thread_loop ()
#3  0x00007ffea8e511b8 in vnc_worker_thread ()
#4  0x00007ffea771952a in start_thread () at /lib64/libpthread.so.0
#5  0x00007ffe9dcb279d in clone () at /lib64/libc.so.6

Thread 5 (Thread 0x7ffdeeffd700 (LWP 31356)):
#0  0x00007ffea77207f0 in sem_timedwait () at /lib64/libpthread.so.0
#1  0x00007ffea8eccec7 in qemu_sem_timedwait ()
#2  0x00007ffea8e5523c in worker_thread ()
#3  0x00007ffea771952a in start_thread () at /lib64/libpthread.so.0
#4  0x00007ffe9dcb279d in clone () at /lib64/libc.so.6

Thread 4 (Thread 0x7ffdef7fe700 (LWP 32680)):
#0  0x00007ffea77207f0 in sem_timedwait () at /lib64/libpthread.so.0
#1  0x00007ffea8eccec7 in qemu_sem_timedwait ()
#2  0x00007ffea8e5523c in worker_thread ()
#3  0x00007ffea771952a in start_thread () at /lib64/libpthread.so.0
#4  0x00007ffe9dcb279d in clone () at /lib64/libc.so.6

Thread 3 (Thread 0x7ffe0ffff700 (LWP 32681)):
#0  0x00007ffea77207f0 in sem_timedwait () at /lib64/libpthread.so.0
#1  0x00007ffea8eccec7 in qemu_sem_timedwait ()
#2  0x00007ffea8e5523c in worker_thread ()
#3  0x00007ffea771952a in start_thread () at /lib64/libpthread.so.0
#4  0x00007ffe9dcb279d in clone () at /lib64/libc.so.6

Thread 2 (Thread 0x7ffe8131d700 (LWP 32682)):
#0  0x00007ffea77207f0 in sem_timedwait () at /lib64/libpthread.so.0
#1  0x00007ffea8eccec7 in qemu_sem_timedwait ()
#2  0x00007ffea8e5523c in worker_thread ()
#3  0x00007ffea771952a in start_thread () at /lib64/libpthread.so.0
#4  0x00007ffe9dcb279d in clone () at /lib64/libc.so.6

Thread 1 (Thread 0x7ffea8aa5a80 (LWP 14139)):
#0  0x00007ffe9dca72c1 in ppoll () at /lib64/libc.so.6
#1  0x00007ffea8e6289c in qemu_poll_ns ()
#2  0x00007ffea8e62034 in main_loop_wait ()
#3  0x00007ffea8bfdcdd in main ()