* Q: Stopped VM still using host cpu CPU ?
@ 2009-10-01 10:32 Daniel Schwager
2009-10-01 11:47 ` Daniel Schwager
[not found] ` <4AC8B6D9.7070003@redhat.com>
0 siblings, 2 replies; 9+ messages in thread
From: Daniel Schwager @ 2009-10-01 10:32 UTC (permalink / raw)
To: kvm; +Cc: Daniel Schwager
Hi,
we are running some stopped (sending "stop" via kvm-monitor socket)
vm's on our system. My intention was to pause (stop) the vm's and
unpause (cont) them on demand (very fast, without time delay, within 2
seconds ..).
After 'stop'ing, the vm's still using CPU-load, like the "top" will
tell:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25983 root 20 0 495m 407m 1876 R 8.9 2.5 228:09.15
qemu-system-x86
25523 root 20 0 495m 2040 1868 S 7.9 0.0 2700:16
qemu-system-x86
19738 root 20 0 1139m 40m 1876 R 7.3 0.3 2501:57
qemu-system-x86
32521 root 20 0 495m 407m 1876 S 6.9 2.5 136:23.23
qemu-system-x86
21773 root 20 0 495m 407m 1876 S 5.6 2.5 575:01.44
qemu-system-x86
6720 root 20 0 2168m 2.0g 1876 R 4.6 12.9 524:55.70
qemu-system-x86
6819 root 20 0 2168m 2.0g 1876 S 4.6 12.9 489:24.24
qemu-system-x86
12752 root 20 0 495m 407m 1876 S 4.6 2.5 0:44.57
qemu-system-x86
22803 root 20 0 1139m 211m 1868 S 4.0 1.3 1217:21
qemu-system-x86
27083 root 20 0 1139m 1.0g 1868 S 4.0 6.5 121:58.05
qemu-system-x86
21970 root 20 0 495m 407m 1876 S 3.6 2.5 434:48.36
qemu-system-x86
3649 root 20 0 496m 394m 1908 S 3.3 2.5 39:41.61
qemu-system-x86
19919 root 20 0 559m 402m 1880 S 3.3 2.5 469:57.14
qemu-system-x86
24461 root 20 0 495m 2048 1876 S 3.3 0.0 1081:41
qemu-system-x86
24619 root 20 0 688m 536m 1880 S 3.0 3.3 293:15.38
qemu-system-x86
24776 root 20 0 495m 407m 1856 S 3.0 2.5 92:08.95
qemu-system-x86
...
So, how can I prevent the paused/stopped VM's to use my CPU from
the hostsystem ? Is there a way to handle this ?
Sure, stop+migrating the vm to filesystem will solve my problem - but
restart
(migrate back from filesystem to RAM) takes a lot (>30 seconds) of time.
best regards
Danny
^ permalink raw reply [flat|nested] 9+ messages in thread* RE: Q: Stopped VM still using host cpu CPU ? 2009-10-01 10:32 Q: Stopped VM still using host cpu CPU ? Daniel Schwager @ 2009-10-01 11:47 ` Daniel Schwager [not found] ` <4AC8B6D9.7070003@redhat.com> 1 sibling, 0 replies; 9+ messages in thread From: Daniel Schwager @ 2009-10-01 11:47 UTC (permalink / raw) To: kvm One more, > So, how can I prevent the paused/stopped VM's to use my CPU from > the hostsystem ? Is there a way to handle this ? If i send a signal STOP/CONT (kill -STOP <pid> or kill -CONT <pid>) to the KVM-process, it looks like the kvm does not (sure ;-) use any host CPU usage. - Are there some side effects using this approach ? (e.g. with networking, ...) - And, why is there a way to send STOP/CONT via socket to KVM-process ? Why not using the sending-signal-apporch ? best regards Danny ^ permalink raw reply [flat|nested] 9+ messages in thread
[parent not found: <4AC8B6D9.7070003@redhat.com>]
[parent not found: <EB31672367A401439CD5A4A10889D57B02118226@exchange.dtnet.de>]
[parent not found: <4AC8C018.6050302@redhat.com>]
* RE: Q: Stopped VM still using host cpu CPU ? [not found] ` <4AC8C018.6050302@redhat.com> @ 2009-10-16 21:35 ` Daniel Schwager 2009-10-18 6:59 ` Avi Kivity 0 siblings, 1 reply; 9+ messages in thread From: Daniel Schwager @ 2009-10-16 21:35 UTC (permalink / raw) To: Avi Kivity; +Cc: kvm > On 10/04/2009 05:21 PM, Daniel Schwager wrote: > > > >> How long is this after the 'stop'? > >> > > 30 seconds or 2 days ... the process takes CPU all the time > > > > Can you take an oprofile run to see where it's spending its time? I'm not that familiar with oprofile ... so, setup with opcontrol --deinit; modprobe oprofile timer=1; opcontrol --start gave me the following result: kvm03:~# opreport CPU: CPU with timer interrupt, speed 0 MHz (estimated) Profiling through timer interrupt TIMER:0| samples| %| ------------------ 613327 95.3150 no-vmlinux 12416 1.9295 qemu-system-x86_64 TIMER:0| samples| %| ------------------ 10888 87.6933 qemu-system-x86_64 209 1.6833 [vdso] (tgid:2164 range:0x7ffff37ff000-0x7ffff3800000) 171 1.3773 [vdso] (tgid:28126 range:0x7fffe7dff000-0x7fffe7e00000) 168 1.3531 [vdso] (tgid:11172 range:0x7ffff67fe000-0x7ffff67ff000) 147 1.1840 [vdso] (tgid:1511 range:0x7fff741fe000-0x7fff741ff000) 143 1.1517 [vdso] (tgid:28327 range:0x7fffdcbfe000-0x7fffdcbff000) 141 1.1356 [vdso] (tgid:17010 range:0x7fff2a1fe000-0x7fff2a1ff000) 127 1.0229 [vdso] (tgid:27527 range:0x7fff2cffe000-0x7fff2cfff000) 111 0.8940 [vdso] (tgid:27644 range:0x7fffec7fe000-0x7fffec7ff000) 92 0.7410 [vdso] (tgid:11565 range:0x7fff247fe000-0x7fff247ff000) 81 0.6524 [vdso] (tgid:27438 range:0x7fff7c3fe000-0x7fff7c3ff000) 76 0.6121 [vdso] (tgid:26857 range:0x7ffffa7fe000-0x7ffffa7ff000) 62 0.4994 [vdso] (tgid:1676 range:0x7fff2f5ff000-0x7fff2f600000) 11488 1.7853 libpthread-2.8.so .... Try to start without timer=1 hung up my physical machine after daemon is started: kvm03:/opt/kvm-86-vnc-patch# opcontrol --deinit; modprobe oprofile ; opcontrol --start Stopping profiling. Killing daemon. Unloading oprofile module Using default event: CPU_CLK_UNHALTED:100000:0:1:1 Using 2.6+ OProfile kernel interface. Using log file /var/lib/oprofile/samples/oprofiled.log Daemon started. Could I give you more information ? Should I setup oprofile in another way ? regards Danny ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Q: Stopped VM still using host cpu CPU ? 2009-10-16 21:35 ` Daniel Schwager @ 2009-10-18 6:59 ` Avi Kivity 2009-10-18 18:29 ` Daniel Schwager 0 siblings, 1 reply; 9+ messages in thread From: Avi Kivity @ 2009-10-18 6:59 UTC (permalink / raw) To: Daniel Schwager; +Cc: kvm On 10/17/2009 06:35 AM, Daniel Schwager wrote: > > I'm not that familiar with oprofile ... > > so, setup with > opcontrol --deinit; modprobe oprofile timer=1; opcontrol --start > > gave me the following result: > > kvm03:~# opreport > Use 'opreport -l'. Make sure your qemu isn't stripped. > Try to start without timer=1 hung up my physical machine after daemon is > started: > kvm03:/opt/kvm-86-vnc-patch# opcontrol --deinit; modprobe oprofile ; > opcontrol --start > Stopping profiling. > Killing daemon. > Unloading oprofile module > Using default event: CPU_CLK_UNHALTED:100000:0:1:1 > Using 2.6+ OProfile kernel interface. > Using log file /var/lib/oprofile/samples/oprofiled.log > Daemon started. > > > Could I give you more information ? Should I setup oprofile in another > way ? > > Is there an oops in dmesg, or is the machine dead? What's the host kernel version? -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. ^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: Q: Stopped VM still using host cpu CPU ? 2009-10-18 6:59 ` Avi Kivity @ 2009-10-18 18:29 ` Daniel Schwager 2009-10-18 18:47 ` Daniel Schwager 0 siblings, 1 reply; 9+ messages in thread From: Daniel Schwager @ 2009-10-18 18:29 UTC (permalink / raw) To: Avi Kivity; +Cc: kvm > > so, setup with > > opcontrol --deinit; modprobe oprofile timer=1; opcontrol --start > > > Use 'opreport -l'. Make sure your qemu isn't stripped. ok ... will do > > > Try to start without timer=1 hung up my physical machine after daemon is > > started: > > Is there an oops in dmesg, or is the machine dead? What's the host > kernel version? ´ Machine was dead. no infos inside dmesg / messages. We use FC9, Linux kvm03.dev.softwaredemo.de 2.6.27.21-78.2.41.fc9.x86_64 #1 SMP Mon Mar 23 23:22:15 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux regards Danny ^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: Q: Stopped VM still using host cpu CPU ? 2009-10-18 18:29 ` Daniel Schwager @ 2009-10-18 18:47 ` Daniel Schwager 2009-10-18 21:57 ` Avi Kivity 2009-10-21 20:29 ` Daniel Schwager 0 siblings, 2 replies; 9+ messages in thread From: Daniel Schwager @ 2009-10-18 18:47 UTC (permalink / raw) To: Daniel Schwager, Avi Kivity; +Cc: kvm > > > so, setup with > > > opcontrol --deinit; modprobe oprofile timer=1; opcontrol --start > > > > > > Use 'opreport -l'. Make sure your qemu isn't stripped. Build kvm-86 with "./configure ... --disable-strip" and paused vm, now I got: opreport -l CPU: CPU with timer interrupt, speed 0 MHz (estimated) Profiling through timer interrupt samples % image name app name symbol name 112184 99.1919 no-vmlinux no-vmlinux (no symbols) 266 0.2352 qemu-system-x86_64 qemu-system-x86_64 (no symbols) 94 0.0831 libpthread-2.8.so libpthread-2.8.so __read_nocancel 72 0.0637 opreport opreport (no symbols) 63 0.0557 libc-2.8.so libc-2.8.so __select_nocancel 55 0.0486 libpthread-2.8.so libpthread-2.8.so __write_nocancel 37 0.0327 libpthread-2.8.so libpthread-2.8.so __pthread_enable_asynccancel 29 0.0256 libpthread-2.8.so libpthread-2.8.so __pthread_disable_asynccancel 28 0.0248 libpthread-2.8.so libpthread-2.8.so __libc_sigaction 28 0.0248 libpthread-2.8.so libpthread-2.8.so pthread_mutex_lock 21 0.0186 librt-2.8.so librt-2.8.so clock_gettime 20 0.0177 bash bash (no symbols) 19 0.0168 [vdso] (tgid:4783 range:0x7ffffb3ff000-0x7ffffb400000) qemu-system-x86_64 (no symbols) 19 0.0168 libc-2.8.so libc-2.8.so __libc_disable_asynccancel 17 0.0150 librt-2.8.so librt-2.8.so timer_settime@@GLIBC_2.3.3 15 0.0133 libpthread-2.8.so libpthread-2.8.so __pthread_mutex_unlock_usercnt 15 0.0133 librt-2.8.so librt-2.8.so timer_gettime@@GLIBC_2.3.3 13 0.0115 oprofiled oprofiled (no symbols) 12 0.0106 libc-2.8.so libc-2.8.so __libc_enable_asynccancel 12 0.0106 libstdc++.so.6.0.10 libstdc++.so.6.0.10 (no symbols) 10 0.0088 libc-2.8.so libc-2.8.so _int_malloc 6 ... And top tell me PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4783 root 20 0 631m 536m 1956 S 7.0 6.7 0:22.94 qemu-system-x86 But the VM is paused - definitely .. Danny ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Q: Stopped VM still using host cpu CPU ? 2009-10-18 18:47 ` Daniel Schwager @ 2009-10-18 21:57 ` Avi Kivity 2009-10-21 20:29 ` Daniel Schwager 1 sibling, 0 replies; 9+ messages in thread From: Avi Kivity @ 2009-10-18 21:57 UTC (permalink / raw) To: Daniel Schwager; +Cc: kvm On 10/19/2009 03:47 AM, Daniel Schwager wrote: > Build kvm-86 with "./configure ... --disable-strip" > and paused vm, now I got: > > opreport -l > > CPU: CPU with timer interrupt, speed 0 MHz (estimated) > Profiling through timer interrupt > samples % image name app name > symbol name > 112184 99.1919 no-vmlinux no-vmlinux (no > symbols) > 266 0.2352 qemu-system-x86_64 qemu-system-x86_64 (no > symbols) > Are you using the right qemu? Note 'no symbols'. Also, most samples come from the kernel, install the kernel-debuginfo package as well. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. ^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: Q: Stopped VM still using host cpu CPU ? 2009-10-18 18:47 ` Daniel Schwager 2009-10-18 21:57 ` Avi Kivity @ 2009-10-21 20:29 ` Daniel Schwager 2009-10-22 13:29 ` Avi Kivity 1 sibling, 1 reply; 9+ messages in thread From: Daniel Schwager @ 2009-10-21 20:29 UTC (permalink / raw) To: kvm; +Cc: avi Hi Avi, > > > > so, setup with > > > > opcontrol --deinit; modprobe oprofile timer=1; opcontrol --start > > > > > > > > > Use 'opreport -l'. Make sure your qemu isn't stripped. All VM's are in paused state: top - 22:08:15 up 2 days, 12:18, 8 users, load average: 0.12, 0.19, 0.14 Tasks: 185 total, 1 running, 182 sleeping, 2 stopped, 0 zombie Cpu(s): 0.5%us, 1.7%sy, 0.0%ni, 97.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 8196468k total, 4166468k used, 4030000k free, 76044k buffers Swap: 39780320k total, 70232k used, 39710088k free, 475876k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9587 root 20 0 1145m 1.0g 1944 S 6.6 13.1 0:28.78 qemu-system-x86 9525 root 20 0 1143m 1.0g 1900 S 6.3 13.1 0:27.26 qemu-system-x86 9305 root 20 0 1143m 1.0g 1900 S 5.0 13.1 0:22.86 qemu-system-x86 **** opreport -l --symbols | less CPU: CPU with timer interrupt, speed 0 MHz (estimated) Profiling through timer interrupt samples % image name app name symbol name 418814 98.5250 no-vmlinux no-vmlinux (no symbols) 1228 0.2889 qemu-system-x86_64 qemu-system-x86_64 main_loop_wait 888 0.2089 libpthread-2.8.so libpthread-2.8.so __read_nocancel ... **** opreport -l /opt/kvm-86-vnc-patch.oprofile/usr/bin/qemu-system-x86_64 --symbols | less CPU: CPU with timer interrupt, speed 0 MHz (estimated) Profiling through timer interrupt samples % image name symbol name 1036 52.2441 qemu-system-x86_64 main_loop_wait 155 7.8164 qemu-system-x86_64 dynticks_rearm_timer 85 4.2864 [vdso] (tgid:9525 range:0x7fff565fe000-0x7fff565ff000) (no symbols) 77 3.8830 qemu-system-x86_64 qemu_get_clock 63 3.1770 qemu-system-x86_64 .plt 59 2.9753 qemu-system-x86_64 host_alarm_handler 57 2.8744 qemu-system-x86_64 qemu_shutdown_requested 51 2.5719 [vdso] (tgid:9305 range:0x7fff3d9ff000-0x7fff3da00000) (no symbols) 49 2.4710 qemu-system-x86_64 sigfd_handler 47 2.3701 qemu-system-x86_64 tap_can_send 41 2.0676 [vdso] (tgid:9587 range:0x7ffffabfe000-0x7ffffabff000) (no symbols) 38 1.9163 qemu-system-x86_64 kvm_mutex_lock 36 1.8154 qemu-system-x86_64 kvm_main_loop 31 1.5633 qemu-system-x86_64 get_clock 29 1.4624 qemu-system-x86_64 qemu_bh_poll 24 1.2103 qemu-system-x86_64 io_thread_wakeup 23 1.1599 qemu-system-x86_64 qemu_event_read 19 0.9581 qemu-system-x86_64 qemu_powerdown_requested 18 0.9077 qemu-system-x86_64 kvm_mutex_unlock 12 0.6051 qemu-system-x86_64 qemu_kvm_notify_work 12 0.6051 qemu-system-x86_64 qemu_reset_requested 11 0.5547 qemu-system-x86_64 e1000_can_receive 6 0.3026 qemu-system-x86_64 slirp_is_inited 3 0.1513 qemu-system-x86_64 qemu_notify_event 1 0.0504 qemu-system-x86_64 qemu_set_fd_handler2 I do not understand, why only ~1036 kvm-samples produces a load of (in sum) 18% ... I'm not able to convert vmlinuz to vmlinux (for profiling the kernel ...) on my old FC9 - sorry. Because I cannot update my old fc9, I will move to CentOS later - and hopefully get the kernel profiling running. May these oprofile-information will help you to track down the problem - if not, I try to came back soon with kernel-profiling information on CentOS. regards Danny ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Q: Stopped VM still using host cpu CPU ? 2009-10-21 20:29 ` Daniel Schwager @ 2009-10-22 13:29 ` Avi Kivity 0 siblings, 0 replies; 9+ messages in thread From: Avi Kivity @ 2009-10-22 13:29 UTC (permalink / raw) To: Daniel Schwager; +Cc: kvm On 10/21/2009 10:29 PM, Daniel Schwager wrote: > > **** opreport -l --symbols | less > CPU: CPU with timer interrupt, speed 0 MHz (estimated) > Profiling through timer interrupt > samples % image name app name > symbol name > 418814 98.5250 no-vmlinux no-vmlinux (no > symbols) > 1228 0.2889 qemu-system-x86_64 qemu-system-x86_64 > main_loop_wait > 888 0.2089 libpthread-2.8.so libpthread-2.8.so > __read_nocancel > ... > > Sorry, I gave you the wrong instructions. Since most of the time is spent in vmlinux, we want to trace vmlinux, not qemu. Three ways to see what is happening: - set up vmlinux (you'll need to install the correct kernel-debuginfo package to get this). - strace -p `pgrep qemu` and see what syscalls take place - run 'top' and look at the wchan field ('f' 'y' will show it if it is hidden) Probably easiest in reverse order as listed. -- error compiling committee.c: too many arguments to function ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2009-10-22 13:29 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-10-01 10:32 Q: Stopped VM still using host cpu CPU ? Daniel Schwager
2009-10-01 11:47 ` Daniel Schwager
[not found] ` <4AC8B6D9.7070003@redhat.com>
[not found] ` <EB31672367A401439CD5A4A10889D57B02118226@exchange.dtnet.de>
[not found] ` <4AC8C018.6050302@redhat.com>
2009-10-16 21:35 ` Daniel Schwager
2009-10-18 6:59 ` Avi Kivity
2009-10-18 18:29 ` Daniel Schwager
2009-10-18 18:47 ` Daniel Schwager
2009-10-18 21:57 ` Avi Kivity
2009-10-21 20:29 ` Daniel Schwager
2009-10-22 13:29 ` Avi Kivity
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).