From: Michael Tokarev <mjt@tls.msk.ru>
To: KVM list <kvm@vger.kernel.org>
Cc: Svante Signell <srs@kth.se>
Subject: major slowdown in i/o (due to high CPU usage) from 2.6.32 to current
Date: Tue, 19 Jul 2011 00:16:25 +0400 [thread overview]
Message-ID: <4E249499.4050502@msgid.tls.msk.ru> (raw)
It all started by Svante Signell on IRC, see
http://bugs.debian.org/634149 . The short story:
running hurd guest image (on ide drive); when changing
only host kernel from 2.6.32 to (in my case) 3.0.0-rc7,
the i/o (especially write) speed of the guest reduces
dramatically, as is host CPU usage. I used a cd-1.iso
install image from the URL provided in the bugreport,
http://people.debian.org/~sthibault/hurd-i386/installer/cdimage/ --
even installing packages is very significantly slower
in current kernels than it was in 2.6.32.
Typical kvm_stats output on 3.0.0 host kernel:
kvm statistics
exits 350842433 277118
io_exits 335217689 274154
insn_emulation 276092554 259665
host_state_reload 265635092 259610
mmu_pte_write 83612845 3413
irq_injections 643860 449
irq_exits 134069 100
fpu_reload 6284 0
pf_fixed 144563 0
halt_exits 4544 0
halt_wakeup 4544 0
mmio_exits 2864 0
mmu_shadow_zapped 1675 0
mmu_cache_miss 1402 0
remote_tlb_flush 7 0
signal_exits 1 0
The same, on 2.6.32:
kvm statistics
exits 47692767 460212
io_exits 46815092 459058
host_state_reload 5093435 23353
fpu_reload 5093387 23344
irq_injections 475229 973
insn_emulation 852122 864
irq_exits 51672 235
mmio_exits 173630 0
pf_fixed 148051 0
halt_exits 3150 0
mmu_shadow_zapped 2650 0
mmu_cache_miss 1531 0
halt_wakeup 60 0
remote_tlb_flush 12 0
signal_exits 1 0
(this is while the installation process is running).
typical output from trace-cmd, for 3.0.0:
kvm-18166 [001] 47471.398399: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753
kvm-18166 [001] 47471.398399: kvm_emulate_insn: 80000000:160750: f3 66 6f
kvm-18166 [001] 47471.398400: kvm_pio: pio_write at 0x1f0 size 2 count 1
kvm-18166 [001] 47471.398400: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-18166 [001] 47471.398401: kvm_entry: vcpu 0
kvm-18166 [001] 47471.398402: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753
kvm-18166 [001] 47471.398403: kvm_emulate_insn: 80000000:160750: f3 66 6f
kvm-18166 [001] 47471.398403: kvm_pio: pio_write at 0x1f0 size 2 count 1
kvm-18166 [001] 47471.398403: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-18166 [001] 47471.398405: kvm_entry: vcpu 0
kvm-18166 [001] 47471.398405: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753
kvm-18166 [001] 47471.398406: kvm_emulate_insn: 80000000:160750: f3 66 6f
kvm-18166 [001] 47471.398406: kvm_pio: pio_write at 0x1f0 size 2 count 1
kvm-18166 [001] 47471.398407: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-18166 [001] 47471.398408: kvm_entry: vcpu 0
kvm-18166 [001] 47471.398409: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753
kvm-18166 [001] 47471.398409: kvm_emulate_insn: 80000000:160750: f3 66 6f
kvm-18166 [001] 47471.398410: kvm_pio: pio_write at 0x1f0 size 2 count 1
kvm-18166 [001] 47471.398410: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-18166 [001] 47471.398411: kvm_entry: vcpu 0
kvm-18166 [001] 47471.398412: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753
kvm-18166 [001] 47471.398413: kvm_emulate_insn: 80000000:160750: f3 66 6f
kvm-18166 [001] 47471.398413: kvm_pio: pio_write at 0x1f0 size 2 count 1
kvm-18166 [001] 47471.398413: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-18166 [001] 47471.398415: kvm_entry: vcpu 0
kvm-18166 [001] 47471.398415: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753
kvm-18166 [001] 47471.398416: kvm_emulate_insn: 80000000:160750: f3 66 6f
kvm-18166 [001] 47471.398416: kvm_pio: pio_write at 0x1f0 size 2 count 1
kvm-18166 [001] 47471.398417: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-18166 [001] 47471.398418: kvm_entry: vcpu 0
kvm-18166 [001] 47471.398419: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753
kvm-18166 [001] 47471.398419: kvm_emulate_insn: 80000000:160750: f3 66 6f
kvm-18166 [001] 47471.398420: kvm_pio: pio_write at 0x1f0 size 2 count 1
kvm-18166 [001] 47471.398420: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-18166 [001] 47471.398421: kvm_entry: vcpu 0
kvm-18166 [001] 47471.398422: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753
kvm-18166 [001] 47471.398423: kvm_emulate_insn: 80000000:160750: f3 66 6f
kvm-18166 [001] 47471.398423: kvm_pio: pio_write at 0x1f0 size 2 count 1
kvm-18166 [001] 47471.398423: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-18166 [001] 47471.398425: kvm_entry: vcpu 0
kvm-18166 [001] 47471.398425: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753
kvm-18166 [001] 47471.398426: kvm_emulate_insn: 80000000:160750: f3 66 6f
kvm-18166 [001] 47471.398427: kvm_pio: pio_write at 0x1f0 size 2 count 1
kvm-18166 [001] 47471.398427: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-18166 [001] 47471.398428: kvm_entry: vcpu 0
kvm-18166 [001] 47471.398429: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753
kvm-18166 [001] 47471.398429: kvm_emulate_insn: 80000000:160750: f3 66 6f
kvm-18166 [001] 47471.398430: kvm_pio: pio_write at 0x1f0 size 2 count 1
kvm-18166 [001] 47471.398430: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-18166 [001] 47471.398431: kvm_entry: vcpu 0
kvm-18166 [001] 47471.398432: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753
kvm-18166 [001] 47471.398433: kvm_emulate_insn: 80000000:160750: f3 66 6f
kvm-18166 [001] 47471.398433: kvm_pio: pio_write at 0x1f0 size 2 count 1
kvm-18166 [001] 47471.398433: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-18166 [001] 47471.398435: kvm_entry: vcpu 0
kvm-18166 [001] 47471.398435: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753
kvm-18166 [001] 47471.398436: kvm_emulate_insn: 80000000:160750: f3 66 6f
kvm-18166 [001] 47471.398437: kvm_pio: pio_write at 0x1f0 size 2 count 1
kvm-18166 [001] 47471.398437: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-18166 [001] 47471.398438: kvm_entry: vcpu 0
kvm-18166 [001] 47471.398439: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753
kvm-18166 [001] 47471.398440: kvm_emulate_insn: 80000000:160750: f3 66 6f
kvm-18166 [001] 47471.398440: kvm_pio: pio_write at 0x1f0 size 2 count 1
kvm-18166 [001] 47471.398440: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-18166 [001] 47471.398442: kvm_entry: vcpu 0
kvm-18166 [001] 47471.398442: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753
kvm-18166 [001] 47471.398443: kvm_emulate_insn: 80000000:160750: f3 66 6f
kvm-18166 [001] 47471.398444: kvm_pio: pio_write at 0x1f0 size 2 count 1
kvm-18166 [001] 47471.398444: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-18166 [001] 47471.398445: kvm_entry: vcpu 0
kvm-18166 [001] 47471.398446: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753
kvm-18166 [001] 47471.398447: kvm_emulate_insn: 80000000:160750: f3 66 6f
kvm-18166 [001] 47471.398447: kvm_pio: pio_write at 0x1f0 size 2 count 1
kvm-18166 [001] 47471.398447: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-18166 [001] 47471.398449: kvm_entry: vcpu 0
kvm-18166 [001] 47471.398449: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753
kvm-18166 [001] 47471.398450: kvm_emulate_insn: 80000000:160750: f3 66 6f
kvm-18166 [001] 47471.398450: kvm_pio: pio_write at 0x1f0 size 2 count 1
kvm-18166 [001] 47471.398451: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-18166 [001] 47471.398452: kvm_entry: vcpu 0
kvm-18166 [001] 47471.398453: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753
kvm-18166 [001] 47471.398453: kvm_emulate_insn: 80000000:160750: f3 66 6f
kvm-18166 [001] 47471.398454: kvm_pio: pio_write at 0x1f0 size 2 count 1
kvm-18166 [001] 47471.398454: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-18166 [001] 47471.398456: kvm_entry: vcpu 0
kvm-18166 [001] 47471.398456: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753
kvm-18166 [001] 47471.398457: kvm_emulate_insn: 80000000:160750: f3 66 6f
kvm-18166 [001] 47471.398458: kvm_pio: pio_write at 0x1f0 size 2 count 1
The same, on 2.6.32:
kvm-2937 [001] 310.808825: kvm_entry: vcpu 0
kvm-2937 [001] 310.808828: kvm_exit: reason UNKNOWN rip 0x107c08
kvm-2937 [001] 310.808829: kvm_pio: pio_write at 0x21 size 1 count 1
kvm-2937 [001] 310.808829: kvm_entry: vcpu 0
kvm-2937 [001] 310.808830: kvm_exit: reason UNKNOWN rip 0x107c11
kvm-2937 [001] 310.808830: kvm_pio: pio_write at 0xa1 size 1 count 1
kvm-2937 [001] 310.808831: kvm_entry: vcpu 0
kvm-2937 [001] 310.808832: kvm_exit: reason UNKNOWN rip 0x107a9d
kvm-2937 [001] 310.808832: kvm_pio: pio_write at 0x21 size 1 count 1
kvm-2937 [001] 310.808832: kvm_entry: vcpu 0
kvm-2937 [001] 310.808833: kvm_exit: reason UNKNOWN rip 0x107aa6
kvm-2937 [001] 310.808833: kvm_pio: pio_write at 0xa1 size 1 count 1
kvm-2937 [001] 310.808833: kvm_entry: vcpu 0
kvm-2937 [001] 310.808846: kvm_exit: reason UNKNOWN rip 0x107c08
kvm-2937 [001] 310.808846: kvm_pio: pio_write at 0x21 size 1 count 1
kvm-2937 [001] 310.808847: kvm_entry: vcpu 0
kvm-2937 [001] 310.808847: kvm_exit: reason UNKNOWN rip 0x107c11
kvm-2937 [001] 310.808847: kvm_pio: pio_write at 0xa1 size 1 count 1
kvm-2937 [001] 310.808848: kvm_entry: vcpu 0
kvm-2937 [001] 310.808849: kvm_exit: reason UNKNOWN rip 0x107a9d
kvm-2937 [001] 310.808849: kvm_pio: pio_write at 0x21 size 1 count 1
kvm-2937 [001] 310.808849: kvm_entry: vcpu 0
kvm-2937 [001] 310.808850: kvm_exit: reason UNKNOWN rip 0x107aa6
kvm-2937 [001] 310.808850: kvm_pio: pio_write at 0xa1 size 1 count 1
kvm-2937 [001] 310.808850: kvm_entry: vcpu 0
kvm-2937 [001] 310.808851: kvm_exit: reason UNKNOWN rip 0x107c08
kvm-2937 [001] 310.808851: kvm_pio: pio_write at 0x21 size 1 count 1
kvm-2937 [001] 310.808852: kvm_entry: vcpu 0
kvm-2937 [001] 310.808852: kvm_exit: reason UNKNOWN rip 0x107c11
kvm-2937 [001] 310.808853: kvm_pio: pio_write at 0xa1 size 1 count 1
kvm-2937 [001] 310.808853: kvm_entry: vcpu 0
kvm-2937 [001] 310.808854: kvm_exit: reason UNKNOWN rip 0x107a9d
kvm-2937 [001] 310.808854: kvm_pio: pio_write at 0x21 size 1 count 1
kvm-2937 [001] 310.808854: kvm_entry: vcpu 0
kvm-2937 [001] 310.808855: kvm_exit: reason UNKNOWN rip 0x107aa6
kvm-2937 [001] 310.808855: kvm_pio: pio_write at 0xa1 size 1 count 1
kvm-2937 [001] 310.808856: kvm_entry: vcpu 0
kvm-2937 [001] 310.808856: kvm_exit: reason UNKNOWN rip 0x107c08
kvm-2937 [001] 310.808856: kvm_pio: pio_write at 0x21 size 1 count 1
kvm-2937 [001] 310.808857: kvm_entry: vcpu 0
kvm-2937 [001] 310.808858: kvm_exit: reason UNKNOWN rip 0x107c11
kvm-2937 [001] 310.808858: kvm_pio: pio_write at 0xa1 size 1 count 1
kvm-2937 [001] 310.808858: kvm_entry: vcpu 0
kvm-2937 [001] 310.808859: kvm_exit: reason UNKNOWN rip 0x107a9d
kvm-2937 [001] 310.808859: kvm_pio: pio_write at 0x21 size 1 count 1
kvm-2937 [001] 310.808860: kvm_entry: vcpu 0
kvm-2937 [001] 310.808860: kvm_exit: reason UNKNOWN rip 0x107aa6
kvm-2937 [001] 310.808861: kvm_pio: pio_write at 0xa1 size 1 count 1
kvm-2937 [001] 310.808861: kvm_entry: vcpu 0
kvm-2937 [001] 310.808866: kvm_exit: reason UNKNOWN rip 0x107c08
kvm-2937 [001] 310.808866: kvm_pio: pio_write at 0x21 size 1 count 1
kvm-2937 [001] 310.808866: kvm_entry: vcpu 0
kvm-2937 [001] 310.808867: kvm_exit: reason UNKNOWN rip 0x107c11
kvm-2937 [001] 310.808868: kvm_pio: pio_write at 0xa1 size 1 count 1
kvm-2937 [001] 310.808869: kvm_entry: vcpu 0
kvm-2937 [001] 310.808870: kvm_exit: reason UNKNOWN rip 0x107a9d
kvm-2937 [001] 310.808870: kvm_pio: pio_write at 0x21 size 1 count 1
kvm-2937 [001] 310.808870: kvm_entry: vcpu 0
kvm-2937 [001] 310.808871: kvm_exit: reason UNKNOWN rip 0x107aa6
kvm-2937 [001] 310.808871: kvm_pio: pio_write at 0xa1 size 1 count 1
kvm-2937 [001] 310.808871: kvm_entry: vcpu 0
kvm-2937 [001] 310.808901: kvm_exit: reason UNKNOWN rip 0x107c08
kvm-2937 [001] 310.808901: kvm_pio: pio_write at 0x21 size 1 count 1
kvm-2937 [001] 310.808902: kvm_entry: vcpu 0
kvm-2937 [001] 310.808903: kvm_exit: reason UNKNOWN rip 0x107c11
kvm-2937 [001] 310.808903: kvm_pio: pio_write at 0xa1 size 1 count 1
kvm-2937 [001] 310.808903: kvm_entry: vcpu 0
kvm-2937 [001] 310.808904: kvm_exit: reason UNKNOWN rip 0x107a9d
kvm-2937 [001] 310.808904: kvm_pio: pio_write at 0x21 size 1 count 1
kvm-2937 [001] 310.808905: kvm_entry: vcpu 0
kvm-2937 [001] 310.808905: kvm_exit: reason UNKNOWN rip 0x107aa6
kvm-2937 [001] 310.808905: kvm_pio: pio_write at 0xa1 size 1 count 1
kvm-2937 [001] 310.808906: kvm_entry: vcpu 0
kvm-2937 [001] 310.808914: kvm_exit: reason UNKNOWN rip 0x107c08
kvm-2937 [001] 310.808914: kvm_pio: pio_write at 0x21 size 1 count 1
kvm-2937 [001] 310.808915: kvm_entry: vcpu 0
kvm-2937 [001] 310.808916: kvm_exit: reason UNKNOWN rip 0x107c11
kvm-2937 [001] 310.808916: kvm_pio: pio_write at 0xa1 size 1 count 1
kvm-2937 [001] 310.808916: kvm_entry: vcpu 0
kvm-2937 [001] 310.808917: kvm_exit: reason UNKNOWN rip 0x107a9d
kvm-2937 [001] 310.808917: kvm_pio: pio_write at 0x21 size 1 count 1
kvm-2937 [001] 310.808917: kvm_entry: vcpu 0
kvm-2937 [001] 310.808918: kvm_exit: reason UNKNOWN rip 0x107aa6
kvm-2937 [001] 310.808918: kvm_pio: pio_write at 0xa1 size 1 count 1
kvm-2937 [001] 310.808919: kvm_entry: vcpu 0
kvm-2937 [001] 310.808919: kvm_exit: reason UNKNOWN rip 0x107c08
kvm-2937 [001] 310.808920: kvm_pio: pio_write at 0x21 size 1 count 1
kvm-2937 [001] 310.808920: kvm_entry: vcpu 0
kvm-2937 [001] 310.808921: kvm_exit: reason UNKNOWN rip 0x107c11
kvm-2937 [001] 310.808921: kvm_pio: pio_write at 0xa1 size 1 count 1
kvm-2937 [001] 310.808921: kvm_entry: vcpu 0
kvm-2937 [001] 310.808922: kvm_exit: reason UNKNOWN rip 0x107a9d
kvm-2937 [001] 310.808922: kvm_pio: pio_write at 0x21 size 1 count 1
kvm-2937 [001] 310.808922: kvm_entry: vcpu 0
kvm-2937 [001] 310.808923: kvm_exit: reason UNKNOWN rip 0x107aa6
kvm-2937 [001] 310.808923: kvm_pio: pio_write at 0xa1 size 1 count 1
kvm-2937 [001] 310.808924: kvm_entry: vcpu 0
kvm-2937 [001] 310.808924: kvm_exit: reason UNKNOWN rip 0x107c08
kvm-2937 [001] 310.808925: kvm_pio: pio_write at 0x21 size 1 count 1
kvm-2937 [001] 310.808925: kvm_entry: vcpu 0
kvm-2937 [001] 310.808926: kvm_exit: reason UNKNOWN rip 0x107c11
kvm-2937 [001] 310.808926: kvm_pio: pio_write at 0xa1 size 1 count 1
kvm-2937 [001] 310.808926: kvm_entry: vcpu 0
kvm-2937 [001] 310.808927: kvm_exit: reason UNKNOWN rip 0x107a9d
kvm-2937 [001] 310.808927: kvm_pio: pio_write at 0x21 size 1 count 1
kvm-2937 [001] 310.808927: kvm_entry: vcpu 0
kvm-2937 [001] 310.808928: kvm_exit: reason UNKNOWN rip 0x107aa6
kvm-2937 [001] 310.808928: kvm_pio: pio_write at 0xa1 size 1 count 1
kvm-2937 [001] 310.808929: kvm_entry: vcpu 0
kvm-2937 [001] 310.808938: kvm_exit: reason UNKNOWN rip 0x150b17
kvm-2937 [001] 310.808939: kvm_pio: pio_write at 0xa1 size 1 count 1
kvm-2937 [001] 310.808939: kvm_entry: vcpu 0
kvm-2937 [001] 310.808940: kvm_exit: reason UNKNOWN rip 0x1633cc
kvm-2937 [001] 310.808940: kvm_pio: pio_write at 0x1f6 size 1 count 1
kvm-2937 [001] 310.808943: kvm_entry: vcpu 0
Gleb, Avi -- what we're missing here?
It does not look like the problem we have with FreeBSD,
this is apparently something different...
Thank you!
/mjt
next reply other threads:[~2011-07-18 20:16 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-07-18 20:16 Michael Tokarev [this message]
2011-07-19 9:14 ` major slowdown in i/o (due to high CPU usage) from 2.6.32 to current Avi Kivity
2011-07-19 9:16 ` Gleb Natapov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4E249499.4050502@msgid.tls.msk.ru \
--to=mjt@tls.msk.ru \
--cc=kvm@vger.kernel.org \
--cc=srs@kth.se \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox