[Bug 118191] New: performance regression since dynamic halt-polling

[Bug 118191] New: performance regression since dynamic halt-polling

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=118191

            Bug ID: 118191
           Summary: performance regression since dynamic halt-polling
           Product: Virtualization
           Version: unspecified
    Kernel Version: >=4.2
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: kvm
          Assignee: virtualization_kvm@kernel-bugs.osdl.org
          Reporter: sp2.blub@speed.at
        Regression: No

Since commit aca6ff29c40 (KVM: dynamic halt-polling) a VM under network load
with virtio network produces extremely high cpu usage on the host.

Bisected on git://github.com/torvalds/linux master

Testcase:

Host: starting with the above mentioned commit (using a debian based linux
(PVE4.2))
Using iperf to test:
 $ iperf -us

Guest VM: qemu/kvm (running linux) with this network device:
 -netdev
type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on
 -device
virtio-net-pci,mac=32:32:37:39:33:62,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
Connecting to the iperf server via:
 $ iperf -uc 10.0.0.1 -b 100m

Behavior before the commit: ~60% cpu usage
After the commit: 100% cpu usage and much lower network throughput.

Related external threads:
http://pve.proxmox.com/pipermail/pve-user/2016-May/010302.html
https://forum.proxmox.com/threads/wrong-cpu-usage.27080/

The iperf result linked in the mailing list thread is the same I'm seeing in my
testcase after the commit.
(<https://gist.github.com/gilou/15b620a7a067fd1d58a7616942e025b4#file-perf_virtionet_4-4-txt>)
Whereas before the commit the KVM process iperf output starts with:
5.93% [kernel] [k] kvm_arch_vcpu_ioctl_run
4.92% [kernel] [k] vmx_vcpu_run
3.62% [kernel] [k] native_write_msr_safe
3.14% [kernel] [k] _raw_spin_lock_irqsave

And vhost-net with:
9.66% [kernel] [k] __br_fbd_get
4.55% [kernel] [k] copy_user_enhanced_fast_string
3.01% [kernel] [k] update_cfs_shares
2.43% [kernel] [k] __netif_receive_skb_core
2.25% [kernel] [k] vhost_worker

Loading the kvm module with the 2 new options introduced by the above commit
set to 0 reverts to the original CPU usage from before (which makes sense,
given that as far as I can tell this reverts to the old behavior).
  halt_poll_ns_grow=0 halt_poll_ns_shrink=0

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 118191] performance regression since dynamic halt-polling

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=118191

Wanpeng Li <wanpeng.li@hotmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |wanpeng.li@hotmail.com

--- Comment #1 from Wanpeng Li <wanpeng.li@hotmail.com> ---
Behavior before the commit: ~60% cpu usage
After the commit: 100% cpu 

===============================================

I think this is the right phenomena which you will observe, it doesn't
influence scheduling on host, the poll will stop immediately once another
candidate task appears.


usage and much lower network throughput.

==============================================

There is a trace off between throughput and latency, you can stop dynamic
halt-polling by the interfaces which you have already found for throughput
oriented workloads.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 118191] performance regression since dynamic halt-polling

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=118191

--- Comment #2 from Wanpeng Li <wanpeng.li@hotmail.com> ---
You can set halt_poll_ns to zero in order to stop dynamic halt-polling directly
instead of other interfaces.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.