Re: Regression in throughput between kvm guests over virtual bridge

[Jason Wang <jasowang@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 4108 bytes --]



On 2017年09月18日 11:13, Jason Wang wrote:
>
>
> On 2017年09月16日 03:19, Matthew Rosato wrote:
>>> It looks like vhost is slowed down for some reason which leads to more
>>> idle time on 4.13+VHOST_RX_BATCH=1. Appreciated if you can collect the
>>> perf.diff on host, one for rx and one for tx.
>>>
>> perf data below for the associated vhost threads, baseline=4.12,
>> delta1=4.13, delta2=4.13+VHOST_RX_BATCH=1
>>
>> Client vhost:
>>
>> 60.12%  -11.11%  -12.34%  [kernel.vmlinux]   [k] raw_copy_from_user
>> 13.76%   -1.28%   -0.74%  [kernel.vmlinux]   [k] get_page_from_freelist
>>   2.00%   +3.69%   +3.54%  [kernel.vmlinux]   [k] __wake_up_sync_key
>>   1.19%   +0.60%   +0.66%  [kernel.vmlinux]   [k] __alloc_pages_nodemask
>>   1.12%   +0.76%   +0.86%  [kernel.vmlinux]   [k] copy_page_from_iter
>>   1.09%   +0.28%   +0.35%  [vhost]            [k] vhost_get_vq_desc
>>   1.07%   +0.31%   +0.26%  [kernel.vmlinux]   [k] alloc_skb_with_frags
>>   0.94%   +0.42%   +0.65%  [kernel.vmlinux]   [k] alloc_pages_current
>>   0.91%   -0.19%   -0.18%  [kernel.vmlinux]   [k] memcpy
>>   0.88%   +0.26%   +0.30%  [kernel.vmlinux]   [k] __next_zones_zonelist
>>   0.85%   +0.05%   +0.12%  [kernel.vmlinux]   [k] iov_iter_advance
>>   0.79%   +0.09%   +0.19%  [vhost]            [k] __vhost_add_used_n
>>   0.74%                    [kernel.vmlinux]   [k] get_task_policy.part.7
>>   0.74%   -0.01%   -0.05%  [kernel.vmlinux]   [k] tun_net_xmit
>>   0.60%   +0.17%   +0.33%  [kernel.vmlinux]   [k] policy_nodemask
>>   0.58%   -0.15%   -0.12%  [ebtables]         [k] ebt_do_table
>>   0.52%   -0.25%   -0.22%  [kernel.vmlinux]   [k] __alloc_skb
>>     ...
>>   0.42%   +0.58%   +0.59%  [kernel.vmlinux]   [k] eventfd_signal
>>     ...
>>   0.32%   +0.96%   +0.93%  [kernel.vmlinux]   [k] finish_task_switch
>>     ...
>>           +1.50%   +1.16%  [kernel.vmlinux]   [k] get_task_policy.part.9
>>           +0.40%   +0.42%  [kernel.vmlinux]   [k] __skb_get_hash_symmetr
>>           +0.39%   +0.40%  [kernel.vmlinux]   [k] _copy_from_iter_full
>>           +0.24%   +0.23%  [vhost_net]        [k] vhost_net_buf_peek
>>
>> Server vhost:
>>
>> 61.93%  -10.72%  -10.91%  [kernel.vmlinux]   [k] raw_copy_to_user
>>   9.25%   +0.47%   +0.86%  [kernel.vmlinux]   [k] free_hot_cold_page
>>   5.16%   +1.41%   +1.57%  [vhost]            [k] vhost_get_vq_desc
>>   5.12%   -3.81%   -3.78%  [kernel.vmlinux]   [k] skb_release_data
>>   3.30%   +0.42%   +0.55%  [kernel.vmlinux]   [k] raw_copy_from_user
>>   1.29%   +2.20%   +2.28%  [kernel.vmlinux]   [k] copy_page_to_iter
>>   1.24%   +1.65%   +0.45%  [vhost_net]        [k] handle_rx
>>   1.08%   +3.03%   +2.85%  [kernel.vmlinux]   [k] __wake_up_sync_key
>>   0.96%   +0.70%   +1.10%  [vhost]            [k] translate_desc
>>   0.69%   -0.20%   -0.22%  [kernel.vmlinux]   [k] tun_do_read.part.10
>>   0.69%                    [kernel.vmlinux]   [k] tun_peek_len
>>   0.67%   +0.75%   +0.78%  [kernel.vmlinux]   [k] eventfd_signal
>>   0.52%   +0.96%   +0.98%  [kernel.vmlinux]   [k] finish_task_switch
>>   0.50%   +0.05%   +0.09%  [vhost]            [k] vhost_add_used_n
>>     ...
>>           +0.63%   +0.58%  [vhost_net]        [k] vhost_net_buf_peek
>>           +0.32%   +0.32%  [kernel.vmlinux]   [k] _copy_to_iter
>>           +0.19%   +0.19%  [kernel.vmlinux]   [k] __skb_get_hash_symmetr
>>           +0.11%   +0.21%  [vhost]            [k] vhost_umem_interval_tr
>>
>
> Looks like for some unknown reason which leads more wakeups.
>
> Could you please try to attached patch to see if it solves or mitigate 
> the issue?
>
> Thanks 

My bad, please try this.

Thanks

[-- Attachment #2: 0001-vhost_net-conditionally-enable-tx-polling.patch --]
[-- Type: text/x-patch, Size: 898 bytes --]

>From 8be3edfcd415ba6157ab34d250127c6f2b21ff5d Mon Sep 17 00:00:00 2001
From: Jason Wang <jasowang@redhat.com>
Date: Mon, 18 Sep 2017 10:56:30 +0800
Subject: [PATCH] vhost_net: conditionally enable tx polling

Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/vhost/net.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index 58585ec..2b308e0 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -471,6 +471,7 @@ static void handle_tx(struct vhost_net *net)
 		goto out;
 
 	vhost_disable_notify(&net->dev, vq);
+	vhost_net_disable_vq(net, vq);
 
 	hdr_size = nvq->vhost_hlen;
 	zcopy = nvq->ubufs;
@@ -562,6 +563,8 @@ static void handle_tx(struct vhost_net *net)
 					% UIO_MAXIOV;
 			}
 			vhost_discard_vq_desc(vq, 1);
+			if (err == -EAGAIN)
+				vhost_net_enable_vq(net, vq);
 			break;
 		}
 		if (err != len)
-- 
2.7.4