From mboxrd@z Thu Jan 1 00:00:00 1970 From: Wei Liu Subject: Re: Poor network performance between DomU with multiqueue support Date: Fri, 5 Dec 2014 12:42:33 +0000 Message-ID: <20141205124233.GD31446@zion.uk.xensource.com> References: <20141202110133.GA5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23931FA1@SZXEMA512-MBX.china.huawei.com> <20141202121151.GD5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393216B@SZXEMA512-MBX.china.huawei.com> <20141202155832.GH5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393301C@SZXEMA512-MBX.china.huawei.com> <20141204105021.GA16532@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23933CDA@SZXEMA512-MBX.china.huawei.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: <3A6795EA1206904E94BEC8EF9DF109AE23933CDA@SZXEMA512-MBX.china.huawei.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: "Zhangleiqiang (Trump)" Cc: "Luohao (brian)" , Wei Liu , Zhuangyuxin , zhangleiqiang , "Yuzhou (C)" , "xen-devel@lists.xen.org" , "Xiaoding (B)" List-Id: xen-devel@lists.xenproject.org On Fri, Dec 05, 2014 at 01:17:16AM +0000, Zhangleiqiang (Trump) wrote: [...] > > I think that's expected, because guest RX data path still uses grant_copy while > > guest TX uses grant_map to do zero-copy transmit. > > As far as I know, there are three main grant-related operations used in split device model: grant mapping, grant transfer and grant copy. > Grant transfer has not used now, and grant mapping and grant transfer both involve "TLB" refresh work for hypervisor, am I right? Or only grant transfer has this overhead? Transfer is not used so I can't tell. Grant unmap causes TLB flush. I saw in an email the other day XenServer folks has some planned improvement to avoid TLB flush in Xen to upstream in 4.6 window. I can't speak for sure it will get upstreamed as I don't work on that. > Does grant copy surely has more overhead than grant mapping? > At the very least the zero-copy TX path is faster than previous copying path. But speaking of the micro operation I'm not sure. There was once persistent map prototype netback / netfront that establishes a memory pool between FE and BE then use memcpy to copy data. Unfortunately that prototype was not done right so the result was not good. > >From the code, I see that in TX, netback will do gnttab_batch_copy as well as gnttab_map_refs: > > //netback.c:xenvif_tx_action > xenvif_tx_build_gops(queue, budget, &nr_cops, &nr_mops); > > if (nr_cops == 0) > return 0; > > gnttab_batch_copy(queue->tx_copy_ops, nr_cops); > if (nr_mops != 0) { > ret = gnttab_map_refs(queue->tx_map_ops, > NULL, > queue->pages_to_map, > nr_mops); > BUG_ON(ret); > } > > The copy is for the packet header. Mapping is for packet data. We need to copy header from guest so that it doesn't change under netback's feet. Wei.