Re: Poor network performance between DomU with multiqueue support

[Wei Liu <wei.liu2@citrix.com>]

On Fri, Dec 05, 2014 at 01:17:16AM +0000, Zhangleiqiang (Trump) wrote:
[...]
> > I think that's expected, because guest RX data path still uses grant_copy while
> > guest TX uses grant_map to do zero-copy transmit.
> 
> As far as I know, there are three main grant-related operations used in split device model: grant mapping, grant transfer and grant copy. 
> Grant transfer has not used now, and grant mapping and grant transfer both involve "TLB" refresh work for hypervisor, am I right?  Or only grant transfer has this overhead?

Transfer is not used so I can't tell. Grant unmap causes TLB flush.

I saw in an email the other day XenServer folks has some planned
improvement to avoid TLB flush in Xen to upstream in 4.6 window. I can't
speak for sure it will get upstreamed as I don't work on that.

> Does grant copy surely has more overhead than grant mapping? 
> 

At the very least the zero-copy TX path is faster than previous copying
path.

But speaking of the micro operation I'm not sure.

There was once persistent map prototype netback / netfront that
establishes a memory pool between FE and BE then use memcpy to copy
data. Unfortunately that prototype was not done right so the result was
not good.

> >From the code, I see that in TX, netback will do gnttab_batch_copy as well as gnttab_map_refs:
> 
> <code> //netback.c:xenvif_tx_action
> 	xenvif_tx_build_gops(queue, budget, &nr_cops, &nr_mops);
> 
> 	if (nr_cops == 0)
> 		return 0;
> 
> 	gnttab_batch_copy(queue->tx_copy_ops, nr_cops);
> 	if (nr_mops != 0) {
> 		ret = gnttab_map_refs(queue->tx_map_ops,
> 				      NULL,
> 				      queue->pages_to_map,
> 				      nr_mops);
> 		BUG_ON(ret);
> 	}
> </code>
> 

The copy is for the packet header. Mapping is for packet data.

We need to copy header from guest so that it doesn't change under
netback's feet.

Wei.