Re: Poor network performance between DomU with multiqueue support

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Fri, Dec 05, 2014 at 03:20:55PM +0000, Zoltan Kiss wrote:
> 
> 
> On 04/12/14 14:31, Zhangleiqiang (Trump) wrote:
> >>-----Original Message-----
> >>From: Zoltan Kiss [mailto:zoltan.kiss@linaro.org]
> >>Sent: Thursday, December 04, 2014 9:35 PM
> >>To: Zhangleiqiang (Trump); Wei Liu; xen-devel@lists.xen.org
> >>Cc: Xiaoding (B); Zhuangyuxin; zhangleiqiang; Luohao (brian); Yuzhou (C)
> >>Subject: Re: [Xen-devel] Poor network performance between DomU with
> >>multiqueue support
> >>
> >>
> >>
> >>On 04/12/14 12:09, Zhangleiqiang (Trump) wrote:
> >>>>I think that's expected, because guest RX data path still uses
> >>>>grant_copy while
> >>>>>guest TX uses grant_map to do zero-copy transmit.
> >>>As I understand, the RX process is as follows:
> >>>1. Phy NIC receive packet
> >>>2. XEN Hypervisor trigger interrupt to Dom0 3. Dom0' s NIC driver do
> >>>the "RX" operation, and the packet is stored into SKB which is also
> >>>owned/shared with netback
> >>Not that easy. There is something between the NIC driver and netback which
> >>directs the packets, e.g. the old bridge driver, ovs, or the IP stack of the kernel.
> >>>4. NetBack notify netfront through event channel that a packet is
> >>>receiving 5. Netfront grant a buffer for receiving and notify netback
> >>>the GR (if using grant-resue mechanism, netfront just notify the GR to
> >>>netback) through IO Ring
> >>It looks a bit confusing in the code, but netfront put "requests" on the ring
> >>buffer, which contains the grant ref of the guest page where the backend can
> >>copy. When the packet comes, netback consumes these requests and send
> >>back a response telling the guest the grant copy of the packet finished, it can
> >>start handling the data. (sending a response means it's placing a response in
> >>the ring and trigger the event channel) And ideally netback should always have
> >>requests in the ring, so it doesn't have to wait for the guest to fill it up.
> >
> >>>6. NetBack do the grant_copy to copy packet from its SKB to the buffer
> >>>referenced by GR, and notify netfront through event channel 7.
> >>>Netfront copy the data from buffer to user-level app's SKB
> >>Or wherever that SKB should go, yes. Like with any received packet on a real
> >>network interface.
> >>>
> >>>Am I right? Why not using zero-copy transmit in guest RX data pash too ?
> >>Because that means you are mapping that memory to the guest, and you won't
> >>have any guarantee when the guest will release them. And netback can't just
> >>unmap them forcibly after a timeout, because finding a correct timeout value
> >>would be quite impossible.
> >>A malicious/buggy/overloaded guest can hold on to Dom0 memory indefinitely,
> >>but it even becomes worse if the memory came from another
> >>guest: you can't shutdown that guest for example, until all its memory is
> >>returned to him.
> >
> >Thanks for your detailed explanation about RX data path, I have get it, :)
> >
> >About the issue that poor performance between DomU to DomU, but high throughout between Dom0 to remote Dom0/DomU mentioned in my previous mail, do you have any idea about it?
> >
> >I am wondering if netfront/netback can be optimized to reach the 10Gbps throughout between DomUs running on different hosts connected with 10GE network. Currently, it seems like the TX is not the bottleneck, because we can reach the aggregate throughout of 9Gbps when sending packets from one DomU to other 3 DomUs running on different host. So I think the bottleneck maybe the RX, are you agreed with me?
> >
> >I am wondering what is the main reason that prevent RX to reach the higher throughout? Compared to KVM+virtio+vhost, which can reach high throughout, the RX has extra grantcopy operation, and the grantcopy operation may be one reason for it. Do you have any idea about it too?
> It's quite sure that the grant copy is the bottleneck for a single queue RX
> traffic. I don't know what's the plan to help that, currently only a faster
> CPU can help you with that.

Could the Intel QuickData help with that?
> 
> >
> >>
> >>Regards,
> >>
> >>Zoli
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel