Re: [PATCH v2 0/5] VSOCK: support mergeable rx buffer in vhost-vsock

["Michael S. Tsirkin" <mst@redhat.com>]

On Fri, Dec 14, 2018 at 06:24:40PM +0800, jiangyiwen wrote:
> On 2018/12/12 23:09, Michael S. Tsirkin wrote:
> > On Wed, Dec 12, 2018 at 05:25:50PM +0800, jiangyiwen wrote:
> >> Now vsock only support send/receive small packet, it can't achieve
> >> high performance. As previous discussed with Jason Wang, I revisit the
> >> idea of vhost-net about mergeable rx buffer and implement the mergeable
> >> rx buffer in vhost-vsock, it can allow big packet to be scattered in
> >> into different buffers and improve performance obviously.
> >>
> >> This series of patches mainly did three things：
> >> - mergeable buffer implementation
> >> - increase the max send pkt size
> >> - add used and signal guest in a batch
> >>
> >> And I write a tool to test the vhost-vsock performance, mainly send big
> >> packet(64K) included guest->Host and Host->Guest. I test performance
> >> independently and the result as follows:
> >>
> >> Before performance:
> >>               Single socket            Multiple sockets(Max Bandwidth)
> >> Guest->Host   ~400MB/s                 ~480MB/s
> >> Host->Guest   ~1450MB/s                ~1600MB/s
> >>
> >> After performance only use implement mergeable rx buffer:
> >>               Single socket            Multiple sockets(Max Bandwidth)
> >> Guest->Host   ~400MB/s                 ~480MB/s
> >> Host->Guest   ~1280MB/s                ~1350MB/s
> >>
> >> In this case, max send pkt size is still limited to 4K, so Host->Guest
> >> performance will worse than before.
> > 
> > It's concerning though, what if application sends small packets?
> > What is the source of the slowdown? Do you know?
> > 
> 
> Hi Michael,
> 
> To the two cases, I test the results included small and big packets as
> follows:
> 
> 64K packets performance comparison:
>                                               Single socket    Multiple sockets
> Host->Guest(before)                           1352.60MB/s      1436.33MB/s
> 
> 
> Host->Guest(only use mergeable rx buffer)     1290.08MB/s      1212.67MB/s
> 
> 4K packets performance comparison:
>                                               Single socket    Multiple sockets
> Host->Guest(before)                           535.47MB/s       688.67MB/s
> Host->Guest(only use mergeable rx buffer)     522.33MB/s       599.00MB/s
> 
> 3K packets performance comparison:
>                                               Single socket    Multiple sockets
> Host->Guest(before)                           359.74MB/s       442.00MB/s
> Host->Guest(only use mergeable rx buffer)     374.47MB/s       452.33MB/s
> 
> We can see an interesting thing, for 64K and 4K packets,
> using mergeable buffer has a poor performance, for 3K packet,
> both have the same performance.
> 
> I guess in mergeable mode, when host send a 4k packet to guest, we
> should call vhost_get_vq_desc() twice in host(hdr + 4k data),
> and in guest we also should call virtqueue_get_buf() twice. So
> when packet is smaller than (4k - hdr), it can be packed in a
> single page, so the performance is the same as before.
> 
> So in the mergeable mode, the performance may be
> worse in ((4k - hdr), 4k] than before.
> 
> Thanks,
> Yiwen.


The conclusion seems to be that mergeable buffers themselves
only hurt performance, but they allow batching which improves
performance. So let's add batching without mergeable buffers then?


> >> After performance increase the max send pkt size to 64K:
> >>               Single socket            Multiple sockets(Max Bandwidth)
> >> Guest->Host   ~1700MB/s                ~2900MB/s
> >> Host->Guest   ~1500MB/s                ~2440MB/s
> >>
> >> After performance all patches are used:
> >>               Single socket            Multiple sockets(Max Bandwidth)
> >> Guest->Host   ~1700MB/s                ~2900MB/s
> >> Host->Guest   ~1700MB/s                ~2900MB/s
> >>
> >> >From the test results, the performance is improved obviously, and guest
> >> memory will not be wasted.
> >>
> >> In addition, in order to support mergeable rx buffer in virtio-vsock,
> >> we need to add a qemu patch to support parse feature.
> >>
> >> ---
> >> v1 -> v2:
> >>  * Addressed comments from Jason Wang.
> >>  * Add performance test result independently.
> >>  * Use Skb_page_frag_refill() which can use high order page and reduce
> >>    the stress of page allocator.
> >>  * Still use fixed size(PAGE_SIZE) to fill rx buffer, because too small
> >>    size can't fill one full packet, we only 128 vq num now.
> >>  * Use iovec to replace buf in struct virtio_vsock_pkt, keep tx and rx
> >>    consistency.
> >>  * Add virtio_transport ops to get max pkt len, in order to be compatible
> >>    with old version.
> >> ---
> >>
> >> Yiwen Jiang (5):
> >>   VSOCK: support fill mergeable rx buffer in guest
> >>   VSOCK: support fill data to mergeable rx buffer in host
> >>   VSOCK: support receive mergeable rx buffer in guest
> >>   VSOCK: increase send pkt len in mergeable mode to improve performance
> >>   VSOCK: batch sending rx buffer to increase bandwidth
> >>
> >>  drivers/vhost/vsock.c                   | 183 ++++++++++++++++++++-----
> >>  include/linux/virtio_vsock.h            |  13 +-
> >>  include/uapi/linux/virtio_vsock.h       |   5 +
> >>  net/vmw_vsock/virtio_transport.c        | 229 +++++++++++++++++++++++++++-----
> >>  net/vmw_vsock/virtio_transport_common.c |  66 ++++++---
> >>  5 files changed, 411 insertions(+), 85 deletions(-)
> >>
> >> -- 
> >> 1.8.3.1
> > 
> > .
> > 
>