[PATCH V4 net-next 0/3] vhost_net tx batching

[PATCH V4 net-next 0/3] vhost_net tx batching

[Jason Wang]

Hi:

This series tries to implement tx batching support for vhost. This was
done by using MSG_MORE as a hint for under layer socket. The backend
(e.g tap) can then batch the packets temporarily in a list and
submit it all once the number of bacthed exceeds a limitation.

Tests shows obvious improvement on guest pktgen over over
mlx4(noqueue) on host:

                                     Mpps  -+%
        rx-frames = 0                0.91  +0%
        rx-frames = 4                1.00  +9.8%
        rx-frames = 8                1.00  +9.8%
        rx-frames = 16               1.01  +10.9%
        rx-frames = 32               1.07  +17.5%
        rx-frames = 48               1.07  +17.5%
        rx-frames = 64               1.08  +18.6%
        rx-frames = 64 (no MSG_MORE) 0.91  +0%

Changes from V3:
- use ethtool instead of module parameter to control the maximum
  number of batched packets
- avoid overhead when MSG_MORE were not set and no packet queued

Changes from V2:
- remove uselss queue limitation check (and we don't drop any packet now)

Changes from V1:
- drop NAPI handler since we don't use NAPI now
- fix the issues that may exceeds max pending of zerocopy
- more improvement on available buffer detection
- move the limitation of batched pacekts from vhost to tuntap

Please review.

Thanks

Jason Wang (3):
  vhost: better detection of available buffers
  vhost_net: tx batching
  tun: rx batching

 drivers/net/tun.c     | 76 +++++++++++++++++++++++++++++++++++++++++++++++----
 drivers/vhost/net.c   | 23 ++++++++++++++--
 drivers/vhost/vhost.c |  8 ++++--
 3 files changed, 96 insertions(+), 11 deletions(-)

-- 
2.7.4

[PATCH V4 net-next 1/3] vhost: better detection of available buffers

[Jason Wang]

This patch tries to do several tweaks on vhost_vq_avail_empty() for a
better performance:

- check cached avail index first which could avoid userspace memory access.
- using unlikely() for the failure of userspace access
- check vq->last_avail_idx instead of cached avail index as the last
  step.

This patch is need for batching supports which needs to peek whether
or not there's still available buffers in the ring.

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/vhost/vhost.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index d643260..9f11838 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -2241,11 +2241,15 @@ bool vhost_vq_avail_empty(struct vhost_dev *dev, struct vhost_virtqueue *vq)
 	__virtio16 avail_idx;
 	int r;
 
+	if (vq->avail_idx != vq->last_avail_idx)
+		return false;
+
 	r = vhost_get_user(vq, avail_idx, &vq->avail->idx);
-	if (r)
+	if (unlikely(r))
 		return false;
+	vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
 
-	return vhost16_to_cpu(vq, avail_idx) == vq->avail_idx;
+	return vq->avail_idx == vq->last_avail_idx;
 }
 EXPORT_SYMBOL_GPL(vhost_vq_avail_empty);
 
-- 
2.7.4

Re: [PATCH V4 net-next 1/3] vhost: better detection of available buffers

[Michael S. Tsirkin]

On Fri, Jan 06, 2017 at 10:13:15AM +0800, Jason Wang wrote:
> This patch tries to do several tweaks on vhost_vq_avail_empty() for a
> better performance:
> 
> - check cached avail index first which could avoid userspace memory access.
> - using unlikely() for the failure of userspace access
> - check vq->last_avail_idx instead of cached avail index as the last
>   step.
> 
> This patch is need for batching supports which needs to peek whether
> or not there's still available buffers in the ring.
> 
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> ---
>  drivers/vhost/vhost.c | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index d643260..9f11838 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -2241,11 +2241,15 @@ bool vhost_vq_avail_empty(struct vhost_dev *dev, struct vhost_virtqueue *vq)
>  	__virtio16 avail_idx;
>  	int r;
>  
> +	if (vq->avail_idx != vq->last_avail_idx)
> +		return false;
> +
>  	r = vhost_get_user(vq, avail_idx, &vq->avail->idx);
> -	if (r)
> +	if (unlikely(r))
>  		return false;
> +	vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
>  
> -	return vhost16_to_cpu(vq, avail_idx) == vq->avail_idx;
> +	return vq->avail_idx == vq->last_avail_idx;
>  }
>  EXPORT_SYMBOL_GPL(vhost_vq_avail_empty);

So again, this did not address the issue I pointed out in v1:
if we have 1 buffer in RX queue and
that is not enough to store the whole packet,
vhost_vq_avail_empty returns false, then we re-read
the descriptors again and again.

You have saved a single index access but not the more expensive
descriptor access.

I think that a way to address this could be to have this
return current index for the caller. Then as long as that
index isn't changed, you don't poke at descriptor ring.

> -- 
> 2.7.4

Re: [PATCH V4 net-next 1/3] vhost: better detection of available buffers

[Jason Wang]

On 2017年01月07日 03:55, Michael S. Tsirkin wrote:
> On Fri, Jan 06, 2017 at 10:13:15AM +0800, Jason Wang wrote:
>> This patch tries to do several tweaks on vhost_vq_avail_empty() for a
>> better performance:
>>
>> - check cached avail index first which could avoid userspace memory access.
>> - using unlikely() for the failure of userspace access
>> - check vq->last_avail_idx instead of cached avail index as the last
>>    step.
>>
>> This patch is need for batching supports which needs to peek whether
>> or not there's still available buffers in the ring.
>>
>> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>> ---
>>   drivers/vhost/vhost.c | 8 ++++++--
>>   1 file changed, 6 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
>> index d643260..9f11838 100644
>> --- a/drivers/vhost/vhost.c
>> +++ b/drivers/vhost/vhost.c
>> @@ -2241,11 +2241,15 @@ bool vhost_vq_avail_empty(struct vhost_dev *dev, struct vhost_virtqueue *vq)
>>   	__virtio16 avail_idx;
>>   	int r;
>>   
>> +	if (vq->avail_idx != vq->last_avail_idx)
>> +		return false;
>> +
>>   	r = vhost_get_user(vq, avail_idx, &vq->avail->idx);
>> -	if (r)
>> +	if (unlikely(r))
>>   		return false;
>> +	vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
>>   
>> -	return vhost16_to_cpu(vq, avail_idx) == vq->avail_idx;
>> +	return vq->avail_idx == vq->last_avail_idx;
>>   }
>>   EXPORT_SYMBOL_GPL(vhost_vq_avail_empty);
> So again, this did not address the issue I pointed out in v1:
> if we have 1 buffer in RX queue and
> that is not enough to store the whole packet,
> vhost_vq_avail_empty returns false, then we re-read
> the descriptors again and again.
>
> You have saved a single index access but not the more expensive
> descriptor access.

Looks not, if I understand the code correctly, in this case, 
get_rx_bufs() will return zero, and we will try to enable rx kick and 
exit the loop.

Thanks

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V4 net-next 1/3] vhost: better detection of available buffers

[Michael S. Tsirkin]

On Mon, Jan 09, 2017 at 10:59:16AM +0800, Jason Wang wrote:
> 
> 
> On 2017年01月07日 03:55, Michael S. Tsirkin wrote:
> > On Fri, Jan 06, 2017 at 10:13:15AM +0800, Jason Wang wrote:
> > > This patch tries to do several tweaks on vhost_vq_avail_empty() for a
> > > better performance:
> > > 
> > > - check cached avail index first which could avoid userspace memory access.
> > > - using unlikely() for the failure of userspace access
> > > - check vq->last_avail_idx instead of cached avail index as the last
> > >    step.
> > > 
> > > This patch is need for batching supports which needs to peek whether
> > > or not there's still available buffers in the ring.
> > > 
> > > Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > ---
> > >   drivers/vhost/vhost.c | 8 ++++++--
> > >   1 file changed, 6 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> > > index d643260..9f11838 100644
> > > --- a/drivers/vhost/vhost.c
> > > +++ b/drivers/vhost/vhost.c
> > > @@ -2241,11 +2241,15 @@ bool vhost_vq_avail_empty(struct vhost_dev *dev, struct vhost_virtqueue *vq)
> > >   	__virtio16 avail_idx;
> > >   	int r;
> > > +	if (vq->avail_idx != vq->last_avail_idx)
> > > +		return false;
> > > +
> > >   	r = vhost_get_user(vq, avail_idx, &vq->avail->idx);
> > > -	if (r)
> > > +	if (unlikely(r))
> > >   		return false;
> > > +	vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
> > > -	return vhost16_to_cpu(vq, avail_idx) == vq->avail_idx;
> > > +	return vq->avail_idx == vq->last_avail_idx;
> > >   }
> > >   EXPORT_SYMBOL_GPL(vhost_vq_avail_empty);
> > So again, this did not address the issue I pointed out in v1:
> > if we have 1 buffer in RX queue and
> > that is not enough to store the whole packet,
> > vhost_vq_avail_empty returns false, then we re-read
> > the descriptors again and again.
> > 
> > You have saved a single index access but not the more expensive
> > descriptor access.
> 
> Looks not, if I understand the code correctly, in this case, get_rx_bufs()
> will return zero, and we will try to enable rx kick and exit the loop.
> 
> Thanks

I mean this:

                while (vhost_can_busy_poll(vq->dev, endtime) &&
                       vhost_vq_avail_empty(vq->dev, vq))
                        cpu_relax();
                preempt_enable();
                r = vhost_get_vq_desc(vq, vq->iov, ARRAY_SIZE(vq->iov),
                                      out_num, in_num, NULL, NULL);


vhost_vq_avail_empty returns false so we break out of the loop
and call vhost_get_vq_desc.


-- 
MST
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V4 net-next 1/3] vhost: better detection of available buffers

[Jason Wang]

On 2017年01月10日 07:10, Michael S. Tsirkin wrote:
> On Mon, Jan 09, 2017 at 10:59:16AM +0800, Jason Wang wrote:
>>
>> On 2017年01月07日 03:55, Michael S. Tsirkin wrote:
>>> On Fri, Jan 06, 2017 at 10:13:15AM +0800, Jason Wang wrote:
>>>> This patch tries to do several tweaks on vhost_vq_avail_empty() for a
>>>> better performance:
>>>>
>>>> - check cached avail index first which could avoid userspace memory access.
>>>> - using unlikely() for the failure of userspace access
>>>> - check vq->last_avail_idx instead of cached avail index as the last
>>>>     step.
>>>>
>>>> This patch is need for batching supports which needs to peek whether
>>>> or not there's still available buffers in the ring.
>>>>
>>>> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
>>>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>>>> ---
>>>>    drivers/vhost/vhost.c | 8 ++++++--
>>>>    1 file changed, 6 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
>>>> index d643260..9f11838 100644
>>>> --- a/drivers/vhost/vhost.c
>>>> +++ b/drivers/vhost/vhost.c
>>>> @@ -2241,11 +2241,15 @@ bool vhost_vq_avail_empty(struct vhost_dev *dev, struct vhost_virtqueue *vq)
>>>>    	__virtio16 avail_idx;
>>>>    	int r;
>>>> +	if (vq->avail_idx != vq->last_avail_idx)
>>>> +		return false;
>>>> +
>>>>    	r = vhost_get_user(vq, avail_idx, &vq->avail->idx);
>>>> -	if (r)
>>>> +	if (unlikely(r))
>>>>    		return false;
>>>> +	vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
>>>> -	return vhost16_to_cpu(vq, avail_idx) == vq->avail_idx;
>>>> +	return vq->avail_idx == vq->last_avail_idx;
>>>>    }
>>>>    EXPORT_SYMBOL_GPL(vhost_vq_avail_empty);
>>> So again, this did not address the issue I pointed out in v1:
>>> if we have 1 buffer in RX queue and
>>> that is not enough to store the whole packet,
>>> vhost_vq_avail_empty returns false, then we re-read
>>> the descriptors again and again.
>>>
>>> You have saved a single index access but not the more expensive
>>> descriptor access.
>> Looks not, if I understand the code correctly, in this case, get_rx_bufs()
>> will return zero, and we will try to enable rx kick and exit the loop.
>>
>> Thanks
> I mean this:
>
>                  while (vhost_can_busy_poll(vq->dev, endtime) &&
>                         vhost_vq_avail_empty(vq->dev, vq))
>                          cpu_relax();
>                  preempt_enable();
>                  r = vhost_get_vq_desc(vq, vq->iov, ARRAY_SIZE(vq->iov),
>                                        out_num, in_num, NULL, NULL);
>
>
> vhost_vq_avail_empty returns false so we break out of the loop
> and call vhost_get_vq_desc.
>
>

But this is the code for polling tx vq not rx I think?

Thanks

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V4 net-next 1/3] vhost: better detection of available buffers

[Michael S. Tsirkin]

On Tue, Jan 10, 2017 at 10:22:42AM +0800, Jason Wang wrote:
> 
> 
> On 2017年01月10日 07:10, Michael S. Tsirkin wrote:
> > On Mon, Jan 09, 2017 at 10:59:16AM +0800, Jason Wang wrote:
> > > 
> > > On 2017年01月07日 03:55, Michael S. Tsirkin wrote:
> > > > On Fri, Jan 06, 2017 at 10:13:15AM +0800, Jason Wang wrote:
> > > > > This patch tries to do several tweaks on vhost_vq_avail_empty() for a
> > > > > better performance:
> > > > > 
> > > > > - check cached avail index first which could avoid userspace memory access.
> > > > > - using unlikely() for the failure of userspace access
> > > > > - check vq->last_avail_idx instead of cached avail index as the last
> > > > >     step.
> > > > > 
> > > > > This patch is need for batching supports which needs to peek whether
> > > > > or not there's still available buffers in the ring.
> > > > > 
> > > > > Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > ---
> > > > >    drivers/vhost/vhost.c | 8 ++++++--
> > > > >    1 file changed, 6 insertions(+), 2 deletions(-)
> > > > > 
> > > > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> > > > > index d643260..9f11838 100644
> > > > > --- a/drivers/vhost/vhost.c
> > > > > +++ b/drivers/vhost/vhost.c
> > > > > @@ -2241,11 +2241,15 @@ bool vhost_vq_avail_empty(struct vhost_dev *dev, struct vhost_virtqueue *vq)
> > > > >    	__virtio16 avail_idx;
> > > > >    	int r;
> > > > > +	if (vq->avail_idx != vq->last_avail_idx)
> > > > > +		return false;
> > > > > +
> > > > >    	r = vhost_get_user(vq, avail_idx, &vq->avail->idx);
> > > > > -	if (r)
> > > > > +	if (unlikely(r))
> > > > >    		return false;
> > > > > +	vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
> > > > > -	return vhost16_to_cpu(vq, avail_idx) == vq->avail_idx;
> > > > > +	return vq->avail_idx == vq->last_avail_idx;
> > > > >    }
> > > > >    EXPORT_SYMBOL_GPL(vhost_vq_avail_empty);
> > > > So again, this did not address the issue I pointed out in v1:
> > > > if we have 1 buffer in RX queue and
> > > > that is not enough to store the whole packet,
> > > > vhost_vq_avail_empty returns false, then we re-read
> > > > the descriptors again and again.
> > > > 
> > > > You have saved a single index access but not the more expensive
> > > > descriptor access.
> > > Looks not, if I understand the code correctly, in this case, get_rx_bufs()
> > > will return zero, and we will try to enable rx kick and exit the loop.
> > > 
> > > Thanks
> > I mean this:
> > 
> >                  while (vhost_can_busy_poll(vq->dev, endtime) &&
> >                         vhost_vq_avail_empty(vq->dev, vq))
> >                          cpu_relax();
> >                  preempt_enable();
> >                  r = vhost_get_vq_desc(vq, vq->iov, ARRAY_SIZE(vq->iov),
> >                                        out_num, in_num, NULL, NULL);
> > 
> > 
> > vhost_vq_avail_empty returns false so we break out of the loop
> > and call vhost_get_vq_desc.
> > 
> > 
> 
> But this is the code for polling tx vq not rx I think?
> 
> Thanks

Oh, right.
I'll re-read this.


-- 
MST
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[PATCH V4 net-next 2/3] vhost_net: tx batching

[Jason Wang]

This patch tries to utilize tuntap rx batching by peeking the tx
virtqueue during transmission, if there's more available buffers in
the virtqueue, set MSG_MORE flag for a hint for backend (e.g tuntap)
to batch the packets.

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/vhost/net.c | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)

diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index 5dc3465..c42e9c3 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -351,6 +351,15 @@ static int vhost_net_tx_get_vq_desc(struct vhost_net *net,
 	return r;
 }
 
+static bool vhost_exceeds_maxpend(struct vhost_net *net)
+{
+	struct vhost_net_virtqueue *nvq = &net->vqs[VHOST_NET_VQ_TX];
+	struct vhost_virtqueue *vq = &nvq->vq;
+
+	return (nvq->upend_idx + vq->num - VHOST_MAX_PEND) % UIO_MAXIOV
+		== nvq->done_idx;
+}
+
 /* Expects to be always run from workqueue - which acts as
  * read-size critical section for our kind of RCU. */
 static void handle_tx(struct vhost_net *net)
@@ -394,8 +403,7 @@ static void handle_tx(struct vhost_net *net)
 		/* If more outstanding DMAs, queue the work.
 		 * Handle upend_idx wrap around
 		 */
-		if (unlikely((nvq->upend_idx + vq->num - VHOST_MAX_PEND)
-			      % UIO_MAXIOV == nvq->done_idx))
+		if (unlikely(vhost_exceeds_maxpend(net)))
 			break;
 
 		head = vhost_net_tx_get_vq_desc(net, vq, vq->iov,
@@ -454,6 +462,16 @@ static void handle_tx(struct vhost_net *net)
 			msg.msg_control = NULL;
 			ubufs = NULL;
 		}
+
+		total_len += len;
+		if (total_len < VHOST_NET_WEIGHT &&
+		    !vhost_vq_avail_empty(&net->dev, vq) &&
+		    likely(!vhost_exceeds_maxpend(net))) {
+			msg.msg_flags |= MSG_MORE;
+		} else {
+			msg.msg_flags &= ~MSG_MORE;
+		}
+
 		/* TODO: Check specific error and bomb out unless ENOBUFS? */
 		err = sock->ops->sendmsg(sock, &msg, len);
 		if (unlikely(err < 0)) {
@@ -472,7 +490,6 @@ static void handle_tx(struct vhost_net *net)
 			vhost_add_used_and_signal(&net->dev, vq, head, 0);
 		else
 			vhost_zerocopy_signal_used(net, vq);
-		total_len += len;
 		vhost_net_tx_packet(net);
 		if (unlikely(total_len >= VHOST_NET_WEIGHT)) {
 			vhost_poll_queue(&vq->poll);
-- 
2.7.4

[PATCH V4 net-next 3/3] tun: rx batching

[Jason Wang]

We can only process 1 packet at one time during sendmsg(). This often
lead bad cache utilization under heavy load. So this patch tries to do
some batching during rx before submitting them to host network
stack. This is done through accepting MSG_MORE as a hint from
sendmsg() caller, if it was set, batch the packet temporarily in a
linked list and submit them all once MSG_MORE were cleared.

Tests were done by pktgen (burst=128) in guest over mlx4(noqueue) on host:

                                 Mpps  -+%
    rx-frames = 0                0.91  +0%
    rx-frames = 4                1.00  +9.8%
    rx-frames = 8                1.00  +9.8%
    rx-frames = 16               1.01  +10.9%
    rx-frames = 32               1.07  +17.5%
    rx-frames = 48               1.07  +17.5%
    rx-frames = 64               1.08  +18.6%
    rx-frames = 64 (no MSG_MORE) 0.91  +0%

User were allowed to change per device batched packets through
ethtool -C rx-frames. NAPI_POLL_WEIGHT were used as upper limitation
to prevent bh from being disabled too long.

Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/net/tun.c | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 70 insertions(+), 6 deletions(-)

diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index cd8e02c..6c93926 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -218,6 +218,7 @@ struct tun_struct {
 	struct list_head disabled;
 	void *security;
 	u32 flow_count;
+	u32 rx_batched;
 	struct tun_pcpu_stats __percpu *pcpu_stats;
 };
 
@@ -522,6 +523,7 @@ static void tun_queue_purge(struct tun_file *tfile)
 	while ((skb = skb_array_consume(&tfile->tx_array)) != NULL)
 		kfree_skb(skb);
 
+	skb_queue_purge(&tfile->sk.sk_write_queue);
 	skb_queue_purge(&tfile->sk.sk_error_queue);
 }
 
@@ -1140,10 +1142,45 @@ static struct sk_buff *tun_alloc_skb(struct tun_file *tfile,
 	return skb;
 }
 
+static void tun_rx_batched(struct tun_struct *tun, struct tun_file *tfile,
+			   struct sk_buff *skb, int more)
+{
+	struct sk_buff_head *queue = &tfile->sk.sk_write_queue;
+	struct sk_buff_head process_queue;
+	u32 rx_batched = tun->rx_batched;
+	bool rcv = false;
+
+	if (!rx_batched || (!more && skb_queue_empty(queue))) {
+		local_bh_disable();
+		netif_receive_skb(skb);
+		local_bh_enable();
+		return;
+	}
+
+	spin_lock(&queue->lock);
+	if (!more || skb_queue_len(queue) == rx_batched) {
+		__skb_queue_head_init(&process_queue);
+		skb_queue_splice_tail_init(queue, &process_queue);
+		rcv = true;
+	} else {
+		__skb_queue_tail(queue, skb);
+	}
+	spin_unlock(&queue->lock);
+
+	if (rcv) {
+		struct sk_buff *nskb;
+		local_bh_disable();
+		while ((nskb = __skb_dequeue(&process_queue)))
+			netif_receive_skb(nskb);
+		netif_receive_skb(skb);
+		local_bh_enable();
+	}
+}
+
 /* Get packet from user space buffer */
 static ssize_t tun_get_user(struct tun_struct *tun, struct tun_file *tfile,
 			    void *msg_control, struct iov_iter *from,
-			    int noblock)
+			    int noblock, bool more)
 {
 	struct tun_pi pi = { 0, cpu_to_be16(ETH_P_IP) };
 	struct sk_buff *skb;
@@ -1283,10 +1320,9 @@ static ssize_t tun_get_user(struct tun_struct *tun, struct tun_file *tfile,
 	skb_probe_transport_header(skb, 0);
 
 	rxhash = skb_get_hash(skb);
+
 #ifndef CONFIG_4KSTACKS
-	local_bh_disable();
-	netif_receive_skb(skb);
-	local_bh_enable();
+	tun_rx_batched(tun, tfile, skb, more);
 #else
 	netif_rx_ni(skb);
 #endif
@@ -1312,7 +1348,8 @@ static ssize_t tun_chr_write_iter(struct kiocb *iocb, struct iov_iter *from)
 	if (!tun)
 		return -EBADFD;
 
-	result = tun_get_user(tun, tfile, NULL, from, file->f_flags & O_NONBLOCK);
+	result = tun_get_user(tun, tfile, NULL, from,
+			      file->f_flags & O_NONBLOCK, false);
 
 	tun_put(tun);
 	return result;
@@ -1570,7 +1607,8 @@ static int tun_sendmsg(struct socket *sock, struct msghdr *m, size_t total_len)
 		return -EBADFD;
 
 	ret = tun_get_user(tun, tfile, m->msg_control, &m->msg_iter,
-			   m->msg_flags & MSG_DONTWAIT);
+			   m->msg_flags & MSG_DONTWAIT,
+			   m->msg_flags & MSG_MORE);
 	tun_put(tun);
 	return ret;
 }
@@ -1771,6 +1809,7 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
 		tun->align = NET_SKB_PAD;
 		tun->filter_attached = false;
 		tun->sndbuf = tfile->socket.sk->sk_sndbuf;
+		tun->rx_batched = 0;
 
 		tun->pcpu_stats = netdev_alloc_pcpu_stats(struct tun_pcpu_stats);
 		if (!tun->pcpu_stats) {
@@ -2439,6 +2478,29 @@ static void tun_set_msglevel(struct net_device *dev, u32 value)
 #endif
 }
 
+static int tun_get_coalesce(struct net_device *dev,
+			    struct ethtool_coalesce *ec)
+{
+	struct tun_struct *tun = netdev_priv(dev);
+
+	ec->rx_max_coalesced_frames = tun->rx_batched;
+
+	return 0;
+}
+
+static int tun_set_coalesce(struct net_device *dev,
+			    struct ethtool_coalesce *ec)
+{
+	struct tun_struct *tun = netdev_priv(dev);
+
+	if (ec->rx_max_coalesced_frames > NAPI_POLL_WEIGHT)
+		return -EINVAL;
+
+	tun->rx_batched = ec->rx_max_coalesced_frames;
+
+	return 0;
+}
+
 static const struct ethtool_ops tun_ethtool_ops = {
 	.get_settings	= tun_get_settings,
 	.get_drvinfo	= tun_get_drvinfo,
@@ -2446,6 +2508,8 @@ static const struct ethtool_ops tun_ethtool_ops = {
 	.set_msglevel	= tun_set_msglevel,
 	.get_link	= ethtool_op_get_link,
 	.get_ts_info	= ethtool_op_get_ts_info,
+	.get_coalesce   = tun_get_coalesce,
+	.set_coalesce   = tun_set_coalesce,
 };
 
 static int tun_queue_resize(struct tun_struct *tun)
-- 
2.7.4

Re: [PATCH V4 net-next 3/3] tun: rx batching

[Michael S. Tsirkin]

On Fri, Jan 06, 2017 at 10:13:17AM +0800, Jason Wang wrote:
> We can only process 1 packet at one time during sendmsg(). This often
> lead bad cache utilization under heavy load. So this patch tries to do
> some batching during rx before submitting them to host network
> stack. This is done through accepting MSG_MORE as a hint from
> sendmsg() caller, if it was set, batch the packet temporarily in a
> linked list and submit them all once MSG_MORE were cleared.
> 
> Tests were done by pktgen (burst=128) in guest over mlx4(noqueue) on host:
> 
>                                  Mpps  -+%
>     rx-frames = 0                0.91  +0%
>     rx-frames = 4                1.00  +9.8%
>     rx-frames = 8                1.00  +9.8%
>     rx-frames = 16               1.01  +10.9%
>     rx-frames = 32               1.07  +17.5%
>     rx-frames = 48               1.07  +17.5%
>     rx-frames = 64               1.08  +18.6%
>     rx-frames = 64 (no MSG_MORE) 0.91  +0%
> 
> User were allowed to change per device batched packets through
> ethtool -C rx-frames. NAPI_POLL_WEIGHT were used as upper limitation
> to prevent bh from being disabled too long.
> 
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> ---
>  drivers/net/tun.c | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++-----
>  1 file changed, 70 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/net/tun.c b/drivers/net/tun.c
> index cd8e02c..6c93926 100644
> --- a/drivers/net/tun.c
> +++ b/drivers/net/tun.c
> @@ -218,6 +218,7 @@ struct tun_struct {
>  	struct list_head disabled;
>  	void *security;
>  	u32 flow_count;
> +	u32 rx_batched;
>  	struct tun_pcpu_stats __percpu *pcpu_stats;
>  };
>  
> @@ -522,6 +523,7 @@ static void tun_queue_purge(struct tun_file *tfile)
>  	while ((skb = skb_array_consume(&tfile->tx_array)) != NULL)
>  		kfree_skb(skb);
>  
> +	skb_queue_purge(&tfile->sk.sk_write_queue);
>  	skb_queue_purge(&tfile->sk.sk_error_queue);
>  }
>  
> @@ -1140,10 +1142,45 @@ static struct sk_buff *tun_alloc_skb(struct tun_file *tfile,
>  	return skb;
>  }
>  
> +static void tun_rx_batched(struct tun_struct *tun, struct tun_file *tfile,
> +			   struct sk_buff *skb, int more)
> +{
> +	struct sk_buff_head *queue = &tfile->sk.sk_write_queue;
> +	struct sk_buff_head process_queue;
> +	u32 rx_batched = tun->rx_batched;
> +	bool rcv = false;
> +
> +	if (!rx_batched || (!more && skb_queue_empty(queue))) {
> +		local_bh_disable();
> +		netif_receive_skb(skb);
> +		local_bh_enable();
> +		return;
> +	}
> +
> +	spin_lock(&queue->lock);
> +	if (!more || skb_queue_len(queue) == rx_batched) {
> +		__skb_queue_head_init(&process_queue);
> +		skb_queue_splice_tail_init(queue, &process_queue);
> +		rcv = true;
> +	} else {
> +		__skb_queue_tail(queue, skb);
> +	}
> +	spin_unlock(&queue->lock);
> +
> +	if (rcv) {
> +		struct sk_buff *nskb;
> +		local_bh_disable();
> +		while ((nskb = __skb_dequeue(&process_queue)))
> +			netif_receive_skb(nskb);
> +		netif_receive_skb(skb);
> +		local_bh_enable();
> +	}
> +}
> +
>  /* Get packet from user space buffer */
>  static ssize_t tun_get_user(struct tun_struct *tun, struct tun_file *tfile,
>  			    void *msg_control, struct iov_iter *from,
> -			    int noblock)
> +			    int noblock, bool more)
>  {
>  	struct tun_pi pi = { 0, cpu_to_be16(ETH_P_IP) };
>  	struct sk_buff *skb;
> @@ -1283,10 +1320,9 @@ static ssize_t tun_get_user(struct tun_struct *tun, struct tun_file *tfile,
>  	skb_probe_transport_header(skb, 0);
>  
>  	rxhash = skb_get_hash(skb);
> +
>  #ifndef CONFIG_4KSTACKS
> -	local_bh_disable();
> -	netif_receive_skb(skb);
> -	local_bh_enable();
> +	tun_rx_batched(tun, tfile, skb, more);
>  #else
>  	netif_rx_ni(skb);
>  #endif
> @@ -1312,7 +1348,8 @@ static ssize_t tun_chr_write_iter(struct kiocb *iocb, struct iov_iter *from)
>  	if (!tun)
>  		return -EBADFD;
>  
> -	result = tun_get_user(tun, tfile, NULL, from, file->f_flags & O_NONBLOCK);
> +	result = tun_get_user(tun, tfile, NULL, from,
> +			      file->f_flags & O_NONBLOCK, false);
>  
>  	tun_put(tun);
>  	return result;
> @@ -1570,7 +1607,8 @@ static int tun_sendmsg(struct socket *sock, struct msghdr *m, size_t total_len)
>  		return -EBADFD;
>  
>  	ret = tun_get_user(tun, tfile, m->msg_control, &m->msg_iter,
> -			   m->msg_flags & MSG_DONTWAIT);
> +			   m->msg_flags & MSG_DONTWAIT,
> +			   m->msg_flags & MSG_MORE);
>  	tun_put(tun);
>  	return ret;
>  }
> @@ -1771,6 +1809,7 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
>  		tun->align = NET_SKB_PAD;
>  		tun->filter_attached = false;
>  		tun->sndbuf = tfile->socket.sk->sk_sndbuf;
> +		tun->rx_batched = 0;
>  
>  		tun->pcpu_stats = netdev_alloc_pcpu_stats(struct tun_pcpu_stats);
>  		if (!tun->pcpu_stats) {
> @@ -2439,6 +2478,29 @@ static void tun_set_msglevel(struct net_device *dev, u32 value)
>  #endif
>  }
>  
> +static int tun_get_coalesce(struct net_device *dev,
> +			    struct ethtool_coalesce *ec)
> +{
> +	struct tun_struct *tun = netdev_priv(dev);
> +
> +	ec->rx_max_coalesced_frames = tun->rx_batched;
> +
> +	return 0;
> +}
> +
> +static int tun_set_coalesce(struct net_device *dev,
> +			    struct ethtool_coalesce *ec)
> +{
> +	struct tun_struct *tun = netdev_priv(dev);
> +
> +	if (ec->rx_max_coalesced_frames > NAPI_POLL_WEIGHT)
> +		return -EINVAL;

So what should userspace do? Keep trying until it succeeds?
I think it's better to just use NAPI_POLL_WEIGHT instead and DTRT here.

> +
> +	tun->rx_batched = ec->rx_max_coalesced_frames;
> +
> +	return 0;
> +}
> +
>  static const struct ethtool_ops tun_ethtool_ops = {
>  	.get_settings	= tun_get_settings,
>  	.get_drvinfo	= tun_get_drvinfo,
> @@ -2446,6 +2508,8 @@ static const struct ethtool_ops tun_ethtool_ops = {
>  	.set_msglevel	= tun_set_msglevel,
>  	.get_link	= ethtool_op_get_link,
>  	.get_ts_info	= ethtool_op_get_ts_info,
> +	.get_coalesce   = tun_get_coalesce,
> +	.set_coalesce   = tun_set_coalesce,
>  };
>  
>  static int tun_queue_resize(struct tun_struct *tun)
> -- 
> 2.7.4

Re: [PATCH V4 net-next 3/3] tun: rx batching

[Jason Wang]

On 2017年01月07日 03:47, Michael S. Tsirkin wrote:
>> +static int tun_get_coalesce(struct net_device *dev,
>> +			    struct ethtool_coalesce *ec)
>> +{
>> +	struct tun_struct *tun = netdev_priv(dev);
>> +
>> +	ec->rx_max_coalesced_frames = tun->rx_batched;
>> +
>> +	return 0;
>> +}
>> +
>> +static int tun_set_coalesce(struct net_device *dev,
>> +			    struct ethtool_coalesce *ec)
>> +{
>> +	struct tun_struct *tun = netdev_priv(dev);
>> +
>> +	if (ec->rx_max_coalesced_frames > NAPI_POLL_WEIGHT)
>> +		return -EINVAL;
> So what should userspace do? Keep trying until it succeeds?
> I think it's better to just use NAPI_POLL_WEIGHT instead and DTRT here.
>

Well, looking at how set_coalesce is implemented in other drivers, 
-EINVAL is usually used when user give a value that exceeds the 
limitation. For tuntap, what missed here is probably just a 
documentation for coalescing in tuntap.txt. (Or extend ethtool to return 
the max value). This seems much better than silently reduce the value to 
the limitation.

Thanks
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V4 net-next 3/3] tun: rx batching

[Michael S. Tsirkin]

On Mon, Jan 09, 2017 at 10:39:55AM +0800, Jason Wang wrote:
> 
> 
> On 2017年01月07日 03:47, Michael S. Tsirkin wrote:
> > > +static int tun_get_coalesce(struct net_device *dev,
> > > +			    struct ethtool_coalesce *ec)
> > > +{
> > > +	struct tun_struct *tun = netdev_priv(dev);
> > > +
> > > +	ec->rx_max_coalesced_frames = tun->rx_batched;
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +static int tun_set_coalesce(struct net_device *dev,
> > > +			    struct ethtool_coalesce *ec)
> > > +{
> > > +	struct tun_struct *tun = netdev_priv(dev);
> > > +
> > > +	if (ec->rx_max_coalesced_frames > NAPI_POLL_WEIGHT)
> > > +		return -EINVAL;
> > So what should userspace do? Keep trying until it succeeds?
> > I think it's better to just use NAPI_POLL_WEIGHT instead and DTRT here.
> > 
> 
> Well, looking at how set_coalesce is implemented in other drivers, -EINVAL
> is usually used when user give a value that exceeds the limitation. For
> tuntap, what missed here is probably just a documentation for coalescing in
> tuntap.txt. (Or extend ethtool to return the max value). This seems much
> better than silently reduce the value to the limitation.
> 
> Thanks


I don't think it's better, it's mostly that
1. there's a hardware limit so it does not change much
2. default is enabled and no one bothers changing

I don't see how will tuntap.txt help if we want to change it
in the future.

-- 
MST
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V4 net-next 3/3] tun: rx batching

[Jason Wang]

On 2017年01月10日 07:12, Michael S. Tsirkin wrote:
> On Mon, Jan 09, 2017 at 10:39:55AM +0800, Jason Wang wrote:
>> On 2017年01月07日 03:47, Michael S. Tsirkin wrote:
>>>> +static int tun_get_coalesce(struct net_device *dev,
>>>> +			    struct ethtool_coalesce *ec)
>>>> +{
>>>> +	struct tun_struct *tun = netdev_priv(dev);
>>>> +
>>>> +	ec->rx_max_coalesced_frames = tun->rx_batched;
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int tun_set_coalesce(struct net_device *dev,
>>>> +			    struct ethtool_coalesce *ec)
>>>> +{
>>>> +	struct tun_struct *tun = netdev_priv(dev);
>>>> +
>>>> +	if (ec->rx_max_coalesced_frames > NAPI_POLL_WEIGHT)
>>>> +		return -EINVAL;
>>> So what should userspace do? Keep trying until it succeeds?
>>> I think it's better to just use NAPI_POLL_WEIGHT instead and DTRT here.
>>>
>> Well, looking at how set_coalesce is implemented in other drivers, -EINVAL
>> is usually used when user give a value that exceeds the limitation. For
>> tuntap, what missed here is probably just a documentation for coalescing in
>> tuntap.txt. (Or extend ethtool to return the max value). This seems much
>> better than silently reduce the value to the limitation.
>>
>> Thanks
> I don't think it's better, it's mostly that
> 1. there's a hardware limit so it does not change much
> 2. default is enabled and no one bothers changing
>
> I don't see how will tuntap.txt help if we want to change it
> in the future.

Ok, so I will limit it to NAPI_POLL_WEIGHT if user gives a value that is 
greater than that.

Thanks
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization