Re: qlen check in tun.c

[Jason Wang <jasowang@redhat.com>]

On 06/26/2013 01:32 PM, Jerry Chu wrote:
> On Tue, Jun 25, 2013 at 10:23 PM, Jason Wang <jasowang@redhat.com
> <mailto:jasowang@redhat.com>> wrote:
>
>     On 06/26/2013 06:23 AM, Jerry Chu wrote:
>     > On Thu, Jun 20, 2013 at 1:07 AM, Michael S. Tsirkin
>     <mst@redhat.com <mailto:mst@redhat.com>> wrote:
>     >> On Wed, Jun 19, 2013 at 12:39:34PM -0700, Jerry Chu wrote:
>     >>> Hi Jason,
>     >>>
>     >>> On Tue, Jun 18, 2013 at 8:29 PM, Jason Wang
>     <jasowang@redhat.com <mailto:jasowang@redhat.com>> wrote:
>     >>>> On 06/19/2013 10:31 AM, Jerry Chu wrote:
>     >>>>> In tun_net_xmit() the max qlen is computed as
>     >>>>> dev->tx_queue_len / tun->numqueues. For multi-queue
>     configuration the
>     >>>>> latter may be way too small, forcing one to adjust
>     txqueuelen based
>     >>>>> on number of queues created. (Well the default txqueuelen of
>     >>>>> 500/TUN_READQ_SIZE already seems too small even for single
>     queue.)
>     >>>> Hi Jerry:
>     >>>>
>     >>>> Do you have some test result of this? Anyway, tun allows
>     userspace to
>     >>>> adjust this value based on its requirement.
>     >>> Sure, but the default size of 500 is just way too small. queue
>     overflows even
>     >>> with a simple single-stream throughput test through
>     Openvswitch due to CPU
>     >>> scheduler anomaly. On our loaded multi-stream test even 8192
>     can't prevent
>     >>> queue overflow. But then with 8192 we'll be deep into the "buffer
>     >>> bloat" territory.
>     >>> We haven't figured out an optimal strategy for thruput vs
>     latency, but
>     >>> suffice to
>     >>> say 500 is too small.
>     >>>
>     >>> Jerry
>     >> Maybe TSO is off for you?
>     >> With TSO you can get 64Kbyte packets, 500 of these is 30 Mbytes!
>     >> We really should consider setting byte limits, not packet limits.
>     > Sorry for the delay. TSO was on when I was seeing lots of pkts
>     drops.
>     > But I realized the catch was GRO was off, which caused lots of MTU
>     > size pkts to pile up on the receive side overflowing the small
>     tuntap
>     > queue.
>     >
>     > I just finished implementing GRE support in the GRO stack. When I
>     > turned it on, there were much less pkt drops. I do notice now
>     the many
>     > acks triggered by the thruput tests will cause the tuntap queue to
>     > overflow.
>
>     Looks like you've modified tuntap codes since currently transmit
>     GRE gso
>     packet were forbidden.
>
>
> Not sure what you meant above. The change I made was all in the GRO
> stack (to support GRE pkts). No change to tuntap code. (Hopefully I
> can find
> the time to submit the patch in the near future.)

I infer from the above since you say "I just finished implementing GRE
support in the GRO stack. When I turned it on, there were much less pkt
drops." Since virtio-net does not use GRO, so I thought the GRE GRO were
enabled by host, and you see less packet drops in tuntap? If yes, looks
strange since tuntap can't transmit GRO GSO packet.
>  
>
>     >
>     > In any case, with a large tx queue there should probably have some
>     > queue mgmt or BQL logic going with it.
>
>     It's not hard to do BQL for tuntap, but since it may cause packets
>     to be
>     queued in qdisc which seems conflict with
>     5d097109257c03a71845729f8db6b5770c4bbedc (tun: only queue packets on
>     device) who just does the queuing in device.
>
>
> I don't see how changing the max qlen in the device conflicts with the
> above
> change, which is simply done by never flow control the qdisc above it.

I mean BQL may conflict with the change. Just changing the max qlen is
ok but it may hard to find a value which is good for all kinds of workload.
>
>
>     Btw, I suspect this may be another reason to cause the packets to be
>     dropped in your case.
>
>     > Jerry
>     > --
>     > To unsubscribe from this list: send the line "unsubscribe netdev" in
>     > the body of a message to majordomo@vger.kernel.org
>     <mailto:majordomo@vger.kernel.org>
>     > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>