Re: network stream fairness

[Niklas Cassel <niklas.cassel@axis.com>]

On 11/09/2015 04:50 PM, Eric Dumazet wrote:
> On Mon, 2015-11-09 at 16:41 +0100, Niklas Cassel wrote:
>> I have a ethernet driver for a 100 Mbps NIC.
>> The NIC has dedicated hardware for offloading.
>> The driver has implemented TSO, GSO and BQL.
>> Since the CPU on the SoC is rather weak, I'd rather
>> not increase the CPU load by turning off offloading.
>>
>> Since commit
>> 605ad7f184b6 ("tcp: refine TSO autosizing")
>>
>> the bandwidth is no longer fair between streams.
>> see output at the end of the mail, where I'm testing with 2 streams.
>>
>>
>> If I revert 605ad7f184b6 on 4.3, I get a stable 45 Mbps per stream.
>>
>> I can also use vanilla 4.3 and do:
>> echo 3000 > /sys/class/net/eth0/queues/tx-0/byte_queue_limits/limit_max
>> to also get a stable 45 Mbps per stream.
>>
>> My question is, am I supposed to set the BQL limit explicitly?
>> It is possible that I have missed something in my driver,
>> but my understanding is that the TCP stack sets and adjusts
>> the BQL limit automatically.
>>
>>
>> Perhaps the following info might help:
>>
>> After running iperf3 on vanilla 4.3:
>> /sys/class/net/eth0/queues/tx-0/byte_queue_limits/
>> limit 89908
>> limit_max 1879048192
>>
>> After running iperf3 on vanilla 4.3 + BQL explicitly set:
>> /sys/class/net/eth0/queues/tx-0/byte_queue_limits/
>> limit 3000
>> limit_max 3000
>>
>> After running iperf3 on 4.3 + 605ad7f184b6 reverted:
>> /sys/class/net/eth0/queues/tx-0/byte_queue_limits/
>> limit 8886
>> limit_max 1879048192
>>
> 
> There is absolutely nothing ensuring fairness among multiple TCP flows.
> 
> One TCP flow can very easily grab whole bandwidth for itself, there are
> numerous descriptions of this phenomena in various TCP studies. 
> 
> This is why we have packet schedulers ;)

Oh.. How stupid of me, I forgot to mention.. all of the measurements were
done with fq_codel.

> 
> tc qdisc replace dev eth0 root fq
> 
> This will probably help : No need to change BQL settings, so that you
> keep minimal latencies for the interactive traffic (like ping)
> 
> 
> 
>