Re: [BUG?] ixgbe: only num_online_cpus() of the tx queues are enabled

[John Fastabend <john.fastabend@gmail.com>]

On 03/08/2014 07:37 PM, Eric Dumazet wrote:
> On Sat, 2014-03-08 at 19:53 -0500, Ming Chen wrote:
>> Hi Eric,
>>
>> We noticed many changes in the TCP stack, and a lot of them come from you :-)
>>
>> Actually, we have a question about this patch you submitted
>> (http://lwn.net/Articles/564979/) regarding an experiment we conducted
>> in the 3.12.0 kernel. The results we observed in shown in the second
>> figure of panel 6 in this poster at
>> http://www.fsl.cs.sunysb.edu/~mchen/fast14poster-hashcast-portrait.pdf
>> .  We have repeated the same experiment for 100 times, and observed
>> that results like that appeared 4 times. For this experiment, we
>> observed that all five flows are using dedicated tx queues.  But what
>> makes a big difference is the average packet sizes of the flows.
>> Client4 has an average packet size of around 3KB while all other
>> clients generate packet sizes over 50KB. We suspect it might be caused
>> by this TSO Packets Automatic Sizing feaure. Our reasoning is this: if
>> a TCP flow starts slowly, this feature will assign it a small packet
>> size. The packet size and the sending rate can somehow form a feedback
>> loop, which can force the TCP flow's rate to stay low. What do you
>> think about this?
>
> I think nothing at all. TCP is not fair. TCP tries to steal whole
> bandwidth by definition. One flow can have much more than the neighbour.
>
> With FQ, you can force some fairness, but if you use multiqueue, there
> is no guarantee at all, unless you make sure :
>
> - no more than one flow per queue.
> - Nic is able to provide fairness among all active TX queues.
>

The NIC by default will round robin amongst the queues and should be
reasonably fair. We could increase the number of TX queues the driver
enables and for a small number of flows the first condition is easier
to meet. Although it wont help as the flow count increases.

Using FQ as a root qdisc though I think will really hurt performance
on small packet sizes. For larger packet sizes its probably less
noticeable. Each queue can use FQ as noted previously.


> Thats the ideal condition, and that is quite hard to meet.
>
> The feedback loop you mention should be solved by the patch I sent
> today : TCP Small queue make sure that you have no more than 2 packets
> per flow on qdisc / TX queues. So on 'fast' flow cannot have 90% of the
> packets in the qdisc. cwnd is maintained to very small values, assuming
> receiver is behaving normally.
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


-- 
John Fastabend         Intel Corporation