Re: linux-3.14-rc1 & PACKET_QDISC_BYPASS : slow_path warning

[Daniel Borkmann <dborkman@redhat.com>]

On 02/04/2014 03:48 PM, Felix Fietkau wrote:
> On 2014-02-04 15:35, Mathias Kretschmer wrote:
>> On 02/04/2014 03:25 PM, Daniel Borkmann wrote:
>>> On 02/04/2014 02:13 PM, Mathias Kretschmer wrote:
>>>> On 02/04/2014 01:56 PM, Daniel Borkmann wrote:
>>>>> On 02/03/2014 11:47 PM, Mathias Kretschmer wrote:
>>> ...
>>>>>> we are developing a wired/wireless MPLS switch. Currently the data plane runs in
>>>>>> user space using PF_PACKET sockets via RX_RING/TX_RING.
>>>>>>
>>>>>> We had hoped to test the PACKET_QDISC_BYPASS option since this seems to be the
>>>>>> proper optimization for our purposes.
>>>>>>
>>>>>> Unfortunately, we're seeing a 'slow path' warning for every packet that is being
>>>>>> sent out. With PACKET_QDISC_BYPASS disabled, no warnings are dumped. Hardware is
>>>>>> an older AMD Geode LX embedded board (ALiX).
>>>>>>
>>>>>> BTW, this happens while sending via a wireless (802.11) adhoc interface. Hence, it
>>>>>> might be an interaction with the ieee80211 sub system.
>>>>>
>>>>> Hm, so the WARN_ON() is triggered inside ath9k driver in relation to 802.11 QoS,
>>>>> and came in from commit 066dae93bdf ("ath9k: rework tx queue selection and fix
>>>>> queue stopping/waking"). We did the stress testing of that option for PF_PACKET
>>>>> on 10Gbit/s NICs. Seems to me you might be running into the same issue when using
>>>>> pktgen as it randomly or per round-robin selects tx queues as well? Not entirely
>>>>> sure how necessary this WARN_ON() is though, Felix? I think QDISC_BYPASS might not
>>>>> be the best option in your case, perhaps you will run into increased power usage
>>>>> in your NIC as a side-effect?
>>>>
>>>> I'm not familiar with the exact implementation details, but from the description
>>>> of this option, it seems to me that this is exactly what one would want to use if
>>>> the goal is to send an Ethernet frame out on a particular interface without any
>>>> further processing by the kernel.
>>>>
>>>> Why would this increase the power usage on the NIC ?  Due to a higher achievable
>>>> packet rate ?  That would be acceptable :)
>>>
>>> I'm not too familiar with the ieee80211 sub system, so I let Felix answer side
>>> effects and if actually the WARN_ON() is needed. ;) PACKET_QDISC_BYPASS is, as
>>> documented, designed for advanced pktgen resp. traffic generator like scenarios
>>> where you just sort of "brute force" packets to your NIC to stress test a remote
>>> machine for further analysis. I don't think it's very useful in your scenario
>>> when you have a wired/wireless MPLS switch, you rather might want to buffer/queue
>>> and therefore use qdisc layer instead.
>>
>> Hm, I was hoping/assuming that we still get to use hardware queues, if provided by
>> the driver. The main goal was to avoid any further PF_PACKET framework overhead.
>>
>> If the WARN_ON() issue gets solved, we will revisit this option and evaluate its
>> applicability.
> The reason for the WARN_ON is probably either the .ndo_select_queue call
> is not run, or its queue selection result is changed before the frame
> hits the driver's tx call.
> This call sets both the queue and the TID (similar to 802.1d tag), which
> makes it into the packet via 802.11e (WMM, QoS).
> It is important to the driver that the TID is in sync with the queue
> selection, if that is not the case, then pending frame counters can get
> messed up.
> If you really want to bypass qdisc, make sure that at least
> ndo_select_queue is called before passing the frame to the netdev.

Ok, thanks for the input, we'll look further into it and eventually come up
with something.

> - Felix
>