linux-3.14-rc1 & PACKET_QDISC_BYPASS : slow_path warning

linux-3.14-rc1 & PACKET_QDISC_BYPASS : slow_path warning

[Mathias Kretschmer]

[-- Attachment #1: Type: text/plain, Size: 958 bytes --]

Hi Daniel,

we are developing a wired/wireless MPLS switch. Currently the data plane 
runs in user space using PF_PACKET sockets via RX_RING/TX_RING.

We had hoped to test the PACKET_QDISC_BYPASS option since this seems to 
be the proper optimization for our purposes.

Unfortunately, we're seeing a 'slow path' warning for every packet that 
is being sent out. With PACKET_QDISC_BYPASS disabled, no warnings are 
dumped. Hardware is an older AMD Geode LX embedded board (ALiX).

BTW, this happens while sending via a wireless (802.11) adhoc interface. 
Hence, it might be an interaction with the ieee80211 sub system.

Please, let me know if you need any further information.

Cheers,

Mathias

-- 
Dr. Mathias Kretschmer, Head of Competence Center
Fraunhofer FOKUS Network Research
A Schloss Birlinghoven, 53754 Sankt Augustin, Germany
T +49-2241-14-3466, F +49-2241-14-1050
E mathias.kretschmer@fokus.fraunhofer.de
W http://www.fokus.fraunhofer.de/en/net

[-- Attachment #2: slow-path-3.14.txt --]
[-- Type: text/plain, Size: 2291 bytes --]

[ 3246.953225] ------------[ cut here ]------------
[ 3246.953225] WARNING: CPU: 0 PID: 778 at drivers/net/wireless/ath/ath9k/xmit.c:2195 ath_tx_start+0x245/0x24d()
[ 3246.953225] Modules linked in: lm90 gpio_keys_polled scx200_acb
[ 3246.953225] CPU: 0 PID: 778 Comm: wiback Tainted: G        W    3.14.0-rc1.geode #1
[ 3246.953225]  c13bec9a c10207ed c14f4de0 00000000 0000030a c151cab0 00000893 c121071e
[ 3246.953225]  c121071e 00000893 cf4ba180 cf791b14 c01d8c20 c102081c 00000009 00000000
[ 3246.953225]  c121071e cec3c358 cf7c8c2c 00000200 c01d9eb0 00000000 c01d94dc 00000000
[ 3246.953225] Call Trace:
[ 3246.953225]  [<c13bec9a>] ? dump_stack+0xa/0x13
[ 3246.953225]  [<c10207ed>] ? warn_slowpath_common+0x7a/0x8e
[ 3246.953225]  [<c121071e>] ? ath_tx_start+0x245/0x24d
[ 3246.953225]  [<c121071e>] ? ath_tx_start+0x245/0x24d
[ 3246.953225]  [<c102081c>] ? warn_slowpath_null+0x1b/0x1f
[ 3246.953225]  [<c121071e>] ? ath_tx_start+0x245/0x24d
[ 3246.953225]  [<c120a43f>] ? ath9k_tx+0x7f/0x162
[ 3246.953225]  [<c13a14cd>] ? __ieee80211_tx+0xcf/0x288
[ 3246.953225]  [<c13a314b>] ? ieee80211_tx+0xa0/0xc0
[ 3246.953225]  [<c13a395c>] ? ieee80211_xmit+0x81/0x99
[ 3246.953225]  [<c13a4454>] ? ieee80211_subif_start_xmit+0x7d0/0x934
[ 3246.953225]  [<c134e8da>] ? packet_direct_xmit+0x6a/0xee
[ 3246.953225]  [<c13506bb>] ? packet_sendmsg+0x4eb/0xd58
[ 3246.953225]  [<c12ae6fb>] ? sock_sendmsg+0x4f/0x6f
[ 3246.953225]  [<c12afe68>] ? SYSC_sendto+0xc8/0xe9
[ 3246.953225]  [<c103a85f>] ? sched_clock_local.constprop.7+0x39/0x152
[ 3246.953225]  [<c12b0c63>] ? SYSC_socketcall+0x3c8/0x8f7
[ 3246.953225]  [<c13c0d68>] ? __schedule+0x164/0x3c0
[ 3246.953225]  [<c103a85f>] ? sched_clock_local.constprop.7+0x39/0x152
[ 3246.953225]  [<c1045fef>] ? ktime_get+0x4c/0xd4
[ 3246.953225]  [<c10ae7b5>] ? ep_send_events_proc+0x69/0x105
[ 3246.953225]  [<c10ae74c>] ? ep_read_events_proc+0x78/0x78
[ 3246.953225]  [<c10aee5f>] ? ep_scan_ready_list.isra.27+0x122/0x13d
[ 3246.953225]  [<c10ae74c>] ? ep_read_events_proc+0x78/0x78
[ 3246.953225]  [<c10b0765>] ? timerfd_read+0xa6/0x25e
[ 3246.953225]  [<c10aef9d>] ? ep_poll+0x109/0x234
[ 3246.953225]  [<c12b123f>] ? SyS_socketcall+0xd/0xe
[ 3246.953225]  [<c13c28ed>] ? syscall_call+0x7/0xb
[ 3246.953225] ---[ end trace c930b576e9270049 ]---

Re: linux-3.14-rc1 & PACKET_QDISC_BYPASS : slow_path warning

[Daniel Borkmann]

Hi Mathias, [cc'ing Felix]

On 02/03/2014 11:47 PM, Mathias Kretschmer wrote:
> Hi Daniel,
>
> we are developing a wired/wireless MPLS switch. Currently the data plane runs in user space using PF_PACKET sockets via RX_RING/TX_RING.
>
> We had hoped to test the PACKET_QDISC_BYPASS option since this seems to be the proper optimization for our purposes.
>
> Unfortunately, we're seeing a 'slow path' warning for every packet that is being sent out. With PACKET_QDISC_BYPASS disabled, no warnings are dumped. Hardware is an older AMD Geode LX embedded board (ALiX).
>
> BTW, this happens while sending via a wireless (802.11) adhoc interface. Hence, it might be an interaction with the ieee80211 sub system.

Hm, so the WARN_ON() is triggered inside ath9k driver in relation to 802.11 QoS,
and came in from commit 066dae93bdf ("ath9k: rework tx queue selection and fix
queue stopping/waking"). We did the stress testing of that option for PF_PACKET
on 10Gbit/s NICs. Seems to me you might be running into the same issue when using
pktgen as it randomly or per round-robin selects tx queues as well? Not entirely
sure how necessary this WARN_ON() is though, Felix? I think QDISC_BYPASS might not
be the best option in your case, perhaps you will run into increased power usage
in your NIC as a side-effect?

Cheers,

Daniel
--
To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux-3.14-rc1 & PACKET_QDISC_BYPASS : slow_path warning

[Mathias Kretschmer]

Hi Daniel,

On 02/04/2014 01:56 PM, Daniel Borkmann wrote:
> Hi Mathias, [cc'ing Felix]
>
> On 02/03/2014 11:47 PM, Mathias Kretschmer wrote:
>> Hi Daniel,
>>
>> we are developing a wired/wireless MPLS switch. Currently the data plane runs in
>> user space using PF_PACKET sockets via RX_RING/TX_RING.
>>
>> We had hoped to test the PACKET_QDISC_BYPASS option since this seems to be the
>> proper optimization for our purposes.
>>
>> Unfortunately, we're seeing a 'slow path' warning for every packet that is being
>> sent out. With PACKET_QDISC_BYPASS disabled, no warnings are dumped. Hardware is
>> an older AMD Geode LX embedded board (ALiX).
>>
>> BTW, this happens while sending via a wireless (802.11) adhoc interface. Hence, it
>> might be an interaction with the ieee80211 sub system.
>
> Hm, so the WARN_ON() is triggered inside ath9k driver in relation to 802.11 QoS,
> and came in from commit 066dae93bdf ("ath9k: rework tx queue selection and fix
> queue stopping/waking"). We did the stress testing of that option for PF_PACKET
> on 10Gbit/s NICs. Seems to me you might be running into the same issue when using
> pktgen as it randomly or per round-robin selects tx queues as well? Not entirely
> sure how necessary this WARN_ON() is though, Felix? I think QDISC_BYPASS might not
> be the best option in your case, perhaps you will run into increased power usage
> in your NIC as a side-effect?

I'm not familiar with the exact implementation details, but from the description of 
this option, it seems to me that this is exactly what one would want to use if the 
goal is to send an Ethernet frame out on a particular interface without any further 
processing by the kernel.

Why would this increase the power usage on the NIC ?  Due to a higher achievable 
packet rate ?  That would be acceptable :)

Cheers,

Mathias


> Cheers,
>
> Daniel

--
To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux-3.14-rc1 & PACKET_QDISC_BYPASS : slow_path warning

[Daniel Borkmann]

On 02/04/2014 02:13 PM, Mathias Kretschmer wrote:
> On 02/04/2014 01:56 PM, Daniel Borkmann wrote:
>> On 02/03/2014 11:47 PM, Mathias Kretschmer wrote:
...
>>> we are developing a wired/wireless MPLS switch. Currently the data plane runs in
>>> user space using PF_PACKET sockets via RX_RING/TX_RING.
>>>
>>> We had hoped to test the PACKET_QDISC_BYPASS option since this seems to be the
>>> proper optimization for our purposes.
>>>
>>> Unfortunately, we're seeing a 'slow path' warning for every packet that is being
>>> sent out. With PACKET_QDISC_BYPASS disabled, no warnings are dumped. Hardware is
>>> an older AMD Geode LX embedded board (ALiX).
>>>
>>> BTW, this happens while sending via a wireless (802.11) adhoc interface. Hence, it
>>> might be an interaction with the ieee80211 sub system.
>>
>> Hm, so the WARN_ON() is triggered inside ath9k driver in relation to 802.11 QoS,
>> and came in from commit 066dae93bdf ("ath9k: rework tx queue selection and fix
>> queue stopping/waking"). We did the stress testing of that option for PF_PACKET
>> on 10Gbit/s NICs. Seems to me you might be running into the same issue when using
>> pktgen as it randomly or per round-robin selects tx queues as well? Not entirely
>> sure how necessary this WARN_ON() is though, Felix? I think QDISC_BYPASS might not
>> be the best option in your case, perhaps you will run into increased power usage
>> in your NIC as a side-effect?
>
> I'm not familiar with the exact implementation details, but from the description of this option, it seems to me that this is exactly what one would want to use if the goal is to send an Ethernet frame out on a particular interface without any further processing by the kernel.
>
> Why would this increase the power usage on the NIC ?  Due to a higher achievable packet rate ?  That would be acceptable :)

I'm not too familiar with the ieee80211 sub system, so I let Felix answer side
effects and if actually the WARN_ON() is needed. ;) PACKET_QDISC_BYPASS is, as
documented, designed for advanced pktgen resp. traffic generator like scenarios
where you just sort of "brute force" packets to your NIC to stress test a remote
machine for further analysis. I don't think it's very useful in your scenario
when you have a wired/wireless MPLS switch, you rather might want to buffer/queue
and therefore use qdisc layer instead.
--
To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux-3.14-rc1 & PACKET_QDISC_BYPASS : slow_path warning

[Mathias Kretschmer]

On 02/04/2014 03:25 PM, Daniel Borkmann wrote:
> On 02/04/2014 02:13 PM, Mathias Kretschmer wrote:
>> On 02/04/2014 01:56 PM, Daniel Borkmann wrote:
>>> On 02/03/2014 11:47 PM, Mathias Kretschmer wrote:
> ...
>>>> we are developing a wired/wireless MPLS switch. Currently the data plane runs in
>>>> user space using PF_PACKET sockets via RX_RING/TX_RING.
>>>>
>>>> We had hoped to test the PACKET_QDISC_BYPASS option since this seems to be the
>>>> proper optimization for our purposes.
>>>>
>>>> Unfortunately, we're seeing a 'slow path' warning for every packet that is being
>>>> sent out. With PACKET_QDISC_BYPASS disabled, no warnings are dumped. Hardware is
>>>> an older AMD Geode LX embedded board (ALiX).
>>>>
>>>> BTW, this happens while sending via a wireless (802.11) adhoc interface. Hence, it
>>>> might be an interaction with the ieee80211 sub system.
>>>
>>> Hm, so the WARN_ON() is triggered inside ath9k driver in relation to 802.11 QoS,
>>> and came in from commit 066dae93bdf ("ath9k: rework tx queue selection and fix
>>> queue stopping/waking"). We did the stress testing of that option for PF_PACKET
>>> on 10Gbit/s NICs. Seems to me you might be running into the same issue when using
>>> pktgen as it randomly or per round-robin selects tx queues as well? Not entirely
>>> sure how necessary this WARN_ON() is though, Felix? I think QDISC_BYPASS might not
>>> be the best option in your case, perhaps you will run into increased power usage
>>> in your NIC as a side-effect?
>>
>> I'm not familiar with the exact implementation details, but from the description
>> of this option, it seems to me that this is exactly what one would want to use if
>> the goal is to send an Ethernet frame out on a particular interface without any
>> further processing by the kernel.
>>
>> Why would this increase the power usage on the NIC ?  Due to a higher achievable
>> packet rate ?  That would be acceptable :)
>
> I'm not too familiar with the ieee80211 sub system, so I let Felix answer side
> effects and if actually the WARN_ON() is needed. ;) PACKET_QDISC_BYPASS is, as
> documented, designed for advanced pktgen resp. traffic generator like scenarios
> where you just sort of "brute force" packets to your NIC to stress test a remote
> machine for further analysis. I don't think it's very useful in your scenario
> when you have a wired/wireless MPLS switch, you rather might want to buffer/queue
> and therefore use qdisc layer instead.

Hm, I was hoping/assuming that we still get to use hardware queues, if provided by 
the driver. The main goal was to avoid any further PF_PACKET framework overhead.

If the WARN_ON() issue gets solved, we will revisit this option and evaluate its 
applicability.

Thanks,

Mathias

Re: linux-3.14-rc1 & PACKET_QDISC_BYPASS : slow_path warning

[Felix Fietkau]

On 2014-02-04 15:35, Mathias Kretschmer wrote:
> On 02/04/2014 03:25 PM, Daniel Borkmann wrote:
>> On 02/04/2014 02:13 PM, Mathias Kretschmer wrote:
>>> On 02/04/2014 01:56 PM, Daniel Borkmann wrote:
>>>> On 02/03/2014 11:47 PM, Mathias Kretschmer wrote:
>> ...
>>>>> we are developing a wired/wireless MPLS switch. Currently the data plane runs in
>>>>> user space using PF_PACKET sockets via RX_RING/TX_RING.
>>>>>
>>>>> We had hoped to test the PACKET_QDISC_BYPASS option since this seems to be the
>>>>> proper optimization for our purposes.
>>>>>
>>>>> Unfortunately, we're seeing a 'slow path' warning for every packet that is being
>>>>> sent out. With PACKET_QDISC_BYPASS disabled, no warnings are dumped. Hardware is
>>>>> an older AMD Geode LX embedded board (ALiX).
>>>>>
>>>>> BTW, this happens while sending via a wireless (802.11) adhoc interface. Hence, it
>>>>> might be an interaction with the ieee80211 sub system.
>>>>
>>>> Hm, so the WARN_ON() is triggered inside ath9k driver in relation to 802.11 QoS,
>>>> and came in from commit 066dae93bdf ("ath9k: rework tx queue selection and fix
>>>> queue stopping/waking"). We did the stress testing of that option for PF_PACKET
>>>> on 10Gbit/s NICs. Seems to me you might be running into the same issue when using
>>>> pktgen as it randomly or per round-robin selects tx queues as well? Not entirely
>>>> sure how necessary this WARN_ON() is though, Felix? I think QDISC_BYPASS might not
>>>> be the best option in your case, perhaps you will run into increased power usage
>>>> in your NIC as a side-effect?
>>>
>>> I'm not familiar with the exact implementation details, but from the description
>>> of this option, it seems to me that this is exactly what one would want to use if
>>> the goal is to send an Ethernet frame out on a particular interface without any
>>> further processing by the kernel.
>>>
>>> Why would this increase the power usage on the NIC ?  Due to a higher achievable
>>> packet rate ?  That would be acceptable :)
>>
>> I'm not too familiar with the ieee80211 sub system, so I let Felix answer side
>> effects and if actually the WARN_ON() is needed. ;) PACKET_QDISC_BYPASS is, as
>> documented, designed for advanced pktgen resp. traffic generator like scenarios
>> where you just sort of "brute force" packets to your NIC to stress test a remote
>> machine for further analysis. I don't think it's very useful in your scenario
>> when you have a wired/wireless MPLS switch, you rather might want to buffer/queue
>> and therefore use qdisc layer instead.
> 
> Hm, I was hoping/assuming that we still get to use hardware queues, if provided by 
> the driver. The main goal was to avoid any further PF_PACKET framework overhead.
> 
> If the WARN_ON() issue gets solved, we will revisit this option and evaluate its 
> applicability.
The reason for the WARN_ON is probably either the .ndo_select_queue call
is not run, or its queue selection result is changed before the frame
hits the driver's tx call.
This call sets both the queue and the TID (similar to 802.1d tag), which
makes it into the packet via 802.11e (WMM, QoS).
It is important to the driver that the TID is in sync with the queue
selection, if that is not the case, then pending frame counters can get
messed up.
If you really want to bypass qdisc, make sure that at least
ndo_select_queue is called before passing the frame to the netdev.

- Felix

Re: linux-3.14-rc1 & PACKET_QDISC_BYPASS : slow_path warning

[Daniel Borkmann]

On 02/04/2014 03:48 PM, Felix Fietkau wrote:
> On 2014-02-04 15:35, Mathias Kretschmer wrote:
>> On 02/04/2014 03:25 PM, Daniel Borkmann wrote:
>>> On 02/04/2014 02:13 PM, Mathias Kretschmer wrote:
>>>> On 02/04/2014 01:56 PM, Daniel Borkmann wrote:
>>>>> On 02/03/2014 11:47 PM, Mathias Kretschmer wrote:
>>> ...
>>>>>> we are developing a wired/wireless MPLS switch. Currently the data plane runs in
>>>>>> user space using PF_PACKET sockets via RX_RING/TX_RING.
>>>>>>
>>>>>> We had hoped to test the PACKET_QDISC_BYPASS option since this seems to be the
>>>>>> proper optimization for our purposes.
>>>>>>
>>>>>> Unfortunately, we're seeing a 'slow path' warning for every packet that is being
>>>>>> sent out. With PACKET_QDISC_BYPASS disabled, no warnings are dumped. Hardware is
>>>>>> an older AMD Geode LX embedded board (ALiX).
>>>>>>
>>>>>> BTW, this happens while sending via a wireless (802.11) adhoc interface. Hence, it
>>>>>> might be an interaction with the ieee80211 sub system.
>>>>>
>>>>> Hm, so the WARN_ON() is triggered inside ath9k driver in relation to 802.11 QoS,
>>>>> and came in from commit 066dae93bdf ("ath9k: rework tx queue selection and fix
>>>>> queue stopping/waking"). We did the stress testing of that option for PF_PACKET
>>>>> on 10Gbit/s NICs. Seems to me you might be running into the same issue when using
>>>>> pktgen as it randomly or per round-robin selects tx queues as well? Not entirely
>>>>> sure how necessary this WARN_ON() is though, Felix? I think QDISC_BYPASS might not
>>>>> be the best option in your case, perhaps you will run into increased power usage
>>>>> in your NIC as a side-effect?
>>>>
>>>> I'm not familiar with the exact implementation details, but from the description
>>>> of this option, it seems to me that this is exactly what one would want to use if
>>>> the goal is to send an Ethernet frame out on a particular interface without any
>>>> further processing by the kernel.
>>>>
>>>> Why would this increase the power usage on the NIC ?  Due to a higher achievable
>>>> packet rate ?  That would be acceptable :)
>>>
>>> I'm not too familiar with the ieee80211 sub system, so I let Felix answer side
>>> effects and if actually the WARN_ON() is needed. ;) PACKET_QDISC_BYPASS is, as
>>> documented, designed for advanced pktgen resp. traffic generator like scenarios
>>> where you just sort of "brute force" packets to your NIC to stress test a remote
>>> machine for further analysis. I don't think it's very useful in your scenario
>>> when you have a wired/wireless MPLS switch, you rather might want to buffer/queue
>>> and therefore use qdisc layer instead.
>>
>> Hm, I was hoping/assuming that we still get to use hardware queues, if provided by
>> the driver. The main goal was to avoid any further PF_PACKET framework overhead.
>>
>> If the WARN_ON() issue gets solved, we will revisit this option and evaluate its
>> applicability.
> The reason for the WARN_ON is probably either the .ndo_select_queue call
> is not run, or its queue selection result is changed before the frame
> hits the driver's tx call.
> This call sets both the queue and the TID (similar to 802.1d tag), which
> makes it into the packet via 802.11e (WMM, QoS).
> It is important to the driver that the TID is in sync with the queue
> selection, if that is not the case, then pending frame counters can get
> messed up.
> If you really want to bypass qdisc, make sure that at least
> ndo_select_queue is called before passing the frame to the netdev.

Ok, thanks for the input, we'll look further into it and eventually come up
with something.

> - Felix
>