problems with b43 and greedy traffic

problems with b43 and greedy traffic

[francesco.gringoli at ing.unibs.it]

Hello Michael,

I'm doing experiments sending greedy udp traffic from a b43 station to a b43 access point. I have noticed that switching from 2.6.34-rc7 to 2.6.35 the sendmsg call becomes "almost" non blocking when sending from a Broadcom nic while it is still as usual with other nics. 

If I load the channel with a 54Mb/s iperf stream (iperf -b54M ...) on < 2.6.35 I see that the application is blocked times to times when calling sendmsg() so that it is slowed down to the channel capabilities and packets are not internally dropped. Clearly they can still be lost on the air :-)

With >= 2.6.35 the application is never blocked and all the packets exceeding the channel capabilities are internally lost by the kernel: in particular it is the asynchronous tx worker (b43_tx_work) that drops them, since it calls b43_dma_tx even if the interface has been stopped because the dma FIFO queue was full. Apart from packets being lost, the CPU load increases since packets cross all the kernel code, from udp_sendmsg down to b43_dma_tx even if they will be dropped.

I don't think this is the expected behavior on Linux: I did some testing to check what happens with other devices and I can experience only the first behavior on Intel and Atheros WiFi nics as well as on Fast Ethernet nics (in this case I run iperf -b100M :-) independently of the kernel version.

Strangely the b43 sources in 2.6.35 are really similar to those in 2.6.34-rc7 and the differences do not seem to justify the different behavior. There are also other weird observations (like qdisc never used in < 2.6.34-rc7) but I would like to have a first opinion from your side.

Many thanks,
-Francesco

P.S. what reported does not depend on the firmware version. I also tried a few cards (4306, 4311 and 4318) and nothing changed.

problems with b43 and greedy traffic

[Michael Büsch]

On Thu, 2011-04-21 at 20:31 +0200, francesco.gringoli at ing.unibs.it
wrote: 
> I'm doing experiments sending greedy udp traffic from a 
> b43 station to a b43 access point. I have noticed that
> switching from 2.6.34-rc7 to 2.6.35 

Please talk to those people who actually modified the driver.
I didn't really touch it for years.

-- 
Greetings Michael.

problems with b43 and greedy traffic

[Larry Finger]

On 04/21/2011 01:31 PM, francesco.gringoli at ing.unibs.it wrote:
> Hello Michael,
>
> I'm doing experiments sending greedy udp traffic from a b43 station to a b43 access point. I have noticed that switching from 2.6.34-rc7 to 2.6.35 the sendmsg call becomes "almost" non blocking when sending from a Broadcom nic while it is still as usual with other nics.
>
> If I load the channel with a 54Mb/s iperf stream (iperf -b54M ...) on<  2.6.35 I see that the application is blocked times to times when calling sendmsg() so that it is slowed down to the channel capabilities and packets are not internally dropped. Clearly they can still be lost on the air :-)
>
> With>= 2.6.35 the application is never blocked and all the packets exceeding the channel capabilities are internally lost by the kernel: in particular it is the asynchronous tx worker (b43_tx_work) that drops them, since it calls b43_dma_tx even if the interface has been stopped because the dma FIFO queue was full. Apart from packets being lost, the CPU load increases since packets cross all the kernel code, from udp_sendmsg down to b43_dma_tx even if they will be dropped.
>
> I don't think this is the expected behavior on Linux: I did some testing to check what happens with other devices and I can experience only the first behavior on Intel and Atheros WiFi nics as well as on Fast Ethernet nics (in this case I run iperf -b100M :-) independently of the kernel version.
>
> Strangely the b43 sources in 2.6.35 are really similar to those in 2.6.34-rc7 and the differences do not seem to justify the different behavior. There are also other weird observations (like qdisc never used in<  2.6.34-rc7) but I would like to have a first opinion from your side.
>
> Many thanks,
> -Francesco
>
> P.S. what reported does not depend on the firmware version. I also tried a few cards (4306, 4311 and 4318) and nothing changed.

Francesco,

I agree that there are no changes in b43 between 2.6.34-rc7 and 2.6.35 that 
would cause this problem. All but one of the changes are for N PHYs, and that 
one only removes some braces that are not needed. In addition, there are no 
changes in ssb that would affect anything other than SPROM loading.

Have you tried your test with a 2.6.38 kernel? Perhaps the problem has already 
been fixed. The other thing to do would be to try to bisect between .35 and 
.34-rc7. If you do that, consider the entire kernel, not just b43. If it is 
impossible for you to do either of the tests, please send me any command files 
that you are using, and I'll try it here.

Larry

problems with b43 and greedy traffic

[francesco.gringoli at ing.unibs.it]

On Apr 21, 2011, at 11:08 PM, Larry Finger wrote:

> On 04/21/2011 01:31 PM, francesco.gringoli at ing.unibs.it wrote:
>> Hello Michael,
>> 
>> I'm doing experiments sending greedy udp traffic from a b43 station to a b43 access point. I have noticed that switching from 2.6.34-rc7 to 2.6.35 the sendmsg call becomes "almost" non blocking when sending from a Broadcom nic while it is still as usual with other nics.
>> 
>> If I load the channel with a 54Mb/s iperf stream (iperf -b54M ...) on<  2.6.35 I see that the application is blocked times to times when calling sendmsg() so that it is slowed down to the channel capabilities and packets are not internally dropped. Clearly they can still be lost on the air :-)
>> 
>> With>= 2.6.35 the application is never blocked and all the packets exceeding the channel capabilities are internally lost by the kernel: in particular it is the asynchronous tx worker (b43_tx_work) that drops them, since it calls b43_dma_tx even if the interface has been stopped because the dma FIFO queue was full. Apart from packets being lost, the CPU load increases since packets cross all the kernel code, from udp_sendmsg down to b43_dma_tx even if they will be dropped.
>> 
>> I don't think this is the expected behavior on Linux: I did some testing to check what happens with other devices and I can experience only the first behavior on Intel and Atheros WiFi nics as well as on Fast Ethernet nics (in this case I run iperf -b100M :-) independently of the kernel version.
>> 
>> Strangely the b43 sources in 2.6.35 are really similar to those in 2.6.34-rc7 and the differences do not seem to justify the different behavior. There are also other weird observations (like qdisc never used in<  2.6.34-rc7) but I would like to have a first opinion from your side.
>> 
>> Many thanks,
>> -Francesco
>> 
>> P.S. what reported does not depend on the firmware version. I also tried a few cards (4306, 4311 and 4318) and nothing changed.
> 
> Francesco,
> 
> I agree that there are no changes in b43 between 2.6.34-rc7 and 2.6.35 that would cause this problem. All but one of the changes are for N PHYs, and that one only removes some braces that are not needed. In addition, there are no changes in ssb that would affect anything other than SPROM loading.
Yes, you're right, I agree something else caused this problem emerge but maybe it should be fixed in b43. All other drivers (at least those I tested) behave as expected.

> Have you tried your test with a 2.6.38 kernel? Perhaps the problem has already been fixed.
Yes, I already tried: all kernels >= 2.6.35 show the same "problem".

> The other thing to do would be to try to bisect between .35 and .34-rc7. If you do that, consider the entire kernel, not just b43. If it is impossible for you to do either of the tests, please send me any command files that you are using, and I'll try it here.
Doing right now, I will let you know asap (if I come to a conclusion).

Thanks,
-Francesco

problems with b43 and greedy traffic

[francesco.gringoli at ing.unibs.it]

On Apr 21, 2011, at 11:08 PM, Larry Finger wrote:

> On 04/21/2011 01:31 PM, francesco.gringoli at ing.unibs.it wrote:
>> Hello Michael,
>> 
>> I'm doing experiments sending greedy udp traffic from a b43 station to a b43 access point. I have noticed that switching from 2.6.34-rc7 to 2.6.35 the sendmsg call becomes "almost" non blocking when sending from a Broadcom nic while it is still as usual with other nics.
>> 
>> If I load the channel with a 54Mb/s iperf stream (iperf -b54M ...) on<  2.6.35 I see that the application is blocked times to times when calling sendmsg() so that it is slowed down to the channel capabilities and packets are not internally dropped. Clearly they can still be lost on the air :-)
>> 
>> With>= 2.6.35 the application is never blocked and all the packets exceeding the channel capabilities are internally lost by the kernel: in particular it is the asynchronous tx worker (b43_tx_work) that drops them, since it calls b43_dma_tx even if the interface has been stopped because the dma FIFO queue was full. Apart from packets being lost, the CPU load increases since packets cross all the kernel code, from udp_sendmsg down to b43_dma_tx even if they will be dropped.
>> 
>> I don't think this is the expected behavior on Linux: I did some testing to check what happens with other devices and I can experience only the first behavior on Intel and Atheros WiFi nics as well as on Fast Ethernet nics (in this case I run iperf -b100M :-) independently of the kernel version.
>> 
>> Strangely the b43 sources in 2.6.35 are really similar to those in 2.6.34-rc7 and the differences do not seem to justify the different behavior. There are also other weird observations (like qdisc never used in<  2.6.34-rc7) but I would like to have a first opinion from your side.
>> 
>> Many thanks,
>> -Francesco
>> 
>> P.S. what reported does not depend on the firmware version. I also tried a few cards (4306, 4311 and 4318) and nothing changed.
> 
> Francesco,
> 
> I agree that there are no changes in b43 between 2.6.34-rc7 and 2.6.35 that would cause this problem. All but one of the changes are for N PHYs, and that one only removes some braces that are not needed. In addition, there are no changes in ssb that would affect anything other than SPROM loading.
> 
> Have you tried your test with a 2.6.38 kernel? Perhaps the problem has already been fixed. The other thing to do would be to try to bisect between .35 and .34-rc7. If you do that, consider the entire kernel, not just b43. If it is impossible for you to do either of the tests, please send me any command files that you are using, and I'll try it here.
Hi Larry,

I just finished bisecting, git tells

	fc6055a5ba31e2c14e36e8939f9bf2b6d586a7f5 is first bad commit
	commit fc6055a5ba31e2c14e36e8939f9bf2b6d586a7f5
	Author: Eric Dumazet <eric.dumazet@gmail.com>
	Date:   Fri Apr 16 12:18:22 2010 +0000

Some info to reproduce what I'm seeing:

- receiver, can be whatever platform, run iperf like this
	$: iperf -s -u -p 20002 -i 1

- sender, whatever firmware, one of 4306-4318-4311 devices, run iperf like this
	$: iperf -c RECEIVER_IP -u -p 20002 -b 1000M -i 1 -n 1

This way you will force iperf on the sender to saturate the channel. Before this patch, the UDP socket used by iperf is blocking, -b 1000 doesn't do anything and everything goes fine, no datagram is lost at the sender. After this patch, -b 1000M will force the sending kernel to continuously accept a new datagrams, descend them to qdisc and device queue, and finally discard when the interface is stopped (that is most of the time since the device can't transmit more packets than the channel can accept).

I don't know what to do in this case: as I said before, this problem shows only with b43, doesn't with Atheros (I only tested ar9170 usb devices) and intel ipw2200. Suggestions?

Many thanks,
-Francesco

problems with b43 and greedy traffic

[Francesco Gringoli]

On Apr 22, 2011, at 9:37 PM, francesco.gringoli at ing.unibs.it wrote:

> On Apr 21, 2011, at 11:08 PM, Larry Finger wrote:
> 
>> On 04/21/2011 01:31 PM, francesco.gringoli at ing.unibs.it wrote:
>>> Hello Michael,
>>> 
>>> I'm doing experiments sending greedy udp traffic from a b43 station to a b43 access point. I have noticed that switching from 2.6.34-rc7 to 2.6.35 the sendmsg call becomes "almost" non blocking when sending from a Broadcom nic while it is still as usual with other nics.
>>> 
>>> If I load the channel with a 54Mb/s iperf stream (iperf -b54M ...) on<  2.6.35 I see that the application is blocked times to times when calling sendmsg() so that it is slowed down to the channel capabilities and packets are not internally dropped. Clearly they can still be lost on the air :-)
>>> 
>>> With>= 2.6.35 the application is never blocked and all the packets exceeding the channel capabilities are internally lost by the kernel: in particular it is the asynchronous tx worker (b43_tx_work) that drops them, since it calls b43_dma_tx even if the interface has been stopped because the dma FIFO queue was full. Apart from packets being lost, the CPU load increases since packets cross all the kernel code, from udp_sendmsg down to b43_dma_tx even if they will be dropped.
>>> 
>>> I don't think this is the expected behavior on Linux: I did some testing to check what happens with other devices and I can experience only the first behavior on Intel and Atheros WiFi nics as well as on Fast Ethernet nics (in this case I run iperf -b100M :-) independently of the kernel version.
>>> 
>>> Strangely the b43 sources in 2.6.35 are really similar to those in 2.6.34-rc7 and the differences do not seem to justify the different behavior. There are also other weird observations (like qdisc never used in<  2.6.34-rc7) but I would like to have a first opinion from your side.
>>> 
>>> Many thanks,
>>> -Francesco
>>> 
>>> P.S. what reported does not depend on the firmware version. I also tried a few cards (4306, 4311 and 4318) and nothing changed.
>> 
>> Francesco,
>> 
>> I agree that there are no changes in b43 between 2.6.34-rc7 and 2.6.35 that would cause this problem. All but one of the changes are for N PHYs, and that one only removes some braces that are not needed. In addition, there are no changes in ssb that would affect anything other than SPROM loading.
>> 
>> Have you tried your test with a 2.6.38 kernel? Perhaps the problem has already been fixed. The other thing to do would be to try to bisect between .35 and .34-rc7. If you do that, consider the entire kernel, not just b43. If it is impossible for you to do either of the tests, please send me any command files that you are using, and I'll try it here.
> Hi Larry,
> 
> I just finished bisecting, git tells
> 
> 	fc6055a5ba31e2c14e36e8939f9bf2b6d586a7f5 is first bad commit
> 	commit fc6055a5ba31e2c14e36e8939f9bf2b6d586a7f5
> 	Author: Eric Dumazet <eric.dumazet@gmail.com>
> 	Date:   Fri Apr 16 12:18:22 2010 +0000
> 
> Some info to reproduce what I'm seeing:
> 
> - receiver, can be whatever platform, run iperf like this
> 	$: iperf -s -u -p 20002 -i 1
> 
> - sender, whatever firmware, one of 4306-4318-4311 devices, run iperf like this
> 	$: iperf -c RECEIVER_IP -u -p 20002 -b 1000M -i 1 -n 1
Larry,

I forgot to say that I only tested on a single core single CPU pc. Don't tested with SMP.

-Francesco

problems with b43 and greedy traffic

[Tom Gundersen]

On Fri, Apr 22, 2011 at 9:58 PM, Francesco Gringoli
<francesco.gringoli@ing.unibs.it> wrote:
>> I just finished bisecting, git tells
>>
>> ? ? ? fc6055a5ba31e2c14e36e8939f9bf2b6d586a7f5 is first bad commit

Relevant: <http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=cff0d6e6edac7672b3f915bb4fb59f279243b7f9>?

-t

problems with b43 and greedy traffic

[francesco.gringoli at ing.unibs.it]

On Apr 22, 2011, at 11:07 PM, Tom Gundersen wrote:

> On Fri, Apr 22, 2011 at 9:58 PM, Francesco Gringoli
> <francesco.gringoli@ing.unibs.it> wrote:
>>> I just finished bisecting, git tells
>>> 
>>>       fc6055a5ba31e2c14e36e8939f9bf2b6d586a7f5 is first bad commit
> 
> Relevant: <http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=cff0d6e6edac7672b3f915bb4fb59f279243b7f9>?
> 
> -t
Hi Tom,

sure, filling that flag for a socket like in the patch does could help but I'm not sure a device driver should ever do that.

Thanks,
-Francesco

problems with b43 and greedy traffic

[francesco.gringoli at ing.unibs.it]

On Apr 21, 2011, at 11:08 PM, Larry Finger wrote:

> On 04/21/2011 01:31 PM, francesco.gringoli at ing.unibs.it wrote:
>> Hello Michael,
>> 
>> I'm doing experiments sending greedy udp traffic from a b43 station to a b43 access point. I have noticed that switching from 2.6.34-rc7 to 2.6.35 the sendmsg call becomes "almost" non blocking when sending from a Broadcom nic while it is still as usual with other nics.
>> 
>> If I load the channel with a 54Mb/s iperf stream (iperf -b54M ...) on<  2.6.35 I see that the application is blocked times to times when calling sendmsg() so that it is slowed down to the channel capabilities and packets are not internally dropped. Clearly they can still be lost on the air :-)
>> 
>> With>= 2.6.35 the application is never blocked and all the packets exceeding the channel capabilities are internally lost by the kernel: in particular it is the asynchronous tx worker (b43_tx_work) that drops them, since it calls b43_dma_tx even if the interface has been stopped because the dma FIFO queue was full. Apart from packets being lost, the CPU load increases since packets cross all the kernel code, from udp_sendmsg down to b43_dma_tx even if they will be dropped.
>> 
>> I don't think this is the expected behavior on Linux: I did some testing to check what happens with other devices and I can experience only the first behavior on Intel and Atheros WiFi nics as well as on Fast Ethernet nics (in this case I run iperf -b100M :-) independently of the kernel version.
>> 
>> Strangely the b43 sources in 2.6.35 are really similar to those in 2.6.34-rc7 and the differences do not seem to justify the different behavior. There are also other weird observations (like qdisc never used in<  2.6.34-rc7) but I would like to have a first opinion from your side.
>> 
>> Many thanks,
>> -Francesco
>> 
>> P.S. what reported does not depend on the firmware version. I also tried a few cards (4306, 4311 and 4318) and nothing changed.
> 
> Francesco,
> 
> I agree that there are no changes in b43 between 2.6.34-rc7 and 2.6.35 that would cause this problem. All but one of the changes are for N PHYs, and that one only removes some braces that are not needed. In addition, there are no changes in ssb that would affect anything other than SPROM loading.
> 
> Have you tried your test with a 2.6.38 kernel? Perhaps the problem has already been fixed. The other thing to do would be to try to bisect between .35 and .34-rc7. If you do that, consider the entire kernel, not just b43. If it is impossible for you to do either of the tests, please send me any command files that you are using, and I'll try it here.
> 
> Larry
Hi Larry,

I tested SMP kernel and it is affected too. Do you think we should report this as a bug to the kernel bug list? Or could this depend on b43?

Unfortunately skb_orphan_try is called before the skb is sent down to the mac80211/driver: it is hence useless setting the "avoid orphan flag" in the skb within the b43 driver as suggested by Thomas. The next packet will have a different flag (I suppose) and it will be orphaned again.

Many thanks,
-Francesco

problems with b43 and greedy traffic

[Rafał Miłecki]

Hi Francesco,

W dniu 26 kwietnia 2011 12:11 u?ytkownik
<francesco.gringoli@ing.unibs.it> napisa?:
> On Apr 21, 2011, at 11:08 PM, Larry Finger wrote:
>
>> On 04/21/2011 01:31 PM, francesco.gringoli at ing.unibs.it wrote:
>>> Hello Michael,
>>>
>>> I'm doing experiments sending greedy udp traffic from a b43 station to a b43 access point. I have noticed that switching from 2.6.34-rc7 to 2.6.35 the sendmsg call becomes "almost" non blocking when sending from a Broadcom nic while it is still as usual with other nics.
>>>
>>> If I load the channel with a 54Mb/s iperf stream (iperf -b54M ...) on< ?2.6.35 I see that the application is blocked times to times when calling sendmsg() so that it is slowed down to the channel capabilities and packets are not internally dropped. Clearly they can still be lost on the air :-)
>>>
>>> With>= 2.6.35 the application is never blocked and all the packets exceeding the channel capabilities are internally lost by the kernel: in particular it is the asynchronous tx worker (b43_tx_work) that drops them, since it calls b43_dma_tx even if the interface has been stopped because the dma FIFO queue was full. Apart from packets being lost, the CPU load increases since packets cross all the kernel code, from udp_sendmsg down to b43_dma_tx even if they will be dropped.
>>>
>>> I don't think this is the expected behavior on Linux: I did some testing to check what happens with other devices and I can experience only the first behavior on Intel and Atheros WiFi nics as well as on Fast Ethernet nics (in this case I run iperf -b100M :-) independently of the kernel version.
>>>
>>> Strangely the b43 sources in 2.6.35 are really similar to those in 2.6.34-rc7 and the differences do not seem to justify the different behavior. There are also other weird observations (like qdisc never used in< ?2.6.34-rc7) but I would like to have a first opinion from your side.
>>>
>>> Many thanks,
>>> -Francesco
>>>
>>> P.S. what reported does not depend on the firmware version. I also tried a few cards (4306, 4311 and 4318) and nothing changed.
>>
>> Francesco,
>>
>> I agree that there are no changes in b43 between 2.6.34-rc7 and 2.6.35 that would cause this problem. All but one of the changes are for N PHYs, and that one only removes some braces that are not needed. In addition, there are no changes in ssb that would affect anything other than SPROM loading.
>>
>> Have you tried your test with a 2.6.38 kernel? Perhaps the problem has already been fixed. The other thing to do would be to try to bisect between .35 and .34-rc7. If you do that, consider the entire kernel, not just b43. If it is impossible for you to do either of the tests, please send me any command files that you are using, and I'll try it here.
>>
>> Larry
> Hi Larry,
>
> I tested SMP kernel and it is affected too. Do you think we should report this as a bug to the kernel bug list? Or could this depend on b43?
>
> Unfortunately skb_orphan_try is called before the skb is sent down to the mac80211/driver: it is hence useless setting the "avoid orphan flag" in the skb within the b43 driver as suggested by Thomas. The next packet will have a different flag (I suppose) and it will be orphaned again.

I don't really have big knowledge about net architecture. I believe we
should try asking patch commiters about this issue. 	Eric Dumazet and
davem maybe?

-- 
Rafa?

problems with b43 and greedy (UDP) traffic

[Larry Finger]

On 04/26/2011 09:53 AM, Rafa? Mi?ecki wrote:
> Hi Francesco,
>
> W dniu 26 kwietnia 2011 12:11 u?ytkownik
> <francesco.gringoli@ing.unibs.it>  napisa?:
>> On Apr 21, 2011, at 11:08 PM, Larry Finger wrote:
>>
>>> On 04/21/2011 01:31 PM, francesco.gringoli at ing.unibs.it wrote:
>>>> Hello Michael,
>>>>
>>>> I'm doing experiments sending greedy udp traffic from a b43 station to a b43 access point. I have noticed that switching from 2.6.34-rc7 to 2.6.35 the sendmsg call becomes "almost" non blocking when sending from a Broadcom nic while it is still as usual with other nics.
>>>>
>>>> If I load the channel with a 54Mb/s iperf stream (iperf -b54M ...) on<    2.6.35 I see that the application is blocked times to times when calling sendmsg() so that it is slowed down to the channel capabilities and packets are not internally dropped. Clearly they can still be lost on the air :-)
>>>>
>>>> With>= 2.6.35 the application is never blocked and all the packets exceeding the channel capabilities are internally lost by the kernel: in particular it is the asynchronous tx worker (b43_tx_work) that drops them, since it calls b43_dma_tx even if the interface has been stopped because the dma FIFO queue was full. Apart from packets being lost, the CPU load increases since packets cross all the kernel code, from udp_sendmsg down to b43_dma_tx even if they will be dropped.
>>>>
>>>> I don't think this is the expected behavior on Linux: I did some testing to check what happens with other devices and I can experience only the first behavior on Intel and Atheros WiFi nics as well as on Fast Ethernet nics (in this case I run iperf -b100M :-) independently of the kernel version.
>>>>
>>>> Strangely the b43 sources in 2.6.35 are really similar to those in 2.6.34-rc7 and the differences do not seem to justify the different behavior. There are also other weird observations (like qdisc never used in<    2.6.34-rc7) but I would like to have a first opinion from your side.
>>>>
>>>> Many thanks,
>>>> -Francesco
>>>>
>>>> P.S. what reported does not depend on the firmware version. I also tried a few cards (4306, 4311 and 4318) and nothing changed.
>>>
>>> Francesco,
>>>
>>> I agree that there are no changes in b43 between 2.6.34-rc7 and 2.6.35 that would cause this problem. All but one of the changes are for N PHYs, and that one only removes some braces that are not needed. In addition, there are no changes in ssb that would affect anything other than SPROM loading.
>>>
>>> Have you tried your test with a 2.6.38 kernel? Perhaps the problem has already been fixed. The other thing to do would be to try to bisect between .35 and .34-rc7. If you do that, consider the entire kernel, not just b43. If it is impossible for you to do either of the tests, please send me any command files that you are using, and I'll try it here.
>>>
>>> Larry
>> Hi Larry,
>>
>> I tested SMP kernel and it is affected too. Do you think we should report this as a bug to the kernel bug list? Or could this depend on b43?
>>
>> Unfortunately skb_orphan_try is called before the skb is sent down to the mac80211/driver: it is hence useless setting the "avoid orphan flag" in the skb within the b43 driver as suggested by Thomas. The next packet will have a different flag (I suppose) and it will be orphaned again.
>
> I don't really have big knowledge about net architecture. I believe we
> should try asking patch commiters about this issue. 	Eric Dumazet and
> davem maybe?

Francesco,

Sorry that I have not gotten back to this issue. You said that Intel and Atheros 
cards work the way that b43 did before the orphan_skb patch. Which specific 
driver/card combinations? I suspect that the others are stopping the mac80211 
queues in a way that b43 is not, but I would need to analyze the other code first.

I have Cc'd the wireless-testing mailing list. My suspicion is that this is a 
b43 problem, but the mac80211 gurus might have some ideas.

Larry

problems with b43 and greedy (UDP) traffic

[francesco.gringoli at ing.unibs.it]

On Apr 26, 2011, at 5:21 PM, Larry Finger wrote:

> On 04/26/2011 09:53 AM, Rafa? Mi?ecki wrote:
>> Hi Francesco,
>> 
>> W dniu 26 kwietnia 2011 12:11 u?ytkownik
>> <francesco.gringoli@ing.unibs.it>  napisa?:
>>> On Apr 21, 2011, at 11:08 PM, Larry Finger wrote:
>>> 
>>>> On 04/21/2011 01:31 PM, francesco.gringoli at ing.unibs.it wrote:
>>>>> Hello Michael,
>>>>> 
>>>>> I'm doing experiments sending greedy udp traffic from a b43 station to a b43 access point. I have noticed that switching from 2.6.34-rc7 to 2.6.35 the sendmsg call becomes "almost" non blocking when sending from a Broadcom nic while it is still as usual with other nics.
>>>>> 
>>>>> If I load the channel with a 54Mb/s iperf stream (iperf -b54M ...) on<    2.6.35 I see that the application is blocked times to times when calling sendmsg() so that it is slowed down to the channel capabilities and packets are not internally dropped. Clearly they can still be lost on the air :-)
>>>>> 
>>>>> With>= 2.6.35 the application is never blocked and all the packets exceeding the channel capabilities are internally lost by the kernel: in particular it is the asynchronous tx worker (b43_tx_work) that drops them, since it calls b43_dma_tx even if the interface has been stopped because the dma FIFO queue was full. Apart from packets being lost, the CPU load increases since packets cross all the kernel code, from udp_sendmsg down to b43_dma_tx even if they will be dropped.
>>>>> 
>>>>> I don't think this is the expected behavior on Linux: I did some testing to check what happens with other devices and I can experience only the first behavior on Intel and Atheros WiFi nics as well as on Fast Ethernet nics (in this case I run iperf -b100M :-) independently of the kernel version.
>>>>> 
>>>>> Strangely the b43 sources in 2.6.35 are really similar to those in 2.6.34-rc7 and the differences do not seem to justify the different behavior. There are also other weird observations (like qdisc never used in<    2.6.34-rc7) but I would like to have a first opinion from your side.
>>>>> 
>>>>> Many thanks,
>>>>> -Francesco
>>>>> 
>>>>> P.S. what reported does not depend on the firmware version. I also tried a few cards (4306, 4311 and 4318) and nothing changed.
>>>> 
>>>> Francesco,
>>>> 
>>>> I agree that there are no changes in b43 between 2.6.34-rc7 and 2.6.35 that would cause this problem. All but one of the changes are for N PHYs, and that one only removes some braces that are not needed. In addition, there are no changes in ssb that would affect anything other than SPROM loading.
>>>> 
>>>> Have you tried your test with a 2.6.38 kernel? Perhaps the problem has already been fixed. The other thing to do would be to try to bisect between .35 and .34-rc7. If you do that, consider the entire kernel, not just b43. If it is impossible for you to do either of the tests, please send me any command files that you are using, and I'll try it here.
>>>> 
>>>> Larry
>>> Hi Larry,
>>> 
>>> I tested SMP kernel and it is affected too. Do you think we should report this as a bug to the kernel bug list? Or could this depend on b43?
>>> 
>>> Unfortunately skb_orphan_try is called before the skb is sent down to the mac80211/driver: it is hence useless setting the "avoid orphan flag" in the skb within the b43 driver as suggested by Thomas. The next packet will have a different flag (I suppose) and it will be orphaned again.
>> 
>> I don't really have big knowledge about net architecture. I believe we
>> should try asking patch commiters about this issue. 	Eric Dumazet and
>> davem maybe?
> 
> Francesco,
> 
> Sorry that I have not gotten back to this issue. You said that Intel and Atheros cards work the way that b43 did before the orphan_skb patch. Which specific driver/card combinations? I suspect that the others are stopping the mac80211 queues in a way that b43 is not, but I would need to analyze the other code first.
Hi Larry,

oops, I should include cc in my filters. I was not checking the b43-dev since a few days.

Atheros:  TP-Link, TL_WN821N, it is a USB stick based on ar9170 chipset. I tested with ar9170 driver. I works as expected also with >=2.6.35
Intel: ipw2200 b/g but I can't now check the exact model. I used the iw2200 driver. Works as expected also with >=2.6.35.

About stopping the queues: I believe there is a code design mismatch in the b43 source. It is the b43_tx_work code, in fact, responsible to send packet to the board: after 2.6.31.14 when mac80211 calls b43_op_tx, packets are simply queueud. Unfortunately the worker does its job without checking the state of the dma, and when dma is stopped, all packets are dropped ultimately by b43_dma_tx. This should not happen, so that packets are kept in the device queue, and new packets from the upper layers are simply enqueued in the qdisc.

I tried to set up a check in the while loop inside b43_tx_work but if I stop sending packets before the queue is empty because the dma is stopped, then the worker is never called again. The station loses the association and the interface dies (though there is no crash). Actually I don't really know how the work_queue code style works.

Many thanks,
-Francesco

> 
> I have Cc'd the wireless-testing mailing list. My suspicion is that this is a b43 problem, but the mac80211 gurus might have some ideas.
> 
> Larry

problems with b43 and greedy traffic

[francesco.gringoli at ing.unibs.it]

On Apr 26, 2011, at 4:53 PM, Rafa? Mi?ecki wrote:

> Hi Francesco,
> 
> W dniu 26 kwietnia 2011 12:11 u?ytkownik
> <francesco.gringoli@ing.unibs.it> napisa?:
>> On Apr 21, 2011, at 11:08 PM, Larry Finger wrote:
>> 
>>> On 04/21/2011 01:31 PM, francesco.gringoli at ing.unibs.it wrote:
>>>> Hello Michael,
>>>> 
>>>> I'm doing experiments sending greedy udp traffic from a b43 station to a b43 access point. I have noticed that switching from 2.6.34-rc7 to 2.6.35 the sendmsg call becomes "almost" non blocking when sending from a Broadcom nic while it is still as usual with other nics.
>>>> 
>>>> If I load the channel with a 54Mb/s iperf stream (iperf -b54M ...) on<  2.6.35 I see that the application is blocked times to times when calling sendmsg() so that it is slowed down to the channel capabilities and packets are not internally dropped. Clearly they can still be lost on the air :-)
>>>> 
>>>> With>= 2.6.35 the application is never blocked and all the packets exceeding the channel capabilities are internally lost by the kernel: in particular it is the asynchronous tx worker (b43_tx_work) that drops them, since it calls b43_dma_tx even if the interface has been stopped because the dma FIFO queue was full. Apart from packets being lost, the CPU load increases since packets cross all the kernel code, from udp_sendmsg down to b43_dma_tx even if they will be dropped.
>>>> 
>>>> I don't think this is the expected behavior on Linux: I did some testing to check what happens with other devices and I can experience only the first behavior on Intel and Atheros WiFi nics as well as on Fast Ethernet nics (in this case I run iperf -b100M :-) independently of the kernel version.
>>>> 
>>>> Strangely the b43 sources in 2.6.35 are really similar to those in 2.6.34-rc7 and the differences do not seem to justify the different behavior. There are also other weird observations (like qdisc never used in<  2.6.34-rc7) but I would like to have a first opinion from your side.
>>>> 
>>>> Many thanks,
>>>> -Francesco
>>>> 
>>>> P.S. what reported does not depend on the firmware version. I also tried a few cards (4306, 4311 and 4318) and nothing changed.
>>> 
>>> Francesco,
>>> 
>>> I agree that there are no changes in b43 between 2.6.34-rc7 and 2.6.35 that would cause this problem. All but one of the changes are for N PHYs, and that one only removes some braces that are not needed. In addition, there are no changes in ssb that would affect anything other than SPROM loading.
>>> 
>>> Have you tried your test with a 2.6.38 kernel? Perhaps the problem has already been fixed. The other thing to do would be to try to bisect between .35 and .34-rc7. If you do that, consider the entire kernel, not just b43. If it is impossible for you to do either of the tests, please send me any command files that you are using, and I'll try it here.
>>> 
>>> Larry
>> Hi Larry,
>> 
>> I tested SMP kernel and it is affected too. Do you think we should report this as a bug to the kernel bug list? Or could this depend on b43?
>> 
>> Unfortunately skb_orphan_try is called before the skb is sent down to the mac80211/driver: it is hence useless setting the "avoid orphan flag" in the skb within the b43 driver as suggested by Thomas. The next packet will have a different flag (I suppose) and it will be orphaned again.
> 
> I don't really have big knowledge about net architecture. I believe we
> should try asking patch commiters about this issue. 	Eric Dumazet and
> davem maybe?

Hi Rafa?,

thanks for your reply. That could be an idea: patching the kernel to avoid orphaning before dev_hard_start_xmit on a per driver basis requires a dedicated field in some bitmask of the net device structure. Should we contact them on netdev?

-Francesco

> 
> -- 
> Rafa?

problems with b43 and greedy traffic

[francesco.gringoli at ing.unibs.it]

On Apr 26, 2011, at 4:53 PM, Rafa? Mi?ecki wrote:

> Hi Francesco,
> 
> W dniu 26 kwietnia 2011 12:11 u?ytkownik
> <francesco.gringoli@ing.unibs.it> napisa?:
>> On Apr 21, 2011, at 11:08 PM, Larry Finger wrote:
>> 
>>> On 04/21/2011 01:31 PM, francesco.gringoli at ing.unibs.it wrote:
>>>> Hello Michael,
>>>> 
>>>> I'm doing experiments sending greedy udp traffic from a b43 station to a b43 access point. I have noticed that switching from 2.6.34-rc7 to 2.6.35 the sendmsg call becomes "almost" non blocking when sending from a Broadcom nic while it is still as usual with other nics.
>>>> 
>>>> If I load the channel with a 54Mb/s iperf stream (iperf -b54M ...) on<  2.6.35 I see that the application is blocked times to times when calling sendmsg() so that it is slowed down to the channel capabilities and packets are not internally dropped. Clearly they can still be lost on the air :-)
>>>> 
>>>> With>= 2.6.35 the application is never blocked and all the packets exceeding the channel capabilities are internally lost by the kernel: in particular it is the asynchronous tx worker (b43_tx_work) that drops them, since it calls b43_dma_tx even if the interface has been stopped because the dma FIFO queue was full. Apart from packets being lost, the CPU load increases since packets cross all the kernel code, from udp_sendmsg down to b43_dma_tx even if they will be dropped.
>>>> 
>>>> I don't think this is the expected behavior on Linux: I did some testing to check what happens with other devices and I can experience only the first behavior on Intel and Atheros WiFi nics as well as on Fast Ethernet nics (in this case I run iperf -b100M :-) independently of the kernel version.
>>>> 
>>>> Strangely the b43 sources in 2.6.35 are really similar to those in 2.6.34-rc7 and the differences do not seem to justify the different behavior. There are also other weird observations (like qdisc never used in<  2.6.34-rc7) but I would like to have a first opinion from your side.
>>>> 
>>>> Many thanks,
>>>> -Francesco
>>>> 
>>>> P.S. what reported does not depend on the firmware version. I also tried a few cards (4306, 4311 and 4318) and nothing changed.
>>> 
>>> Francesco,
>>> 
>>> I agree that there are no changes in b43 between 2.6.34-rc7 and 2.6.35 that would cause this problem. All but one of the changes are for N PHYs, and that one only removes some braces that are not needed. In addition, there are no changes in ssb that would affect anything other than SPROM loading.
>>> 
>>> Have you tried your test with a 2.6.38 kernel? Perhaps the problem has already been fixed. The other thing to do would be to try to bisect between .35 and .34-rc7. If you do that, consider the entire kernel, not just b43. If it is impossible for you to do either of the tests, please send me any command files that you are using, and I'll try it here.
>>> 
>>> Larry
>> Hi Larry,
>> 
>> I tested SMP kernel and it is affected too. Do you think we should report this as a bug to the kernel bug list? Or could this depend on b43?
>> 
>> Unfortunately skb_orphan_try is called before the skb is sent down to the mac80211/driver: it is hence useless setting the "avoid orphan flag" in the skb within the b43 driver as suggested by Thomas. The next packet will have a different flag (I suppose) and it will be orphaned again.
> 
> I don't really have big knowledge about net architecture. I believe we
> should try asking patch commiters about this issue. 	Eric Dumazet and
> davem maybe?

I noticed other strange aspects in b43, and I would like to share with you my concerns

1) I still use 2.6.29 for my experiments where I'm used to add qdisc (e.g., htb and others) to the b43 wlan device. For sure with 2.6.29 everything is like expected. With newer kernels instead the qdisc behaves weirdly, like if it were broken. I took a look at the code and I see that a lot of packets are dropped in b43_dma_tx because the interface is stopped but the worker b43_tx_work calls b43_dma_tx independently of the state of the interface, as I explain here below.

2) Up to 2.6.31.14, when upper layers send a packet, the code goes through the driver up to b43_op_tx: then we have a direct call to b43_dma_tx, there is no worker in the middle. Here there is a comment

/* We get here only because of a bug in mac80211.
 * Because of a race, one packet may be queued after
 * the queue is stopped, thus we got called when we shouldn't.
 * For now, just refuse the transmit. */

which is sound. Since 2.6.32, b43_op_tx simply queues packets: it is the worker to fetch them and push to b43_dma_tx. The comment however is still there but it does not make sense since b43_dma_tx is called only by the worker. Unfortunately that is not just a comment left out, since the worker actually calls b43_dma_tx even if the interface is stopped. This causes the driver to drop a lot of packets. I believe that this is the reason of the qdisc not working.

I would like to point out that this does not impact only UDP greedy traffic: TCP streams with big windows show the same issue.

Regards,
-Francesco
> 
> 
> -- 
> Rafa?

problems with b43 and greedy traffic

[Rafał Miłecki]

2011/4/21  <francesco.gringoli@ing.unibs.it>:
> Hello Michael,
>
> I'm doing experiments sending greedy udp traffic from a b43 station to a b43 access point. I have noticed that switching from 2.6.34-rc7 to 2.6.35 the sendmsg call becomes "almost" non blocking when sending from a Broadcom nic while it is still as usual with other nics.
>
> If I load the channel with a 54Mb/s iperf stream (iperf -b54M ...) on < 2.6.35 I see that the application is blocked times to times when calling sendmsg() so that it is slowed down to the channel capabilities and packets are not internally dropped. Clearly they can still be lost on the air :-)
>
> With >= 2.6.35 the application is never blocked and all the packets exceeding the channel capabilities are internally lost by the kernel: in particular it is the asynchronous tx worker (b43_tx_work) that drops them, since it calls b43_dma_tx even if the interface has been stopped because the dma FIFO queue was full. Apart from packets being lost, the CPU load increases since packets cross all the kernel code, from udp_sendmsg down to b43_dma_tx even if they will be dropped.
>
> I don't think this is the expected behavior on Linux: I did some testing to check what happens with other devices and I can experience only the first behavior on Intel and Atheros WiFi nics as well as on Fast Ethernet nics (in this case I run iperf -b100M :-) independently of the kernel version.
>
> Strangely the b43 sources in 2.6.35 are really similar to those in 2.6.34-rc7 and the differences do not seem to justify the different behavior. There are also other weird observations (like qdisc never used in < 2.6.34-rc7) but I would like to have a first opinion from your side.
>
> Many thanks,
> -Francesco
>
> P.S. what reported does not depend on the firmware version. I also tried a few cards (4306, 4311 and 4318) and nothing changed.

I have (finally) found some time to dig into it. First of all, I
reproduced this on wireless-testing (3.0-rc2) on my BCM4312. I
"achieved" 34 Gb/s of UDP traffic.

The problem is that we do not stop any queue. I've been transferring
UDP stream for 10 seconds, and queued was not stopped even once.

We stop queue when the ring is full, so I've added debugging message
every time "used_slots" changes for TX ring. Over that 10 seconds the
maximal value of "used_slots" was 14.

Does it mean firmware lies us about transmitted packages? Any hints anyone?

-- 
Rafa?

problems with b43 and greedy traffic

[Michael Büsch]

On Sat, 11 Jun 2011 22:28:00 +0200
Rafa? Mi?ecki <zajec5@gmail.com> wrote:

> 2011/4/21  <francesco.gringoli@ing.unibs.it>:
> > Hello Michael,
> >
> > I'm doing experiments sending greedy udp traffic from a b43 station to a b43 access point. I have noticed that switching from 2.6.34-rc7 to 2.6.35 the sendmsg call becomes "almost" non blocking when sending from a Broadcom nic while it is still as usual with other nics.
> >
> > If I load the channel with a 54Mb/s iperf stream (iperf -b54M ...) on < 2.6.35 I see that the application is blocked times to times when calling sendmsg() so that it is slowed down to the channel capabilities and packets are not internally dropped. Clearly they can still be lost on the air :-)
> >
> > With >= 2.6.35 the application is never blocked and all the packets exceeding the channel capabilities are internally lost by the kernel: in particular it is the asynchronous tx worker (b43_tx_work) that drops them, since it calls b43_dma_tx even if the interface has been stopped because the dma FIFO queue was full. Apart from packets being lost, the CPU load increases since packets cross all the kernel code, from udp_sendmsg down to b43_dma_tx even if they will be dropped.
> >
> > I don't think this is the expected behavior on Linux: I did some testing to check what happens with other devices and I can experience only the first behavior on Intel and Atheros WiFi nics as well as on Fast Ethernet nics (in this case I run iperf -b100M :-) independently of the kernel version.
> >
> > Strangely the b43 sources in 2.6.35 are really similar to those in 2.6.34-rc7 and the differences do not seem to justify the different behavior. There are also other weird observations (like qdisc never used in < 2.6.34-rc7) but I would like to have a first opinion from your side.
> >
> > Many thanks,
> > -Francesco
> >
> > P.S. what reported does not depend on the firmware version. I also tried a few cards (4306, 4311 and 4318) and nothing changed.
> 
> I have (finally) found some time to dig into it. First of all, I
> reproduced this on wireless-testing (3.0-rc2) on my BCM4312. I
> "achieved" 34 Gb/s of UDP traffic.
> 
> The problem is that we do not stop any queue. I've been transferring
> UDP stream for 10 seconds, and queued was not stopped even once.
> 
> We stop queue when the ring is full, so I've added debugging message
> every time "used_slots" changes for TX ring. Over that 10 seconds the
> maximal value of "used_slots" was 14.
> 
> Does it mean firmware lies us about transmitted packages? Any hints anyone?

No that would horribly crash.

I'd rather say that we are dropping packets somewhere before they even reach DMA.

problems with b43 and greedy traffic

[Rafał Miłecki]

2011/6/11 Michael B?sch <m@bues.ch>:
> On Sat, 11 Jun 2011 22:28:00 +0200
> Rafa? Mi?ecki <zajec5@gmail.com> wrote:
>
>> 2011/4/21 ?<francesco.gringoli@ing.unibs.it>:
>> > Hello Michael,
>> >
>> > I'm doing experiments sending greedy udp traffic from a b43 station to a b43 access point. I have noticed that switching from 2.6.34-rc7 to 2.6.35 the sendmsg call becomes "almost" non blocking when sending from a Broadcom nic while it is still as usual with other nics.
>> >
>> > If I load the channel with a 54Mb/s iperf stream (iperf -b54M ...) on < 2.6.35 I see that the application is blocked times to times when calling sendmsg() so that it is slowed down to the channel capabilities and packets are not internally dropped. Clearly they can still be lost on the air :-)
>> >
>> > With >= 2.6.35 the application is never blocked and all the packets exceeding the channel capabilities are internally lost by the kernel: in particular it is the asynchronous tx worker (b43_tx_work) that drops them, since it calls b43_dma_tx even if the interface has been stopped because the dma FIFO queue was full. Apart from packets being lost, the CPU load increases since packets cross all the kernel code, from udp_sendmsg down to b43_dma_tx even if they will be dropped.
>> >
>> > I don't think this is the expected behavior on Linux: I did some testing to check what happens with other devices and I can experience only the first behavior on Intel and Atheros WiFi nics as well as on Fast Ethernet nics (in this case I run iperf -b100M :-) independently of the kernel version.
>> >
>> > Strangely the b43 sources in 2.6.35 are really similar to those in 2.6.34-rc7 and the differences do not seem to justify the different behavior. There are also other weird observations (like qdisc never used in < 2.6.34-rc7) but I would like to have a first opinion from your side.
>> >
>> > Many thanks,
>> > -Francesco
>> >
>> > P.S. what reported does not depend on the firmware version. I also tried a few cards (4306, 4311 and 4318) and nothing changed.
>>
>> I have (finally) found some time to dig into it. First of all, I
>> reproduced this on wireless-testing (3.0-rc2) on my BCM4312. I
>> "achieved" 34 Gb/s of UDP traffic.
>>
>> The problem is that we do not stop any queue. I've been transferring
>> UDP stream for 10 seconds, and queued was not stopped even once.
>>
>> We stop queue when the ring is full, so I've added debugging message
>> every time "used_slots" changes for TX ring. Over that 10 seconds the
>> maximal value of "used_slots" was 14.
>>
>> Does it mean firmware lies us about transmitted packages? Any hints anyone?
>
> No that would horribly crash.

I meant lying like "yes, yes, I transmitted packet you gave me", while
firmware just ate it or sth.


> I'd rather say that we are dropping packets somewhere before they even reach DMA.

Weird, every problem should be reported, especially with DMAVERBOSE.
I'll try to trace packets starting with b43_op_tx anyway.

-- 
Rafa?

problems with b43 and greedy traffic

[Rafał Miłecki]

2011/6/11 Rafa? Mi?ecki <zajec5@gmail.com>:
> 2011/6/11 Michael B?sch <m@bues.ch>:
>> On Sat, 11 Jun 2011 22:28:00 +0200
>> Rafa? Mi?ecki <zajec5@gmail.com> wrote:
>>
>>> 2011/4/21 ?<francesco.gringoli@ing.unibs.it>:
>>> > Hello Michael,
>>> >
>>> > I'm doing experiments sending greedy udp traffic from a b43 station to a b43 access point. I have noticed that switching from 2.6.34-rc7 to 2.6.35 the sendmsg call becomes "almost" non blocking when sending from a Broadcom nic while it is still as usual with other nics.
>>> >
>>> > If I load the channel with a 54Mb/s iperf stream (iperf -b54M ...) on < 2.6.35 I see that the application is blocked times to times when calling sendmsg() so that it is slowed down to the channel capabilities and packets are not internally dropped. Clearly they can still be lost on the air :-)
>>> >
>>> > With >= 2.6.35 the application is never blocked and all the packets exceeding the channel capabilities are internally lost by the kernel: in particular it is the asynchronous tx worker (b43_tx_work) that drops them, since it calls b43_dma_tx even if the interface has been stopped because the dma FIFO queue was full. Apart from packets being lost, the CPU load increases since packets cross all the kernel code, from udp_sendmsg down to b43_dma_tx even if they will be dropped.
>>> >
>>> > I don't think this is the expected behavior on Linux: I did some testing to check what happens with other devices and I can experience only the first behavior on Intel and Atheros WiFi nics as well as on Fast Ethernet nics (in this case I run iperf -b100M :-) independently of the kernel version.
>>> >
>>> > Strangely the b43 sources in 2.6.35 are really similar to those in 2.6.34-rc7 and the differences do not seem to justify the different behavior. There are also other weird observations (like qdisc never used in < 2.6.34-rc7) but I would like to have a first opinion from your side.
>>> >
>>> > Many thanks,
>>> > -Francesco
>>> >
>>> > P.S. what reported does not depend on the firmware version. I also tried a few cards (4306, 4311 and 4318) and nothing changed.
>>>
>>> I have (finally) found some time to dig into it. First of all, I
>>> reproduced this on wireless-testing (3.0-rc2) on my BCM4312. I
>>> "achieved" 34 Gb/s of UDP traffic.
>>>
>>> The problem is that we do not stop any queue. I've been transferring
>>> UDP stream for 10 seconds, and queued was not stopped even once.
>>>
>>> We stop queue when the ring is full, so I've added debugging message
>>> every time "used_slots" changes for TX ring. Over that 10 seconds the
>>> maximal value of "used_slots" was 14.
>>>
>>> Does it mean firmware lies us about transmitted packages? Any hints anyone?
>
>> I'd rather say that we are dropping packets somewhere before they even reach DMA.
>
> Weird, every problem should be reported, especially with DMAVERBOSE.
> I'll try to trace packets starting with b43_op_tx anyway.

I've added some simple
"Queuing skb prot:0x%X len:%d data:%d\n", skb->proto, skb->len, skb->data_len
to b43_op_tx.

iperf says to send UDP stream with speed 34.4 Gb/s.
dmesg | grep "4225" | grep Queuing | wc -l
gives me 90 lines (4225 is a timestamp).

So it seems something before b43 eats our packets. What any why? :|

-- 
Rafa?

problems with b43 and greedy traffic

[Michael Büsch]

On Sat, 11 Jun 2011 23:17:50 +0200
Rafa? Mi?ecki <zajec5@gmail.com> wrote:
> iperf says to send UDP stream with speed 34.4 Gb/s.

iperf in UDP mode did never really work for me on any hardware. What about TCP? Does that work?

problems with b43 and greedy traffic

[francesco.gringoli at ing.unibs.it]

On Jun 11, 2011, at 11:46 PM, Michael B?sch wrote:

> On Sat, 11 Jun 2011 23:17:50 +0200
> Rafa? Mi?ecki <zajec5@gmail.com> wrote:
>> iperf says to send UDP stream with speed 34.4 Gb/s.
> 
> iperf in UDP mode did never really work for me on any hardware. What about TCP? Does that work?
TCP partially hides this problem when you run TCP sessions toward a remote host. Anyway if you put some printk in b43_tx_work you will see that some TCP segments are discarded by it trying to send packets to the dma even if the rings are full. This prevents packets to queue up correctly in the qdisc, so shaping traffic through a b43 interface becomes really difficult.

BR,
-Francesco
 

problems with b43 and greedy traffic

[Michael Büsch]

On Sun, 12 Jun 2011 00:55:58 +0200
francesco.gringoli at ing.unibs.it wrote:

> Anyway if you put some printk in b43_tx_work you will see that some TCP segments are discarded by it trying to send packets to the dma even if the rings are full. This prevents packets to queue up correctly in the qdisc, so shaping traffic through a b43 interface becomes really difficult.

So this doesn't sound like a b43 problem to me.
The driver disables the software queues, if the hardware queue is full.
If the stack continues to push packets to the driver while queues are stopped,
the only thing the driver can do is discard them.

problems with b43 and greedy traffic

[francesco.gringoli at ing.unibs.it]

On Jun 12, 2011, at 1:19 AM, Michael B?sch wrote:

> On Sun, 12 Jun 2011 00:55:58 +0200
> francesco.gringoli at ing.unibs.it wrote:
> 
>> Anyway if you put some printk in b43_tx_work you will see that some TCP segments are discarded by it trying to send packets to the dma even if the rings are full. This prevents packets to queue up correctly in the qdisc, so shaping traffic through a b43 interface becomes really difficult.
> 
> So this doesn't sound like a b43 problem to me.
> The driver disables the software queues, if the hardware queue is full.
> If the stack continues to push packets to the driver while queues are stopped,
> the only thing the driver can do is discard them.
I agree with you, and this was perfectly implemented up to 2.6.31. After the introduction of the asynchronous worker, the behavior you are talking about has been almost broken: if b43_dma_tx stops the queue, the stack does not send any more packet to the driver. It is b43_tx_work that keeps sending packets to the hardware (b43_dma_tx). Rather weird :-)

Try to explain better: imagine the worker finished its previous run and gave up because the mac queue was empty. No problem up to now, but suppose the dma ring has only one more descriptor free and that hundreds of stations are competing for the channel so that the firmware is not dequeuing packets for a long time. Now a bunch of packets is injected by the stack: b43_op_tx pushes them to the mac queue and finally it add works to the work queue.

Now b43_tx_work is run again by the kernel, it pops the first packet, send it to the hardware (b43_dma_tx) that ultimately stops the queues because, as I said before, only one more position was free in the queue. From here on the stack stops sending packets to b43_op_tx (maybe one more packet can pass?), it is b43_tx_work that continues to send all the remaining packets in the mac queue to the hardware that can only drop them.

Take a look at b43_dma_tx, and focus on the comment 

/* We get here only because of a bug in mac80211.
 * Because of a race, one packet may be queued after
 * the queue is stopped, thus we got called when we shouldn't.
 * For now, just refuse the transmit. */

It's not true anymore after 2.6.32 but the code has been left as if it were. It is not "one packet may be queued" because of the mac80211, several instead can be passed here, but only by b43_tx_work, not the mac.

BR,
-Francesco

problems with b43 and greedy traffic

[Michael Büsch]

On Sun, 12 Jun 2011 07:28:10 +0200
francesco.gringoli at ing.unibs.it wrote:

> On Jun 12, 2011, at 1:19 AM, Michael B?sch wrote:
> 
> > On Sun, 12 Jun 2011 00:55:58 +0200
> > francesco.gringoli at ing.unibs.it wrote:
> > 
> >> Anyway if you put some printk in b43_tx_work you will see that some TCP segments are discarded by it trying to send packets to the dma even if the rings are full. This prevents packets to queue up correctly in the qdisc, so shaping traffic through a b43 interface becomes really difficult.
> > 
> > So this doesn't sound like a b43 problem to me.
> > The driver disables the software queues, if the hardware queue is full.
> > If the stack continues to push packets to the driver while queues are stopped,
> > the only thing the driver can do is discard them.
> I agree with you, and this was perfectly implemented up to 2.6.31. After the introduction of the asynchronous worker, the behavior you are talking about has been almost broken: if b43_dma_tx stops the queue, the stack does not send any more packet to the driver. It is b43_tx_work that keeps sending packets to the hardware (b43_dma_tx). Rather weird :-)
> 
> Try to explain better: imagine the worker finished its previous run and gave up because the mac queue was empty. No problem up to now, but suppose the dma ring has only one more descriptor free and that hundreds of stations are competing for the channel so that the firmware is not dequeuing packets for a long time. Now a bunch of packets is injected by the stack: b43_op_tx pushes them to the mac queue and finally it add works to the work queue.
> 
> Now b43_tx_work is run again by the kernel, it pops the first packet, send it to the hardware (b43_dma_tx) that ultimately stops the queues because, as I said before, only one more position was free in the queue. From here on the stack stops sending packets to b43_op_tx (maybe one more packet can pass?), it is b43_tx_work that continues to send all the remaining packets in the mac queue to the hardware that can only drop them.
> 
> Take a look at b43_dma_tx, and focus on the comment 
> 
> /* We get here only because of a bug in mac80211.
>  * Because of a race, one packet may be queued after
>  * the queue is stopped, thus we got called when we shouldn't.
>  * For now, just refuse the transmit. */
> 
> It's not true anymore after 2.6.32 but the code has been left as if it were. It is not "one packet may be queued" because of the mac80211, several instead can be passed here, but only by b43_tx_work, not the mac.

Ok, thanks. I see the bug now.
This should be pretty easy to fix, though. Just don't unqueue the packet
from the b43_tx_work queue, if b43_dma_tx failed to transmit it, due to full ring.
And then poke the TX work again, if the queues are unblocked.

problems with b43 and greedy traffic

[Rafał Miłecki]

2011/6/12 Michael B?sch <m@bues.ch>:
> On Sun, 12 Jun 2011 07:28:10 +0200
> francesco.gringoli at ing.unibs.it wrote:
>
>> On Jun 12, 2011, at 1:19 AM, Michael B?sch wrote:
>>
>> > On Sun, 12 Jun 2011 00:55:58 +0200
>> > francesco.gringoli at ing.unibs.it wrote:
>> >
>> >> Anyway if you put some printk in b43_tx_work you will see that some TCP segments are discarded by it trying to send packets to the dma even if the rings are full. This prevents packets to queue up correctly in the qdisc, so shaping traffic through a b43 interface becomes really difficult.
>> >
>> > So this doesn't sound like a b43 problem to me.
>> > The driver disables the software queues, if the hardware queue is full.
>> > If the stack continues to push packets to the driver while queues are stopped,
>> > the only thing the driver can do is discard them.
>> I agree with you, and this was perfectly implemented up to 2.6.31. After the introduction of the asynchronous worker, the behavior you are talking about has been almost broken: if b43_dma_tx stops the queue, the stack does not send any more packet to the driver. It is b43_tx_work that keeps sending packets to the hardware (b43_dma_tx). Rather weird :-)
>>
>> Try to explain better: imagine the worker finished its previous run and gave up because the mac queue was empty. No problem up to now, but suppose the dma ring has only one more descriptor free and that hundreds of stations are competing for the channel so that the firmware is not dequeuing packets for a long time. Now a bunch of packets is injected by the stack: b43_op_tx pushes them to the mac queue and finally it add works to the work queue.
>>
>> Now b43_tx_work is run again by the kernel, it pops the first packet, send it to the hardware (b43_dma_tx) that ultimately stops the queues because, as I said before, only one more position was free in the queue. From here on the stack stops sending packets to b43_op_tx (maybe one more packet can pass?), it is b43_tx_work that continues to send all the remaining packets in the mac queue to the hardware that can only drop them.
>>
>> Take a look at b43_dma_tx, and focus on the comment
>>
>> /* We get here only because of a bug in mac80211.
>> ?* Because of a race, one packet may be queued after
>> ?* the queue is stopped, thus we got called when we shouldn't.
>> ?* For now, just refuse the transmit. */
>>
>> It's not true anymore after 2.6.32 but the code has been left as if it were. It is not "one packet may be queued" because of the mac80211, several instead can be passed here, but only by b43_tx_work, not the mac.
>
> Ok, thanks. I see the bug now.
> This should be pretty easy to fix, though. Just don't unqueue the packet
> from the b43_tx_work queue, if b43_dma_tx failed to transmit it, due to full ring.
> And then poke the TX work again, if the queues are unblocked.

That's not so easy. We have only 1 TX queue (wl->tx_queue). It may
contain packets for different qos (rings).

We should not stop dequeuing all pending packets, just because on ring
is full. There may be packets for more important ring (qos) which can
still be send to hardware, even if one of the rings is full.

-- 
Rafa?

problems with b43 and greedy traffic

[francesco.gringoli at ing.unibs.it]

On Jun 12, 2011, at 10:19 AM, Michael B?sch wrote:

> On Sun, 12 Jun 2011 07:28:10 +0200
> francesco.gringoli at ing.unibs.it wrote:
> 
>> On Jun 12, 2011, at 1:19 AM, Michael B?sch wrote:
>> 
>>> On Sun, 12 Jun 2011 00:55:58 +0200
>>> francesco.gringoli at ing.unibs.it wrote:
>>> 
>>>> Anyway if you put some printk in b43_tx_work you will see that some TCP segments are discarded by it trying to send packets to the dma even if the rings are full. This prevents packets to queue up correctly in the qdisc, so shaping traffic through a b43 interface becomes really difficult.
>>> 
>>> So this doesn't sound like a b43 problem to me.
>>> The driver disables the software queues, if the hardware queue is full.
>>> If the stack continues to push packets to the driver while queues are stopped,
>>> the only thing the driver can do is discard them.
>> I agree with you, and this was perfectly implemented up to 2.6.31. After the introduction of the asynchronous worker, the behavior you are talking about has been almost broken: if b43_dma_tx stops the queue, the stack does not send any more packet to the driver. It is b43_tx_work that keeps sending packets to the hardware (b43_dma_tx). Rather weird :-)
>> 
>> Try to explain better: imagine the worker finished its previous run and gave up because the mac queue was empty. No problem up to now, but suppose the dma ring has only one more descriptor free and that hundreds of stations are competing for the channel so that the firmware is not dequeuing packets for a long time. Now a bunch of packets is injected by the stack: b43_op_tx pushes them to the mac queue and finally it add works to the work queue.
>> 
>> Now b43_tx_work is run again by the kernel, it pops the first packet, send it to the hardware (b43_dma_tx) that ultimately stops the queues because, as I said before, only one more position was free in the queue. From here on the stack stops sending packets to b43_op_tx (maybe one more packet can pass?), it is b43_tx_work that continues to send all the remaining packets in the mac queue to the hardware that can only drop them.
>> 
>> Take a look at b43_dma_tx, and focus on the comment 
>> 
>> /* We get here only because of a bug in mac80211.
>> * Because of a race, one packet may be queued after
>> * the queue is stopped, thus we got called when we shouldn't.
>> * For now, just refuse the transmit. */
>> 
>> It's not true anymore after 2.6.32 but the code has been left as if it were. It is not "one packet may be queued" because of the mac80211, several instead can be passed here, but only by b43_tx_work, not the mac.
> 
> Ok, thanks. I see the bug now.
> This should be pretty easy to fix, though. Just don't unqueue the packet
> from the b43_tx_work queue, if b43_dma_tx failed to transmit it, due to full ring.
> And then poke the TX work again, if the queues are unblocked.
Hi Michael,

thanks, I see. I can work on it. I think also Rafa? comment is right and we should move the ring check code directly in b43_tx_work which is now the only piece of code to trigger the dma (or pio) engine for transmission.

BR,
-Francesco

problems with b43 and greedy traffic

[Rafał Miłecki]

W dniu 11 czerwca 2011 23:46 u?ytkownik Michael B?sch <m@bues.ch> napisa?:
> On Sat, 11 Jun 2011 23:17:50 +0200
> Rafa? Mi?ecki <zajec5@gmail.com> wrote:
>> iperf says to send UDP stream with speed 34.4 Gb/s.
>
> iperf in UDP mode did never really work for me on any hardware. What about TCP? Does that work?

Generally, yes, it works. However when using pio=1 I've to use qos=0.
That's the only way to don't get disconnected on the first
communication (after authentication). And iperf on TCP does not wok in
pio=1 qos=0 configuration. That's why I started playing with UDP.

-- 
Rafa?

problems with b43 and greedy traffic

[francesco.gringoli at ing.unibs.it]

On Jun 11, 2011, at 11:17 PM, Rafa? Mi?ecki wrote:

> 2011/6/11 Rafa? Mi?ecki <zajec5@gmail.com>:
>> 2011/6/11 Michael B?sch <m@bues.ch>:
>>> On Sat, 11 Jun 2011 22:28:00 +0200
>>> Rafa? Mi?ecki <zajec5@gmail.com> wrote:
>>> 
>>>> 2011/4/21  <francesco.gringoli@ing.unibs.it>:
>>>>> Hello Michael,
>>>>> 
>>>>> I'm doing experiments sending greedy udp traffic from a b43 station to a b43 access point. I have noticed that switching from 2.6.34-rc7 to 2.6.35 the sendmsg call becomes "almost" non blocking when sending from a Broadcom nic while it is still as usual with other nics.
>>>>> 
>>>>> If I load the channel with a 54Mb/s iperf stream (iperf -b54M ...) on < 2.6.35 I see that the application is blocked times to times when calling sendmsg() so that it is slowed down to the channel capabilities and packets are not internally dropped. Clearly they can still be lost on the air :-)
>>>>> 
>>>>> With >= 2.6.35 the application is never blocked and all the packets exceeding the channel capabilities are internally lost by the kernel: in particular it is the asynchronous tx worker (b43_tx_work) that drops them, since it calls b43_dma_tx even if the interface has been stopped because the dma FIFO queue was full. Apart from packets being lost, the CPU load increases since packets cross all the kernel code, from udp_sendmsg down to b43_dma_tx even if they will be dropped.
>>>>> 
>>>>> I don't think this is the expected behavior on Linux: I did some testing to check what happens with other devices and I can experience only the first behavior on Intel and Atheros WiFi nics as well as on Fast Ethernet nics (in this case I run iperf -b100M :-) independently of the kernel version.
>>>>> 
>>>>> Strangely the b43 sources in 2.6.35 are really similar to those in 2.6.34-rc7 and the differences do not seem to justify the different behavior. There are also other weird observations (like qdisc never used in < 2.6.34-rc7) but I would like to have a first opinion from your side.
>>>>> 
>>>>> Many thanks,
>>>>> -Francesco
>>>>> 
>>>>> P.S. what reported does not depend on the firmware version. I also tried a few cards (4306, 4311 and 4318) and nothing changed.
>>>> 
>>>> I have (finally) found some time to dig into it. First of all, I
>>>> reproduced this on wireless-testing (3.0-rc2) on my BCM4312. I
>>>> "achieved" 34 Gb/s of UDP traffic.
>>>> 
>>>> The problem is that we do not stop any queue. I've been transferring
>>>> UDP stream for 10 seconds, and queued was not stopped even once.
>>>> 
>>>> We stop queue when the ring is full, so I've added debugging message
>>>> every time "used_slots" changes for TX ring. Over that 10 seconds the
>>>> maximal value of "used_slots" was 14.
>>>> 
>>>> Does it mean firmware lies us about transmitted packages? Any hints anyone?
>> 
>>> I'd rather say that we are dropping packets somewhere before they even reach DMA.
>> 
>> Weird, every problem should be reported, especially with DMAVERBOSE.
>> I'll try to trace packets starting with b43_op_tx anyway.
> 
> I've added some simple
> "Queuing skb prot:0x%X len:%d data:%d\n", skb->proto, skb->len, skb->data_len
> to b43_op_tx.
> 
> iperf says to send UDP stream with speed 34.4 Gb/s.
> dmesg | grep "4225" | grep Queuing | wc -l
> gives me 90 lines (4225 is a timestamp).
> 
> So it seems something before b43 eats our packets. What any why? :|
Hi Rafa?,

first issue is with skb_orphan_try() called by dev_hard_start_xmit() (skb_orphan_try appeared with 2.6.35). It seems that orphaning the skb before actual transmission has problems with b43. Though it works well with a lot of other devices I tested, with b43 the memory allocated to a given socket seems to never reach the threshold to set that socket as blocked. So the application will never stops sending packets. If you comment that call, you should see iperf behaving correctly.

Second issue is in b43. You can trigger the problem, even with skb_orphan_try() commented, in two ways

1) running multiple instances of iperf;
2) forwarding packets from another interface.

The problem is that with 2.6.32, the transmission chain of b43 was broken in two: packets coming from upper layers are queued by b43_ot_tx into mac80211 queues. Then b43_tx_work asynchronously pops packets from the queues and sends them to b43_dma_tx. Unfortunately the worker never checks if the queues are stopped because the dma ring full: it simply continues popping packets until the queues are empty. But if we have 100 packets in the queue, and the dma stops on the first, then we will drop 99% of the packets.

BR,
-Francesco

> 
> -- 
> Rafa?