WARNING at local_bh_enable while tcp_retransmit

WARNING at local_bh_enable while tcp_retransmit

[Ilya Loginov]

	Hi, I am working on some network drivers.
	First one is raw netdevice for RapidIO packets. Second one is
Ethernet network device that encapsulates Ethernet traffic into RapidIO
messages.
	Ethernet device changes skb->dev to RapidIO device, calls
RapidIO create_header and calls dev_queue_xmit on skb.
	All works well for linear skb's but I have trouble with
multi-fragment skb's when frags have bad alignment. In that case my
controller RapidIO fails to transmit packets. While a bit internal tx
queue with descriptors of underlying RapidIO device overflows and it
returns NETDEV_TX_BUSY in start_xmit.
	TCP stack retransmits packets after timeout and I gets this:

------------[ cut here ]------------
WARNING: at kernel/softirq.c:143 local_bh_enable+0x150/0x158()
Modules linked in: rioth rsmp k128 [last unloaded: k128]
Call Trace:
[<ffffffff80022170>] dump_stack+0x8/0x48
[<ffffffff800518e8>] warn_slowpath_common+0x90/0xb8
[<ffffffff80059ca0>] local_bh_enable+0x150/0x158
[<ffffffff80303f04>] dev_queue_xmit+0x55c/0x730
[<c0000000000591b0>] rio_send+0x1b0/0x380 [rsmp]	<- Stack over RapidIO device (similar to can)
[<c000000000068364>] rioth_start_xmit+0x74/0x88 [rioth] <- Ethernet over RapidIO
[<ffffffff80303620>] dev_hard_start_xmit+0x350/0x578
[<ffffffff8031cff4>] sch_direct_xmit+0x214/0x3a8
[<ffffffff80303e20>] dev_queue_xmit+0x478/0x730
[<ffffffff80332ba8>] ip_finish_output+0x168/0x408
[<ffffffff803312ec>] ip_local_out+0x3c/0x58
[<ffffffff80331bf0>] ip_queue_xmit+0x230/0x4a0
[<ffffffff8034bce0>] tcp_transmit_skb+0x4a8/0xaa0
[<ffffffff8034dfb8>] tcp_retransmit_skb+0x260/0x698
[<ffffffff803504d8>] tcp_retransmit_timer+0x110/0x700
[<ffffffff80350cf0>] tcp_write_timer+0x228/0x278
[<ffffffff80062584>] run_timer_softirq+0x174/0x398
[<ffffffff80059804>] __do_softirq+0x174/0x270
[<ffffffff800599c8>] do_softirq+0xc8/0xf8
[<ffffffff80059d7c>] irq_exit+0x7c/0x88
[<ffffffff80001400>] ret_from_irq+0x0/0x4
[<ffffffff8001ed4c>] cpu_idle+0x1c/0xa0
[<ffffffff8049bc80>] start_kernel+0x518/0x628

---[ end trace cc7486cd1e47e9db ]---

	I watched bonding, but I could not realize why it didn't get
same warning. It use very similar scheme of work.

	Do you have any ideas?

-- 
Ilya Loginov <isloginov@gmail.com>

Re: WARNING at local_bh_enable while tcp_retransmit

[Jarek Poplawski]

Ilya Loginov wrote:
> 	Hi, I am working on some network drivers.
...
> 	I watched bonding, but I could not realize why it didn't get
> same warning. It use very similar scheme of work.
> 
> 	Do you have any ideas?

Hi, isn't any of your drivers disabling irqs on this path?
(If no, what is the kernel version?)

Jarek P.

Re: WARNING at local_bh_enable while tcp_retransmit

[Ilya Loginov]

On Tue, 07 Dec 2010 21:56:19 +0100
Jarek Poplawski <jarkao2@gmail.com> wrote:

> Hi, isn't any of your drivers disabling irqs on this path?

No. But we are in soft_irq because tcp_retransmit. So, may be,
interrupts were really disabled.

> (If no, what is the kernel version?)
> 
> Jarek P.

2.6.34. But this does not matter.

I am afraid of this is common problem in all network drivers using
dev_queue_xmit (for example bonding). I didn't see this problem before,
while I have no retransmits.

-- 
Ilya Loginov <isloginov@gmail.com>

Re: WARNING at local_bh_enable while tcp_retransmit

[Jarek Poplawski]

Ilya Loginov wrote:
> On Tue, 07 Dec 2010 21:56:19 +0100
> Jarek Poplawski <jarkao2@gmail.com> wrote:
> 
>> Hi, isn't any of your drivers disabling irqs on this path?
> 
> No. But we are in soft_irq because tcp_retransmit. So, may be,
> interrupts were really disabled.

Should be easy to check by copying this WARN_ON_ONCE from 
kernel/softirq.c:143 local_bh_enable to the beginning of
your rioth_start_xmit.

Jarek P.