NAPI poll routine happens in interrupt context?

NAPI poll routine happens in interrupt context?

[Joshua Wise]

Hello LKML,

I have recently been working on a network driver for an emulated ultra-simple 
network card, and I've run into a few snags with the NAPI. My current issue 
is that it seems to me that my poll routine is being called from an atomic 
context, so when poll calls rx, and rx calls netif_receive_skb, I end up with 
lots of __might_sleep warnings in the various network layers.

This is not so good. I need every cycle I can get, as this emulator is 
incredibly slow, so burning cycles by printing out the reported badness is 
not really acceptible. Conceivably the badness itself is also an issue.

Before posting here, I did search Google for "lkml napi poll interrupt", 
although I did not find anything relevant to my issue.

If interested, the code is available at http://joshuawise.com/lanlan.c . Some 
notes:

The virtual lan-lan is a very very simple device. It consists of an ioreg that 
maintains state of the device, as described by the ioreg bit defines. It also 
has an ioctlreg that can pass through ioctls to the Linux kernel tap device 
that it's sitting on top of. (This goes with the ifreq seen in the struct.) 
One must always write and read in word-aligned chunks to and from it, for 
simplicity's sake.

Feel free to suggest any modifications that this device might need to make it 
more fully functional. Hopefully we can bring this driver to such a state 
where it will be usable as a replacement skeleton driver for the NAPI.

Please cc: Aaron and myself, as neither of us are subscribed to lkml.

Thanks in advance,
joshua

Re: NAPI poll routine happens in interrupt context?

[Joshua Wise]

It has come to my attention that the link that I posted previously was 
nonfunctional. It has been fixed.

As well, here are some other pertinent details:

This is kernel 2.6.13-rc2, the latest that works with MIPS SMP.

Here is a trace:
Debug: sleeping function called from invalid context at 
arch/mips/math-emu/dsemul.c:137
in_atomic():1, irqs_disabled():0
Call Trace:
 [<ffffffff801406e0>] __might_sleep+0x180/0x198
 [<ffffffff802cec00>] ipv6_rcv+0xc0/0x440
 [<ffffffff80140428>] do_dsemulret+0x68/0x1a0
 [<ffffffff8010b3a4>] do_ade+0x24/0x550
 [<ffffffff80102964>] handle_adel_int+0x3c/0x58
 [<ffffffff80268160>] netif_receive_skb+0x1b0/0x2e0
 [<ffffffff802cec04>] ipv6_rcv+0xc4/0x440
 [<ffffffff80268160>] netif_receive_skb+0x1b0/0x2e0
 [<ffffffff802572c8>] lanlan_poll+0x3e0/0x440
 [<ffffffff8026868c>] net_rx_action+0x16c/0x370
 [<ffffffff802686a8>] net_rx_action+0x188/0x370
 [<ffffffff80154f28>] __do_softirq+0x118/0x250
 [<ffffffff80154f28>] __do_softirq+0x118/0x250
 [<ffffffff80155110>] do_softirq+0xb0/0xe0
 [<ffffffff80101930>] mipsIRQ+0x130/0x1e0
 [<ffffffff80101c90>] r4k_wait+0x0/0x10
 [<ffffffff80103e6c>] cpu_idle+0x4c/0x68
 [<ffffffff80103e64>] cpu_idle+0x44/0x68
 [<ffffffff8037fcfc>] start_kernel+0x454/0x4e8
 [<ffffffff8037fcf4>] start_kernel+0x44c/0x4e8

Apologies for any inconvenience.

joshua

On Wednesday 17 August 2005 09:32, Joshua Wise wrote:
> Hello LKML,
>
> I have recently been working on a network driver for an emulated
> ultra-simple network card, and I've run into a few snags with the NAPI. My
> current issue is that it seems to me that my poll routine is being called
> from an atomic context, so when poll calls rx, and rx calls
> netif_receive_skb, I end up with lots of __might_sleep warnings in the
> various network layers.
>
> This is not so good. I need every cycle I can get, as this emulator is
> incredibly slow, so burning cycles by printing out the reported badness is
> not really acceptible. Conceivably the badness itself is also an issue.
>
> Before posting here, I did search Google for "lkml napi poll interrupt",
> although I did not find anything relevant to my issue.
>
> If interested, the code is available at http://joshuawise.com/lanlan.c .
> Some notes:
>
> The virtual lan-lan is a very very simple device. It consists of an ioreg
> that maintains state of the device, as described by the ioreg bit defines.
> It also has an ioctlreg that can pass through ioctls to the Linux kernel
> tap device that it's sitting on top of. (This goes with the ifreq seen in
> the struct.) One must always write and read in word-aligned chunks to and
> from it, for simplicity's sake.
>
> Feel free to suggest any modifications that this device might need to make
> it more fully functional. Hopefully we can bring this driver to such a
> state where it will be usable as a replacement skeleton driver for the
> NAPI.
>
> Please cc: Aaron and myself, as neither of us are subscribed to lkml.
>
> Thanks in advance,
> joshua

Re: NAPI poll routine happens in interrupt context?

[Stephen Hemminger]

On Wed, 17 Aug 2005 09:32:10 -0400
Joshua Wise <Joshua.Wise@sicortex.com> wrote:

> Hello LKML,

You will get more response to network issues on netdev@vger.kernel.org
 
> I have recently been working on a network driver for an emulated ultra-simple 
> network card, and I've run into a few snags with the NAPI. My current issue 
> is that it seems to me that my poll routine is being called from an atomic 
> context, so when poll calls rx, and rx calls netif_receive_skb, I end up with 
> lots of __might_sleep warnings in the various network layers.

NAPI poll is usually called from softirq context.  This means that
hardware interrupts are enabled, but it is not in a thread context that
can sleep.

> This is not so good. I need every cycle I can get, as this emulator is 
> incredibly slow, so burning cycles by printing out the reported badness is 
> not really acceptible. Conceivably the badness itself is also an issue.

You shouldn't be calling things that could sleep! If you are it
is a bug.

> Before posting here, I did search Google for "lkml napi poll interrupt", 
> although I did not find anything relevant to my issue.
> 
> If interested, the code is available at http://joshuawise.com/lanlan.c . Some 
> notes:
> 
> The virtual lan-lan is a very very simple device. It consists of an ioreg that 
> maintains state of the device, as described by the ioreg bit defines. It also 
> has an ioctlreg that can pass through ioctls to the Linux kernel tap device 
> that it's sitting on top of. (This goes with the ifreq seen in the struct.) 
> One must always write and read in word-aligned chunks to and from it, for 
> simplicity's sake.

Harald Welte is working on a generic virtual Ethernet device, perhaps
you could collaborate with him.

> Feel free to suggest any modifications that this device might need to make it 
> more fully functional. Hopefully we can bring this driver to such a state 
> where it will be usable as a replacement skeleton driver for the NAPI.

> Here is a trace:
> Debug: sleeping function called from invalid context at 
> arch/mips/math-emu/dsemul.c:137
> in_atomic():1, irqs_disabled():0
> Call Trace:
>  [<ffffffff801406e0>] __might_sleep+0x180/0x198
>  [<ffffffff802cec00>] ipv6_rcv+0xc0/0x440
>  [<ffffffff80140428>] do_dsemulret+0x68/0x1a0
>  [<ffffffff8010b3a4>] do_ade+0x24/0x550
>  [<ffffffff80102964>] handle_adel_int+0x3c/0x58
>  [<ffffffff80268160>] netif_receive_skb+0x1b0/0x2e0
>  [<ffffffff802cec04>] ipv6_rcv+0xc4/0x440
>  [<ffffffff80268160>] netif_receive_skb+0x1b0/0x2e0
>  [<ffffffff802572c8>] lanlan_poll+0x3e0/0x440
>  [<ffffffff8026868c>] net_rx_action+0x16c/0x370
>  [<ffffffff802686a8>] net_rx_action+0x188/0x370
>  [<ffffffff80154f28>] __do_softirq+0x118/0x250
>  [<ffffffff80154f28>] __do_softirq+0x118/0x250
>  [<ffffffff80155110>] do_softirq+0xb0/0xe0
>  [<ffffffff80101930>] mipsIRQ+0x130/0x1e0
>  [<ffffffff80101c90>] r4k_wait+0x0/0x10
>  [<ffffffff80103e6c>] cpu_idle+0x4c/0x68
>  [<ffffffff80103e64>] cpu_idle+0x44/0x68
>  [<ffffffff8037fcfc>] start_kernel+0x454/0x4e8
>  [<ffffffff8037fcf4>] start_kernel+0x44c/0x4e8

The bug is that ipv6 is doing an operation to handle MIB statistics and
the MIPS architecture math routines seem to need to sleep. 
Previous versions of SNMP code may have done atomic operations, but
current 2.6 code uses per-cpu variables. 
Also, there is no might sleep in the current 2.6 MIPS code either
so the problem is probably fixed if you use current 2.6.12 or later 
kernel.

Thanks
Steve

Re: NAPI poll routine happens in interrupt context?

[Joshua Wise]

On Wednesday 17 August 2005 12:43, Stephen Hemminger wrote:
> You will get more response to network issues on netdev@vger.kernel.org
Okay. Thanks.

> NAPI poll is usually called from softirq context.  This means that
> hardware interrupts are enabled, but it is not in a thread context that
> can sleep.
Okay. I wasn't aware of quite how it was "supposed" to be.

> You shouldn't be calling things that could sleep! If you are it
> is a bug.
I guess I'd better track down this bug, then :)

> Harald Welte is working on a generic virtual Ethernet device, perhaps
> you could collaborate with him.
I assume he is on this mailing list?

> The bug is that ipv6 is doing an operation to handle MIB statistics and
> the MIPS architecture math routines seem to need to sleep.
> Previous versions of SNMP code may have done atomic operations, but
> current 2.6 code uses per-cpu variables.
> Also, there is no might sleep in the current 2.6 MIPS code either
> so the problem is probably fixed if you use current 2.6.12 or later
> kernel.
Hm -- I am using 2.6.13-rc2.
Here is a new trace, showing the same issue with IPv4:

Debug: sleeping function called from invalid context at 
arch/mips/math-emu/dsemul.c:137 
in_atomic():1, irqs_disabled():0
Call Trace:
 [<ffffffff801406e0>] __might_sleep+0x180/0x198 (kernel/sched.c:5223)
 [<ffffffff80101930>] mipsIRQ+0x130/0x1e0 (arch/mips/sc1000/mipsIRQ.S:95)
 [<ffffffff802860fc>] ip_rcv+0x9c/0x7b0 (net/ipv4/ip_input.c:381)
 [<ffffffff80140428>] do_dsemulret+0x68/0x1a0 
(arch/mips/math-emu/dsemul.c:137)
 [<ffffffff8010b3a4>] do_ade+0x24/0x550 (arch/mips/kernel/unaligned.c:506)
 [<ffffffff80102964>] handle_adel_int+0x3c/0x58 (arch/mips/kernel/genex.S:281)
 [<ffffffff80268260>] netif_receive_skb+0x1b0/0x2e0 (net/core/dev.c:1646)
 [<ffffffff80286100>] ip_rcv+0xa0/0x7b0 (net/ipv4/ip_input.c:394)
 [<ffffffff8014da5c>] printk+0x2c/0x38 (kernel/printk.c:515)
 [<ffffffff80268260>] netif_receive_skb+0x1b0/0x2e0 (net/core/dev.c:1646)
 [<ffffffff802573c8>] lanlan_poll+0x3e0/0x440 (drivers/net/lanlan.c:246)
etc, etc.

CC:'ing to linux-mips for obvious reasons. This seems to stem from an 
unaligned access. If this is no longer appropriate for linux-kernel, feel 
free to stop CCing to there, and I will follow.

Thanks,
joshua

Re: NAPI poll routine happens in interrupt context?

[Ralf Baechle]

On Wed, Aug 17, 2005 at 01:21:18PM -0400, Joshua Wise wrote:

> > The bug is that ipv6 is doing an operation to handle MIB statistics and
> > the MIPS architecture math routines seem to need to sleep.

Except nothing in the network stack is using fp - the use of FP inside the
MIPS kernel is not supported in any way.  What happend is probably the
fetching of the opcode of the instruction in the branch delay slot of
the unaligned instruction emulator blew up because it uses a get_user which
again calls might_sleep and that won't exactly work if not called from
process context.

  Ralf

Re: NAPI poll routine happens in interrupt context?

[David S. Miller]

From: Joshua Wise <Joshua.Wise@sicortex.com>
Date: Wed, 17 Aug 2005 09:32:10 -0400

> I have recently been working on a network driver for an emulated ultra-simple 
> network card, and I've run into a few snags with the NAPI. My current issue 
> is that it seems to me that my poll routine is being called from an atomic 
> context, so when poll calls rx, and rx calls netif_receive_skb, I end up with 
> lots of __might_sleep warnings in the various network layers.

NAPI's ->poll method is invoked from software IRQ context.