[PATCH RFC] mac80211: fix "NOHZ: local_softirq_pending 08"

[PATCH RFC] mac80211: fix "NOHZ: local_softirq_pending 08"

[Johannes Stezenbach]

ieee80211_tx_status() documentation says "This function may not be
called in IRQ context", and it is called by rt2800usb
from a workqueue context.  Thus it needs to call
netif_rx_ni() instead of netif_rx().
This change fixes the "NOHZ: local_softirq_pending 08"
messages I've been getting with rt2800usb.

Signed-off-by: Johannes Stezenbach <js@sig21.net>
--
 net/mac80211/status.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/net/mac80211/status.c b/net/mac80211/status.c
index bed7e32..ba8dd8e 100644
--- a/net/mac80211/status.c
+++ b/net/mac80211/status.c
@@ -403,7 +403,7 @@ void ieee80211_tx_status(struct ieee80211_hw *hw, struct sk_buff *skb)
 				skb2 = skb_clone(skb, GFP_ATOMIC);
 				if (skb2) {
 					skb2->dev = prev_dev;
-					netif_rx(skb2);
+					netif_rx_ni(skb2);
 				}
 			}
 
@@ -412,7 +412,7 @@ void ieee80211_tx_status(struct ieee80211_hw *hw, struct sk_buff *skb)
 	}
 	if (prev_dev) {
 		skb->dev = prev_dev;
-		netif_rx(skb);
+		netif_rx_ni(skb);
 		skb = NULL;
 	}
 	rcu_read_unlock();

Re: [PATCH RFC] mac80211: fix "NOHZ: local_softirq_pending 08"

[Johannes Berg]

On Mon, 2010-11-29 at 15:54 +0100, Johannes Stezenbach wrote:
> ieee80211_tx_status() documentation says "This function may not be
> called in IRQ context", and it is called by rt2800usb
> from a workqueue context.  Thus it needs to call
> netif_rx_ni() instead of netif_rx().
> This change fixes the "NOHZ: local_softirq_pending 08"
> messages I've been getting with rt2800usb.

> -					netif_rx(skb2);
> +					netif_rx_ni(skb2);

That's kinda pointless though for drivers that already call it from a
tasklet or similar -- how about instead adding an
ieee80211_tx_status_ni() inline along the lines of ieee80211_rx_ni()?

johannes

Re: [PATCH RFC] mac80211: fix "NOHZ: local_softirq_pending 08"

[Johannes Stezenbach]

On Mon, Nov 29, 2010 at 04:09:09PM +0100, Johannes Berg wrote:
> On Mon, 2010-11-29 at 15:54 +0100, Johannes Stezenbach wrote:
> > ieee80211_tx_status() documentation says "This function may not be
> > called in IRQ context", and it is called by rt2800usb
> > from a workqueue context.  Thus it needs to call
> > netif_rx_ni() instead of netif_rx().
> > This change fixes the "NOHZ: local_softirq_pending 08"
> > messages I've been getting with rt2800usb.
> 
> > -					netif_rx(skb2);
> > +					netif_rx_ni(skb2);
> 
> That's kinda pointless though for drivers that already call it from a
> tasklet or similar -- how about instead adding an
> ieee80211_tx_status_ni() inline along the lines of ieee80211_rx_ni()?

It gets confusing...
There already is ieee80211_tx_status_irqsafe(), but you want a third
option, right?

Johannes

Re: [PATCH RFC] mac80211: fix "NOHZ: local_softirq_pending 08"

[Johannes Berg]

On Mon, 2010-11-29 at 16:27 +0100, Johannes Stezenbach wrote:

> > That's kinda pointless though for drivers that already call it from a
> > tasklet or similar -- how about instead adding an
> > ieee80211_tx_status_ni() inline along the lines of ieee80211_rx_ni()?
> 
> It gets confusing...
> There already is ieee80211_tx_status_irqsafe(), but you want a third
> option, right?

Yes: _irqsafe has to go through the tasklet, _ni just has to disable
local BHs.

johannes

[PATCH v2 RFC] mac80211: fix "NOHZ: local_softirq_pending 08"

[Johannes Stezenbach]

ieee80211_tx_status() documentation says "This function may not be
called in IRQ context", and it is called by rt2800usb
from a workqueue context.  However, ieee80211_tx_status() is
meant to be called from tasklets and thus uses netif_rx().
Add a new ieee80211_tx_status_ni() which does the same
thing but uses netif_rx_ni() instead of netif_rx().

This change fixes the "NOHZ: local_softirq_pending 08"
messages I've been getting with rt2800usb.

Signed-off-by: Johannes Stezenbach <js@sig21.net>
---
v2: add ieee80211_tx_status_ni()

Note: the patch is now incomplete, rt2x00lib_txdone() is used by
several drivers and I'm currently checking if ieee80211_tx_status_ni()
can be used by all of them of if I need to split rt2x00lib_txdone()
in two versions.  An updated patch with the rt2x00
changes will follow, but I'm seeking feedback if the base
change here is OK.


 include/net/mac80211.h |   23 +++++++++++++++++++----
 net/mac80211/status.c  |   19 ++++++++++++++++---
 2 files changed, 35 insertions(+), 7 deletions(-)

diff --git a/include/net/mac80211.h b/include/net/mac80211.h
index eaa4aff..df42312 100644
--- a/include/net/mac80211.h
+++ b/include/net/mac80211.h
@@ -2055,8 +2055,8 @@ static inline void ieee80211_rx_ni(struct ieee80211_hw *hw,
  *
  * This function may not be called in IRQ context. Calls to this function
  * for a single hardware must be synchronized against each other. Calls
- * to this function and ieee80211_tx_status_irqsafe() may not be mixed
- * for a single hardware.
+ * to this function, ieee80211_tx_status_ni() and ieee80211_tx_status_irqsafe()
+ * may not be mixed for a single hardware.
  *
  * @hw: the hardware the frame was transmitted by
  * @skb: the frame that was transmitted, owned by mac80211 after this call
@@ -2065,13 +2065,28 @@ void ieee80211_tx_status(struct ieee80211_hw *hw,
 			 struct sk_buff *skb);
 
 /**
+ * ieee80211_tx_status_ni - transmit status callback (in process context)
+ *
+ * Like ieee80211_tx_status() but can be called in process context.
+ *
+ * Calls to this function, ieee80211_tx_status() and
+ * ieee80211_tx_status_irqsafe() may not be mixed
+ * for a single hardware.
+ *
+ * @hw: the hardware the frame was transmitted by
+ * @skb: the frame that was transmitted, owned by mac80211 after this call
+ */
+void ieee80211_tx_status_ni(struct ieee80211_hw *hw,
+			    struct sk_buff *skb);
+
+/**
  * ieee80211_tx_status_irqsafe - IRQ-safe transmit status callback
  *
  * Like ieee80211_tx_status() but can be called in IRQ context
  * (internally defers to a tasklet.)
  *
- * Calls to this function and ieee80211_tx_status() may not be mixed for a
- * single hardware.
+ * Calls to this function, ieee80211_tx_status() and
+ * ieee80211_tx_status_ni() may not be mixed for a single hardware.
  *
  * @hw: the hardware the frame was transmitted by
  * @skb: the frame that was transmitted, owned by mac80211 after this call
diff --git a/net/mac80211/status.c b/net/mac80211/status.c
index bed7e32..ce4d239 100644
--- a/net/mac80211/status.c
+++ b/net/mac80211/status.c
@@ -170,7 +170,8 @@ static void ieee80211_frame_acked(struct sta_info *sta, struct sk_buff *skb)
  */
 #define STA_LOST_PKT_THRESHOLD	50
 
-void ieee80211_tx_status(struct ieee80211_hw *hw, struct sk_buff *skb)
+void __ieee80211_tx_status(struct ieee80211_hw *hw, struct sk_buff *skb,
+			   int (*netif_rx_func)(struct sk_buff *skb))
 {
 	struct sk_buff *skb2;
 	struct ieee80211_hdr *hdr = (struct ieee80211_hdr *) skb->data;
@@ -403,7 +404,7 @@ void ieee80211_tx_status(struct ieee80211_hw *hw, struct sk_buff *skb)
 				skb2 = skb_clone(skb, GFP_ATOMIC);
 				if (skb2) {
 					skb2->dev = prev_dev;
-					netif_rx(skb2);
+					netif_rx_func(skb2);
 				}
 			}
 
@@ -412,10 +413,22 @@ void ieee80211_tx_status(struct ieee80211_hw *hw, struct sk_buff *skb)
 	}
 	if (prev_dev) {
 		skb->dev = prev_dev;
-		netif_rx(skb);
+		netif_rx_func(skb);
 		skb = NULL;
 	}
 	rcu_read_unlock();
 	dev_kfree_skb(skb);
 }
+
+void ieee80211_tx_status(struct ieee80211_hw *hw, struct sk_buff *skb)
+{
+	__ieee80211_tx_status(hw, skb, netif_rx);
+}
 EXPORT_SYMBOL(ieee80211_tx_status);
+
+void ieee80211_tx_status_ni(struct ieee80211_hw *hw,
+			    struct sk_buff *skb)
+{
+	__ieee80211_tx_status(hw, skb, netif_rx_ni);
+}
+EXPORT_SYMBOL(ieee80211_tx_status_ni);

Re: [PATCH v2 RFC] mac80211: fix "NOHZ: local_softirq_pending 08"

[Johannes Berg]

On Mon, 2010-11-29 at 16:51 +0100, Johannes Stezenbach wrote:


> -void ieee80211_tx_status(struct ieee80211_hw *hw, struct sk_buff *skb)
> +void __ieee80211_tx_status(struct ieee80211_hw *hw, struct sk_buff *skb,
> +			   int (*netif_rx_func)(struct sk_buff *skb))

This seems weird to me -- why not just do

static inline void ieee80211_tx_status_ni(...)
{
	local_bh_disable();
	ieee80211_tx_status(...);
	local_bh_enable();
}

johannes

[PATCH v3 RFC] mac80211: add ieee80211_tx_status_ni()

[Johannes Stezenbach]

ieee80211_tx_status() documentation says "This function may not be
called in IRQ context", and it is called by rt2800usb
from a workqueue context.  However, ieee80211_tx_status() is
meant to be called from tasklets and thus uses netif_rx().
Add a new ieee80211_tx_status_ni() which does the same
thing but is safe to be called from process context.

Using ieee80211_tx_status_ni() for rt2800usb fixes the
"NOHZ: local_softirq_pending 08" messages I've been getting.

Signed-off-by: Johannes Stezenbach <js@sig21.net>
---
v2: add ieee80211_tx_status_ni()
v3: simplify accorindg to review

Note: the patch is now incomplete, rt2x00lib_txdone() is used by
several drivers, rt2800usb calls it from a workqueue while
e.g. while rt2800pci calls it from a tasklet
(rt2800pci_txstatus_tasklet).  I need to sort this out with
the rt2x00 developers.


 include/net/mac80211.h |   28 ++++++++++++++++++++++++----
 1 files changed, 24 insertions(+), 4 deletions(-)

diff --git a/include/net/mac80211.h b/include/net/mac80211.h
index eaa4aff..e411cf8 100644
--- a/include/net/mac80211.h
+++ b/include/net/mac80211.h
@@ -2055,8 +2055,8 @@ static inline void ieee80211_rx_ni(struct ieee80211_hw *hw,
  *
  * This function may not be called in IRQ context. Calls to this function
  * for a single hardware must be synchronized against each other. Calls
- * to this function and ieee80211_tx_status_irqsafe() may not be mixed
- * for a single hardware.
+ * to this function, ieee80211_tx_status_ni() and ieee80211_tx_status_irqsafe()
+ * may not be mixed for a single hardware.
  *
  * @hw: the hardware the frame was transmitted by
  * @skb: the frame that was transmitted, owned by mac80211 after this call
@@ -2065,13 +2065,33 @@ void ieee80211_tx_status(struct ieee80211_hw *hw,
 			 struct sk_buff *skb);
 
 /**
+ * ieee80211_tx_status_ni - transmit status callback (in process context)
+ *
+ * Like ieee80211_tx_status() but can be called in process context.
+ *
+ * Calls to this function, ieee80211_tx_status() and
+ * ieee80211_tx_status_irqsafe() may not be mixed
+ * for a single hardware.
+ *
+ * @hw: the hardware the frame was transmitted by
+ * @skb: the frame that was transmitted, owned by mac80211 after this call
+ */
+static inline void ieee80211_tx_status_ni(struct ieee80211_hw *hw,
+					  struct sk_buff *skb)
+{
+	local_bh_disable();
+	ieee80211_tx_status(hw, skb);
+	local_bh_enable();
+}
+
+/**
  * ieee80211_tx_status_irqsafe - IRQ-safe transmit status callback
  *
  * Like ieee80211_tx_status() but can be called in IRQ context
  * (internally defers to a tasklet.)
  *
- * Calls to this function and ieee80211_tx_status() may not be mixed for a
- * single hardware.
+ * Calls to this function, ieee80211_tx_status() and
+ * ieee80211_tx_status_ni() may not be mixed for a single hardware.
  *
  * @hw: the hardware the frame was transmitted by
  * @skb: the frame that was transmitted, owned by mac80211 after this call

[PATCH v4] mac80211/rt2x00: add ieee80211_tx_status_ni()

[Johannes Stezenbach]

All rt2x00 drivers except rt2800pci call ieee80211_tx_status() from
a workqueue, which causes "NOHZ: local_softirq_pending 08" messages.

To fix it, add ieee80211_tx_status_ni() similar to ieee80211_rx_ni()
which can be called from process context, and call it from
rt2x00lib_txdone().  For the rt2800pci special case a driver
flag is introduced.

Signed-off-by: Johannes Stezenbach <js@sig21.net>
---
v2: add ieee80211_tx_status_ni()
v3: simplify accorindg to review
v4: add rt2x00 as first user of ieee80211_tx_status_ni()

The DRIVER_REQUIRE_TASKLET_CONTEXT flag may seem ugly, but there is
a pending rewrite of the rt2800pci irq handling code by
Helmut, so we decided to use this easy solution.

Note: Only tested with rt2800usb.

 drivers/net/wireless/rt2x00/rt2800pci.c |    1 +
 drivers/net/wireless/rt2x00/rt2x00.h    |    1 +
 drivers/net/wireless/rt2x00/rt2x00dev.c |    9 ++++++---
 include/net/mac80211.h                  |   28 ++++++++++++++++++++++++----
 4 files changed, 32 insertions(+), 7 deletions(-)

diff --git a/drivers/net/wireless/rt2x00/rt2800pci.c b/drivers/net/wireless/rt2x00/rt2800pci.c
index b48d2d3..3ef0228 100644
--- a/drivers/net/wireless/rt2x00/rt2800pci.c
+++ b/drivers/net/wireless/rt2x00/rt2800pci.c
@@ -932,6 +932,7 @@ static int rt2800pci_probe_hw(struct rt2x00_dev *rt2x00dev)
 	__set_bit(DRIVER_REQUIRE_DMA, &rt2x00dev->flags);
 	__set_bit(DRIVER_REQUIRE_L2PAD, &rt2x00dev->flags);
 	__set_bit(DRIVER_REQUIRE_TXSTATUS_FIFO, &rt2x00dev->flags);
+	__set_bit(DRIVER_REQUIRE_TASKLET_CONTEXT, &rt2x00dev->flags);
 	if (!modparam_nohwcrypt)
 		__set_bit(CONFIG_SUPPORT_HW_CRYPTO, &rt2x00dev->flags);
 	__set_bit(DRIVER_SUPPORT_LINK_TUNING, &rt2x00dev->flags);
diff --git a/drivers/net/wireless/rt2x00/rt2x00.h b/drivers/net/wireless/rt2x00/rt2x00.h
index 556b744..28ea59a 100644
--- a/drivers/net/wireless/rt2x00/rt2x00.h
+++ b/drivers/net/wireless/rt2x00/rt2x00.h
@@ -670,6 +670,7 @@ enum rt2x00_flags {
 	DRIVER_REQUIRE_COPY_IV,
 	DRIVER_REQUIRE_L2PAD,
 	DRIVER_REQUIRE_TXSTATUS_FIFO,
+	DRIVER_REQUIRE_TASKLET_CONTEXT,
 
 	/*
 	 * Driver features
diff --git a/drivers/net/wireless/rt2x00/rt2x00dev.c b/drivers/net/wireless/rt2x00/rt2x00dev.c
index 87a421b..fa74acd 100644
--- a/drivers/net/wireless/rt2x00/rt2x00dev.c
+++ b/drivers/net/wireless/rt2x00/rt2x00dev.c
@@ -376,9 +376,12 @@ void rt2x00lib_txdone(struct queue_entry *entry,
 	 * through a mac80211 library call (RTS/CTS) then we should not
 	 * send the status report back.
 	 */
-	if (!(skbdesc_flags & SKBDESC_NOT_MAC80211))
-		ieee80211_tx_status(rt2x00dev->hw, entry->skb);
-	else
+	if (!(skbdesc_flags & SKBDESC_NOT_MAC80211)) {
+		if (test_bit(DRIVER_REQUIRE_TASKLET_CONTEXT, &rt2x00dev->flags))
+			ieee80211_tx_status(rt2x00dev->hw, entry->skb);
+		else
+			ieee80211_tx_status_ni(rt2x00dev->hw, entry->skb);
+	} else
 		dev_kfree_skb_any(entry->skb);
 
 	/*
diff --git a/include/net/mac80211.h b/include/net/mac80211.h
index eaa4aff..e411cf8 100644
--- a/include/net/mac80211.h
+++ b/include/net/mac80211.h
@@ -2055,8 +2055,8 @@ static inline void ieee80211_rx_ni(struct ieee80211_hw *hw,
  *
  * This function may not be called in IRQ context. Calls to this function
  * for a single hardware must be synchronized against each other. Calls
- * to this function and ieee80211_tx_status_irqsafe() may not be mixed
- * for a single hardware.
+ * to this function, ieee80211_tx_status_ni() and ieee80211_tx_status_irqsafe()
+ * may not be mixed for a single hardware.
  *
  * @hw: the hardware the frame was transmitted by
  * @skb: the frame that was transmitted, owned by mac80211 after this call
@@ -2065,13 +2065,33 @@ void ieee80211_tx_status(struct ieee80211_hw *hw,
 			 struct sk_buff *skb);
 
 /**
+ * ieee80211_tx_status_ni - transmit status callback (in process context)
+ *
+ * Like ieee80211_tx_status() but can be called in process context.
+ *
+ * Calls to this function, ieee80211_tx_status() and
+ * ieee80211_tx_status_irqsafe() may not be mixed
+ * for a single hardware.
+ *
+ * @hw: the hardware the frame was transmitted by
+ * @skb: the frame that was transmitted, owned by mac80211 after this call
+ */
+static inline void ieee80211_tx_status_ni(struct ieee80211_hw *hw,
+					  struct sk_buff *skb)
+{
+	local_bh_disable();
+	ieee80211_tx_status(hw, skb);
+	local_bh_enable();
+}
+
+/**
  * ieee80211_tx_status_irqsafe - IRQ-safe transmit status callback
  *
  * Like ieee80211_tx_status() but can be called in IRQ context
  * (internally defers to a tasklet.)
  *
- * Calls to this function and ieee80211_tx_status() may not be mixed for a
- * single hardware.
+ * Calls to this function, ieee80211_tx_status() and
+ * ieee80211_tx_status_ni() may not be mixed for a single hardware.
  *
  * @hw: the hardware the frame was transmitted by
  * @skb: the frame that was transmitted, owned by mac80211 after this call
-- 
1.7.2.3

Re: [PATCH v4] mac80211/rt2x00: add ieee80211_tx_status_ni()

[Johannes Berg]

On Tue, 2010-11-30 at 16:49 +0100, Johannes Stezenbach wrote:

>   * This function may not be called in IRQ context. Calls to this function
>   * for a single hardware must be synchronized against each other. Calls
> - * to this function and ieee80211_tx_status_irqsafe() may not be mixed
> - * for a single hardware.
> + * to this function, ieee80211_tx_status_ni() and ieee80211_tx_status_irqsafe()
> + * may not be mixed for a single hardware.

I'm ok with this, although technically you can mix
ieee80211_tx_status_ni() and ieee80211_tx_status(), just not either or
both of them with _irqsafe().

johannes

Re: [PATCH v4] mac80211/rt2x00: add ieee80211_tx_status_ni()

[Johannes Stezenbach]

On Tue, Nov 30, 2010 at 04:53:00PM +0100, Johannes Berg wrote:
> On Tue, 2010-11-30 at 16:49 +0100, Johannes Stezenbach wrote:
> 
> >   * This function may not be called in IRQ context. Calls to this function
> >   * for a single hardware must be synchronized against each other. Calls
> > - * to this function and ieee80211_tx_status_irqsafe() may not be mixed
> > - * for a single hardware.
> > + * to this function, ieee80211_tx_status_ni() and ieee80211_tx_status_irqsafe()
> > + * may not be mixed for a single hardware.
> 
> I'm ok with this, although technically you can mix
> ieee80211_tx_status_ni() and ieee80211_tx_status(), just not either or
> both of them with _irqsafe().

I copied these from ieee80211_rx() etc. since I wasn't sure if there's
not some subtlety I didn't get.  One can mix ieee80211_rx() and
ieee80211_rx_ni(), too, right?


Thanks
Johannes

Re: [PATCH v4] mac80211/rt2x00: add ieee80211_tx_status_ni()

[Johannes Berg]

On Tue, 2010-11-30 at 17:34 +0100, Johannes Stezenbach wrote:
> On Tue, Nov 30, 2010 at 04:53:00PM +0100, Johannes Berg wrote:
> > On Tue, 2010-11-30 at 16:49 +0100, Johannes Stezenbach wrote:
> > 
> > >   * This function may not be called in IRQ context. Calls to this function
> > >   * for a single hardware must be synchronized against each other. Calls
> > > - * to this function and ieee80211_tx_status_irqsafe() may not be mixed
> > > - * for a single hardware.
> > > + * to this function, ieee80211_tx_status_ni() and ieee80211_tx_status_irqsafe()
> > > + * may not be mixed for a single hardware.
> > 
> > I'm ok with this, although technically you can mix
> > ieee80211_tx_status_ni() and ieee80211_tx_status(), just not either or
> > both of them with _irqsafe().
> 
> I copied these from ieee80211_rx() etc. since I wasn't sure if there's
> not some subtlety I didn't get.  One can mix ieee80211_rx() and
> ieee80211_rx_ni(), too, right?

Yes. Note that all this doesn't make any sense though, since you must
never call them concurrently. And it's hard to imagine a situation where
a single driver calls _ni and non-_ni versions, while making sure they
can't both happen at the same time ...

johannes

Re: [PATCH v4] mac80211/rt2x00: add ieee80211_tx_status_ni()

[John W. Linville]

On Tue, Nov 30, 2010 at 05:35:58PM +0100, Johannes Berg wrote:
> On Tue, 2010-11-30 at 17:34 +0100, Johannes Stezenbach wrote:
> > On Tue, Nov 30, 2010 at 04:53:00PM +0100, Johannes Berg wrote:
> > > On Tue, 2010-11-30 at 16:49 +0100, Johannes Stezenbach wrote:
> > > 
> > > >   * This function may not be called in IRQ context. Calls to this function
> > > >   * for a single hardware must be synchronized against each other. Calls
> > > > - * to this function and ieee80211_tx_status_irqsafe() may not be mixed
> > > > - * for a single hardware.
> > > > + * to this function, ieee80211_tx_status_ni() and ieee80211_tx_status_irqsafe()
> > > > + * may not be mixed for a single hardware.
> > > 
> > > I'm ok with this, although technically you can mix
> > > ieee80211_tx_status_ni() and ieee80211_tx_status(), just not either or
> > > both of them with _irqsafe().
> > 
> > I copied these from ieee80211_rx() etc. since I wasn't sure if there's
> > not some subtlety I didn't get.  One can mix ieee80211_rx() and
> > ieee80211_rx_ni(), too, right?
> 
> Yes. Note that all this doesn't make any sense though, since you must
> never call them concurrently. And it's hard to imagine a situation where
> a single driver calls _ni and non-_ni versions, while making sure they
> can't both happen at the same time ...

Maybe someone can clarify the wording of the comments and send a follow-on patch?

Thanks,

John
-- 
John W. Linville		Someday the world will need a hero, and you
linville@tuxdriver.com			might be all we have.  Be ready.