Re: [PATCH] PATCH 1/2 [SCHED 2.6.24]: Check subqueue status before calling hard_start_xmit

Re: [PATCH] PATCH 1/2 [SCHED 2.6.24]: Check subqueue status before calling hard_start_xmit

[Krishna Kumar]

Hi Peter,

Peter wrote on 11/13/2007 11:14:50 PM:
> @@ -134,7 +134,7 @@ static inline int qdisc_restart(struct net_device *dev)
>  {
>     struct Qdisc *q = dev->qdisc;
>     struct sk_buff *skb;
> -   int ret;
> +   int ret = NETDEV_TX_BUSY;
>  
>     /* Dequeue packet */
>     if (unlikely((skb = dev_dequeue_skb(dev, q)) == NULL))
> @@ -145,7 +145,8 @@ static inline int qdisc_restart(struct net_device *dev)
>     spin_unlock(&dev->queue_lock);
>  
>     HARD_TX_LOCK(dev, smp_processor_id());
> -   ret = dev_hard_start_xmit(skb, dev);
> +   if (!netif_subqueue_stopped(dev, skb))
> +      ret = dev_hard_start_xmit(skb, dev);
>     HARD_TX_UNLOCK(dev);

You could optimize this by getting HARD_TX_LOCK after the check. I
assume that netif_stop_subqueue (from another CPU) would always be
called by the driver xmit, and that is not possible since we hold
the __LINK_STATE_QDISC_RUNNING bit. Does that sound correct?

PATCH
------

diff -ruNp 1/net/sched/sch_generic.c 2/net/sched/sch_generic.c
--- 1/net/sched/sch_generic.c	2007-11-14 11:14:10.000000000 +0530
+++ 2/net/sched/sch_generic.c	2007-11-14 11:18:27.000000000 +0530
@@ -144,10 +144,11 @@ static inline int qdisc_restart(struct n
 	/* And release queue */
 	spin_unlock(&dev->queue_lock);
 
-	HARD_TX_LOCK(dev, smp_processor_id());
 	if (!netif_subqueue_stopped(dev, skb))
+		HARD_TX_LOCK(dev, smp_processor_id());
 		ret = dev_hard_start_xmit(skb, dev);
-	HARD_TX_UNLOCK(dev);
+		HARD_TX_UNLOCK(dev);
+	}
 
 	spin_lock(&dev->queue_lock);
 	q = dev->qdisc;

Thanks,

- KK

Re: [PATCH] PATCH 1/2 [SCHED 2.6.24]: Check subqueue status before calling hard_start_xmit

[David Miller]

From: Krishna Kumar <krkumar2@in.ibm.com>
Date: Wed, 14 Nov 2007 11:34:12 +0530

> Hi Peter,
> 
> Peter wrote on 11/13/2007 11:14:50 PM:
> > @@ -134,7 +134,7 @@ static inline int qdisc_restart(struct net_device *dev)
> >  {
> >     struct Qdisc *q = dev->qdisc;
> >     struct sk_buff *skb;
> > -   int ret;
> > +   int ret = NETDEV_TX_BUSY;
> >  
> >     /* Dequeue packet */
> >     if (unlikely((skb = dev_dequeue_skb(dev, q)) == NULL))
> > @@ -145,7 +145,8 @@ static inline int qdisc_restart(struct net_device *dev)
> >     spin_unlock(&dev->queue_lock);
> >  
> >     HARD_TX_LOCK(dev, smp_processor_id());
> > -   ret = dev_hard_start_xmit(skb, dev);
> > +   if (!netif_subqueue_stopped(dev, skb))
> > +      ret = dev_hard_start_xmit(skb, dev);
> >     HARD_TX_UNLOCK(dev);
> 
> You could optimize this by getting HARD_TX_LOCK after the check. I
> assume that netif_stop_subqueue (from another CPU) would always be
> called by the driver xmit, and that is not possible since we hold
> the __LINK_STATE_QDISC_RUNNING bit. Does that sound correct?

I don't think this is a critical optimization at this time,
but something to certainly do along with the surgery
we'll undoubtedly be doing here in the future :-)

Re: [PATCH] PATCH 1/2 [SCHED 2.6.24]: Check subqueue status before calling hard_start_xmit

[Krishna Kumar2]

Hi Dave,

David Miller <davem@davemloft.net> wrote on 11/14/2007 11:44:39 AM:

> > You could optimize this by getting HARD_TX_LOCK after the check. I
> > assume that netif_stop_subqueue (from another CPU) would always be
> > called by the driver xmit, and that is not possible since we hold
> > the __LINK_STATE_QDISC_RUNNING bit. Does that sound correct?
>
> I don't think this is a critical optimization at this time,
> but something to certainly do along with the surgery
> we'll undoubtedly be doing here in the future :-)

Agree. Will keep in mind for later :)

Thanks,

- KK

Re: [PATCH] PATCH 1/2 [SCHED 2.6.24]: Check subqueue status before calling hard_start_xmit

[Krishna Kumar2]

> > You could optimize this by getting HARD_TX_LOCK after the check. I
> > assume that netif_stop_subqueue (from another CPU) would always be
> > called by the driver xmit, and that is not possible since we hold
> > the __LINK_STATE_QDISC_RUNNING bit. Does that sound correct?
>
> I don't think this is a critical optimization at this time,
> but something to certainly do along with the surgery
> we'll undoubtedly be doing here in the future :-)

Apologies for the spam...

I guess this check can be completely removed once other qdiscs are modified
to
support subqueue. Essentially sch_prio and sch_rr executes subqueue_stopped
twice for every skb while this check is required only for other qdiscs (and
really
belongs in the individual qdisc dequeue handlers).

Thanks,

- KK

RE: [PATCH] PATCH 1/2 [SCHED 2.6.24]: Check subqueue status before calling hard_start_xmit

[Waskiewicz Jr, Peter P]

> You could optimize this by getting HARD_TX_LOCK after the 
> check. I assume that netif_stop_subqueue (from another CPU) 
> would always be called by the driver xmit, and that is not 
> possible since we hold the __LINK_STATE_QDISC_RUNNING bit. 
> Does that sound correct?

Sorry for not responding sooner; Dave hit it on the head though with his
response.  I agree with your changes, and I'll incorporate them in the
lockless stack patches I've been working on (in the software queuing
mode).  Now if I could just find some time to finish them up and get
them out for review...

Thanks,

-PJ Waskiewicz

[PATCH] SCHED: Fix unnecesary driver entries when queue is stopped

[PJ Waskiewicz]

Dave,

This patch addresses an issue with multiqueue devices and non-multiqueue
qdiscs which is causing performance issues.  This patch should be 
considered for both 2.6.23-stable and 2.6.24 upstream.  Basically, if
a driver is using the netif_*_subqueue() calls, then qdisc_restart() will
happily call hard_start_xmit() even if subqueue 0 is stopped, which is bad.
This re-adds the check for the subqueue state.

Note that this check was removed when qdisc_restart() was rewritten.  At that
time though, we didn't understand the full effect of multiqueue with respect
to the qdiscs and queue management from a driver to kernel perspective.  Since
the driver doesn't know what qdisc capabilities live above it, it needs to
decide to use the queue or subqueue functions ahead of time.  This patch is
just cleaning up a miss from that rewrite.

Patch 1 is for 2.6.24, patch 2 is for 2.6.23 stable.

Thanks,
-- 
PJ Waskiewicz <peter.p.waskiewicz.jr@intel.com>

[PATCH] PATCH 1/2 [SCHED 2.6.24]: Check subqueue status before calling hard_start_xmit

[PJ Waskiewicz]

The only qdiscs that check subqueue state before dequeue'ing are PRIO
and RR.  The other qdiscs, including the default pfifo_fast qdisc, will
allow traffic bound for subqueue 0 through to hard_start_xmit.  The check
for netif_queue_stopped() is done above in pkt_sched.h, so it is
unnecessary for qdisc_restart().  However, if the underlying driver is
multiqueue capable, and only sets queue states on subqueues, this will
allow packets to enter the driver when it's currently unable to process
packets, resulting in expensive requeues and driver entries.  This patch
re-adds the check for the subqueue status before calling hard_start_xmit,
so we can try and avoid the driver entry when the queues are stopped.

Signed-off-by: Peter P Waskiewicz Jr <peter.p.waskiewicz.jr@intel.com>
---

 net/sched/sch_generic.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
index fa1a6f4..e595e65 100644
--- a/net/sched/sch_generic.c
+++ b/net/sched/sch_generic.c
@@ -134,7 +134,7 @@ static inline int qdisc_restart(struct net_device *dev)
 {
 	struct Qdisc *q = dev->qdisc;
 	struct sk_buff *skb;
-	int ret;
+	int ret = NETDEV_TX_BUSY;
 
 	/* Dequeue packet */
 	if (unlikely((skb = dev_dequeue_skb(dev, q)) == NULL))
@@ -145,7 +145,8 @@ static inline int qdisc_restart(struct net_device *dev)
 	spin_unlock(&dev->queue_lock);
 
 	HARD_TX_LOCK(dev, smp_processor_id());
-	ret = dev_hard_start_xmit(skb, dev);
+	if (!netif_subqueue_stopped(dev, skb))
+		ret = dev_hard_start_xmit(skb, dev);
 	HARD_TX_UNLOCK(dev);
 
 	spin_lock(&dev->queue_lock);

Re: [PATCH] PATCH 1/2 [SCHED 2.6.24]: Check subqueue status before calling hard_start_xmit

[David Miller]

From: PJ Waskiewicz <peter.p.waskiewicz.jr@intel.com>
Date: Tue, 13 Nov 2007 09:44:50 -0800

> The only qdiscs that check subqueue state before dequeue'ing are PRIO
> and RR.  The other qdiscs, including the default pfifo_fast qdisc, will
> allow traffic bound for subqueue 0 through to hard_start_xmit.  The check
> for netif_queue_stopped() is done above in pkt_sched.h, so it is
> unnecessary for qdisc_restart().  However, if the underlying driver is
> multiqueue capable, and only sets queue states on subqueues, this will
> allow packets to enter the driver when it's currently unable to process
> packets, resulting in expensive requeues and driver entries.  This patch
> re-adds the check for the subqueue status before calling hard_start_xmit,
> so we can try and avoid the driver entry when the queues are stopped.
> 
> Signed-off-by: Peter P Waskiewicz Jr <peter.p.waskiewicz.jr@intel.com>

Applied, and I'll queue up the other one for -stable, thanks.