[PATCH] NET: Multiple queue hardware support

[PATCH] NET: Multiple queue hardware support

[PJ Waskiewicz]

Please consider these patches for 2.6.23 inclusion.

This patchset is an updated version of previous multiqueue network device
support patches.  The general approach of introducing a new API for multiqueue
network devices to register with the stack has remained.  The changes include
adding a round-robin qdisc, heavily based on sch_prio, which will allow
queueing to hardware with no OS-enforced queuing policy.  sch_prio still has
the multiqueue code in it, but has a Kconfig option to compile it out of the
qdisc.  This allows people with hardware containing scheduling policies to
use sch_rr (round-robin), and others without scheduling policies in hardware
to continue using sch_prio if they wish to have some notion of scheduling
priority.

The patches being sent are split into Documentation, Qdisc changes, and
core stack changes.  The requested e1000 changes are still being resolved,
and will be sent at a later date.

I did not modify other users of netif_queue_stopped() in net/core/netpoll.c,
net/core/dev.c, or net/core/pktgen.c, since no classification occurs for
the skb being sent to the device.  Therefore, packets should always be
ending up in queue 0, so there's no need to check the subqueue status either.

The patches to iproute2 for tc will be sent separately, to support sch_rr.

-- 
PJ Waskiewicz <peter.p.waskiewicz.jr@intel.com>

RE: [PATCH] NET: Multiple queue hardware support

[Jan-Bernd Themann]

Hi,

to me it seems that this patch set only include multiple transmit queue support
(for qdisc). Am I right with this observation? If so, are there also plans to 
support multiple receive queues to allow the queues to be processed in parallel
on different CPUs via a standard interface? Currently, some drivers use 
"fake netdevices" to feed netif_rx_schedule().

Thanks,
Jan-Bernd

RE: [PATCH] NET: Multiple queue hardware support

[Waskiewicz Jr, Peter P]

> to me it seems that this patch set only include multiple 
> transmit queue support (for qdisc). Am I right with this 
> observation? If so, are there also plans to support multiple 
> receive queues to allow the queues to be processed in 
> parallel on different CPUs via a standard interface? 
> Currently, some drivers use "fake netdevices" to feed 
> netif_rx_schedule().
> 
> Thanks,
> Jan-Bernd

Jan,
	Yes, these patches are for transmit multiqueue hardware support
only.  There are other efforts to stop using the fake netdevices for Rx
polling, such as using MSI-X vectors assigned to particular receive
queues, but this is still done in the driver (which is ok).  I'm
currently not working on anything in the kernel to support multiqueue Rx
for non MSI-X devices though.

Thanks,
-PJ Waskiewicz

Re: [PATCH] NET: Multiple queue hardware support

[David Miller]

From: Jan-Bernd Themann <ossthema@de.ibm.com>
Date: Wed, 20 Jun 2007 16:58:43 +0200

> to me it seems that this patch set only include multiple transmit
> queue support (for qdisc). Am I right with this observation? If so,
> are there also plans to support multiple receive queues to allow the
> queues to be processed in parallel on different CPUs via a standard
> interface? Currently, some drivers use "fake netdevices" to feed
> netif_rx_schedule().

Yes.

See the "struct net_poll" patches that went out several months
ago, that will help things out in that area.

[PATCH] NET: Multiple queue hardware support

[PJ Waskiewicz]

Please consider these patches for 2.6.23 inclusion.

Updates since the last submission:

1. skb->queue_mapping moved into the iff cacheline.  I looked at moving
   iff and queue_mapping, but there wasn't enough room anywhere else to
   logically group these in a different cacheline that I could see.  Thanks
   Patrick McHardy.

2. netdev->egress_subqueue is now indexed thanks to Dave Miller.

3. sch_rr is now a MODULE_ALIAS of sch_prio.  Thanks Patrick McHardy.

4. Both sch_rr and multiqueue sch_prio expect the number of bands to
   equal the number of queues on the netdev.

This patchset is an updated version of previous multiqueue network device
support patches.  The general approach of introducing a new API for multiqueue
network devices to register with the stack has remained.  The changes include
adding a round-robin qdisc, heavily based on sch_prio, which will allow
queueing to hardware with no OS-enforced queuing policy.  sch_prio still has
the multiqueue code in it, but has a Kconfig option to compile it out of the
qdisc.  This allows people with hardware containing scheduling policies to
use sch_rr (round-robin), and others without scheduling policies in hardware
to continue using sch_prio if they wish to have some notion of scheduling
priority.

The patches being sent are split into Documentation, Qdisc changes, and
core stack changes.  The requested e1000 changes are still being resolved,
and will be sent at a later date.

I did not modify other users of netif_queue_stopped() in net/core/netpoll.c,
net/core/dev.c, or net/core/pktgen.c, since no classification occurs for
the skb being sent to the device.  Therefore, packets should always be
ending up in queue 0, so there's no need to check the subqueue status either.

The patches to iproute2 for tc will be sent separately, to support sch_rr.

-- 
PJ Waskiewicz <peter.p.waskiewicz.jr@intel.com>

Re: [PATCH] NET: Multiple queue hardware support

[Patrick McHardy]

PJ Waskiewicz wrote:
> I did not modify other users of netif_queue_stopped() in net/core/netpoll.c,
> net/core/dev.c, or net/core/pktgen.c, since no classification occurs for
> the skb being sent to the device.  Therefore, packets should always be
> ending up in queue 0, so there's no need to check the subqueue status either.
>   

Thats not correct. Subqueue 0 may be full and the queue still running.

I'll look over the patches later.

RE: [PATCH] NET: Multiple queue hardware support

[Waskiewicz Jr, Peter P]

> PJ Waskiewicz wrote:
> > I did not modify other users of netif_queue_stopped() in 
> > net/core/netpoll.c, net/core/dev.c, or net/core/pktgen.c, since no 
> > classification occurs for the skb being sent to the device.  
> > Therefore, packets should always be ending up in queue 0, 
> so there's no need to check the subqueue status either.
> >   
> 
> Thats not correct. Subqueue 0 may be full and the queue still running.
> 
> I'll look over the patches later.

I'm working something up to address this.  The last time I thought about
this, I had issues with software devices, such as loopback.  They
weren't allocating any subqueues at all, so they would call
netif_subqueue_stopped() and panic the kernel.  However, now with Dave's
request to index egress_subqueue, the first queue is allocated for
everyone, so loopback and other software devices should be happy.  Let
me put these checks back in, test it out, and resend if I don't see any
issues.

Sorry for the thrash,
-PJ Waskiewicz

[PATCH] NET: Multiple queue hardware support

[PJ Waskiewicz]

Please consider these patches for 2.6.23 inclusion.

These patches are built against Patrick McHardy's recently submitted
RTNETLINK nested compat attribute patches.  They're needed to preserve
ABI between sch_{rr|prio} and iproute2.

Updates since the last submission:

1. Added checks for netif_subqueue_stopped() to net/core/netpoll.c,
   net/core/pktgen.c, and to software device hard_start_xmit in
   dev_queue_xmit().

2. Removed TCA_PRIO_TEST and added TCA_PRIO_MQ for sch_prio and sch_rr.

3. Fixed dependancy issues in net/sched/Kconfig with NET_SCH_RR.

4. Implemented the new nested compat attribute API for MQ in NET_SCH_PRIO
   and NET_SCH_RR.

5. Allow sch_rr and sch_prio to turn multiqueue hardware support on and off
   at loadtime.

This patchset is an updated version of previous multiqueue network device
support patches.  The general approach of introducing a new API for multiqueue
network devices to register with the stack has remained.  The changes include
adding a round-robin qdisc, heavily based on sch_prio, which will allow
queueing to hardware with no OS-enforced queuing policy.  sch_prio still has
the multiqueue code in it, but has a Kconfig option to compile it out of the
qdisc.  This allows people with hardware containing scheduling policies to
use sch_rr (round-robin), and others without scheduling policies in hardware
to continue using sch_prio if they wish to have some notion of scheduling
priority.

The patches being sent are split into Documentation, Qdisc changes, and
core stack changes.  The requested e1000 changes are still being resolved,
and will be sent at a later date.

The patches to iproute2 for tc will be sent separately, to support sch_rr.

-- 
PJ Waskiewicz <peter.p.waskiewicz.jr@intel.com>

[PATCH] NET: Multiple queue hardware support

[PJ Waskiewicz]

Please consider these patches for 2.6.23 inclusion.

Updates since the last submission:

1. Fixed alloc_netdev_mq() queue_count bug.

2. Fixed the TCA_PRIO_MQ options layout.

3. Protected sch_prio and sch_rr multiqueue code with NET_SCH_MULTIQUEUE.

4. Added RTA_{GET|PUT}_FLAG in place of RTA_DATA for passing multiqueue
   options to and from the qdisc.

5. Allow sch_prio and sch_rr to take 0 bands when in multiqueue mode.  This
   will set q->bands to dev->egress_subqueue_count; added this also to the
   kernel doc.

This patchset is an updated version of previous multiqueue network device
support patches.  The general approach of introducing a new API for multiqueue
network devices to register with the stack has remained.  The changes include
adding a round-robin qdisc, heavily based on sch_prio, which will allow
queueing to hardware with no OS-enforced queuing policy.  sch_prio still has
the multiqueue code in it, but has a Kconfig option to compile it out of the
qdisc.  This allows people with hardware containing scheduling policies to
use sch_rr (round-robin), and others without scheduling policies in hardware
to continue using sch_prio if they wish to have some notion of scheduling
priority.

The patches being sent are split into Documentation, Qdisc changes, and
core stack changes.

The patches to iproute2 for tc will be sent separately, to support sch_rr.

-- 
PJ Waskiewicz <peter.p.waskiewicz.jr@intel.com>