Re: netif_stop_queue() and multiple hardware queues

Re: netif_stop_queue() and multiple hardware queues

[Jeremy Jackson]

I'd like to raise the question again.  How can the network stack's flow 
control function with hardware that has multiple physical queues?  How 
can these queues be exposed to tc?

After scanning the netdev mailing list activity for the last 6 months, I 
can see no furthur discussion since my original post.  I did notice some 
churn in several 802.11 layer stacks trying to deal with multiple 
hardware queues, the IPW2200 in particular.

I propose that this is an architecture/framework/API issue that the 
Linux core networking code should be dealing with.

There is some fragmentation already; compare the QoS *policy* 
implemented by Madwifi vs IPW2200 vs devicescape's 80211 stack. It's 
worth noting that at the 80211 layer, the first 2 listed above have a 
"priority" argument to (80211)_hard_start_xmit().

It would seem ideal to expose the queues of Ethernet layer hardware 
(provide mechanism) and let userspace tc tools do the classifying 
(sysadmin can set the policy).

Jeremy

Jeremy Jackson wrote:
> Hi,
> 
> I posted this briefly on linux-net, before being redirected here.
> 
> Two pieces of hardware now have Linux driver support for multiple 
> hardware queues: Intel's e1000 (two queues from what I can see in the 
> code) and Atheros's 5212 and up, in support of 802.11e and WME (four 
> hardware queues).  In the GigE case, this just reduces latency due to 
> hardware queueing.  In the WiFi case, the queues actually have 
> significance in access to the shared medium. (ACKs can be disabled as 
> well)  It is also worthy of note that ADSL2, VDSL, and ADSL2+ have 4 
> different latency "queues".  These last two are significant; real-time 
> applications suffer the most from low speed, shared, and/or 
> non-deterministic media.  I wonder where DOCSIS 2 is in this regard.  
> Anyone?  Beuler?
> 
> So my question is, what's it going to take to get dev->hard_start_xmit() 
> to hook up tc queues directly to hardware/driver queues?
> 
> Right now, it seems no matter how elaborate a tc setup you have, 
> everything funnels through one queue, where the only thing that survives 
> from the classifying/queueing is skb-> priority (ie nothing).  The 
> hardware driver can then try to reclassify packets.   I suppose you 
> could hack up an iptables classifyer to set skb->priority...
> 
> The  Atheros driver tries to  do it's own classifying by first wiping 
> out skb->priority, then hard-coding  a mapping (tsk - policy is for the 
> sysadmin!) between VLAN tag priority, IP TOS/DSCP, and skb->priority, 
> then to one of the 4 queues and ACK states, blithely ignoring any fine 
> work done by tc.
> 
> It's be sweet to head this nonsense off at the pass, before others 
> discover the rabbit trail and make it into a trade route.
> 

Re: netif_stop_queue() and multiple hardware queues

[David Miller]

From: Jeremy Jackson <jerj@coplanar.net>
Date: Sun, 02 Jul 2006 22:36:03 -0400

> I propose that this is an architecture/framework/API issue that the 
> Linux core networking code should be dealing with.

Proposals are worthless if nobody wants to do the work to actually
implement the proposal.

Your tone seems to suggest that you feel if you propose something,
someone will just jump out of their seat and implement your proposal.

That's not how this works :)

Re: netif_stop_queue() and multiple hardware queues

[Jeremy Jackson]

David Miller wrote:
> From: Jeremy Jackson <jerj@coplanar.net>
> Date: Sun, 02 Jul 2006 22:36:03 -0400
> 
>> I propose that this is an architecture/framework/API issue that the 
>> Linux core networking code should be dealing with.
> 
> Proposals are worthless if nobody wants to do the work to actually
> implement the proposal.
> 
> Your tone seems to suggest that you feel if you propose something,
> someone will just jump out of their seat and implement your proposal.
> 
> That's not how this works :)

I apologize for the tone.  I *am* interested working on this.  I'm just
surprised nobody has jumped out of their seat to do it :)

I have changed a line or two of kernel code but am by no means an
expert, so I guess I was hoping for some kind of feedback about the
proposal.  I guess submit a patch for discussion?  If it's gross I guess
that would generate some feedback.

What's the best Git tree to base this on?  (maybe it's name means make
svn and and cvs users feel like a stupid git?)

In the current 2.6 network driver interface, the TX path seems to be
based on mydev->hard_start_xmit().  The device driver initiates flow
control with netif_stop_queue().

netif_stop_queue() and other functions (macros in netdevice.h) touch
mydev->state, the bit __LINK_STATE_XOFF:

netif_schedule()
netif_start_queue()
netif_wake_queue()
netif_stop_queue()
netif_queue_stopped()

There is only one queue now, implied by default.  Adding a "hwqueue"
parameter to all of the above would be a start.

It would be ideal if mydev->xmit_lock could be separated out, one for
each hwqueue.  A quick hack to implement this would be for the device to
set the mydev->features NETIF_F_LLTX, and implement per-queue locking.

Teaching qdiscs to use the multiple queues looks tricky in comparison, a 
quick hack might be to have prio's bands map 1:1 with the hardware queues.

Regards,

Jeremy