Custom hardware Qdisc...

Custom hardware Qdisc...

[David Daney]

Our hardware (Octeon SOC) has fairly flexible packet output queuing that 
can be done completely in hardware.

I am thinking of trying to take advantage of this in the driver 
(currently in drivers/staging/octeon).

At a very hige level my tentative plan is to implement a custom  Qdisc 
that has an enqueue method that places the packets directly in the 
output queues after they are classified.

The dequeue method would be a noop returning NULL because once the 
packets are on the queues, no further action is needed from the driver.

Does doing something like this seem plausible?  If not, what are the 
main problems I might be overlooking?

Thanks in advance,
David Daney

Re: Custom hardware Qdisc...

[David Miller]

From: David Daney <ddaney@caviumnetworks.com>
Date: Fri, 11 Dec 2009 10:26:08 -0800

> Our hardware (Octeon SOC) has fairly flexible packet output queuing
> that can be done completely in hardware.

You haven't described sufficiently what your hardware is
capable of.

Generally, implementing qdiscs specifically to support
hardware features is not how things are handled.

But really we need to know a lot more about the details of your
hardware to make any kind of real suggestions.

Re: Custom hardware Qdisc...

[David Daney]

David Miller wrote:
> From: David Daney <ddaney@caviumnetworks.com>
> Date: Fri, 11 Dec 2009 10:26:08 -0800
> 
>> Our hardware (Octeon SOC) has fairly flexible packet output queuing
>> that can be done completely in hardware.
> 
> You haven't described sufficiently what your hardware is
> capable of.
> 

Up to 16 output queues per port. With (almost) any combination of 
absolute and weighted priorities between the various queues.


> Generally, implementing qdiscs specifically to support
> hardware features is not how things are handled.
> 

I think I could emulate the default pfifo Qdisk fairly well in hardware. 
  By making it a Qdisc, the user would still be able to switch to one of 
the software implementations if desired at runtime.


> But really we need to know a lot more about the details of your
> hardware to make any kind of real suggestions.
> 

Well see above for the overview.  I just wanted to get an idea if such 
an approach would be highly frowned upon before coding anything up.

We see significant cacheline bouncing in the Qdisc code and can increase 
  packet forwarding rates by 10%-15% by setting the txqueuelen to zero 
to disable the queuing.  I would like to see if the same thing can be 
achieved without throwing away the features of the Qdisc.

David Daney

Re: Custom hardware Qdisc...

[David Miller]

From: David Daney <ddaney@caviumnetworks.com>
Date: Fri, 11 Dec 2009 16:58:50 -0800

> David Miller wrote:
>> From: David Daney <ddaney@caviumnetworks.com>
>> Date: Fri, 11 Dec 2009 10:26:08 -0800
>> 
>>> Our hardware (Octeon SOC) has fairly flexible packet output queuing
>>> that can be done completely in hardware.
>> You haven't described sufficiently what your hardware is
>> capable of.
>> 
> 
> Up to 16 output queues per port. With (almost) any combination of
> absolute and weighted priorities between the various queues.

It sounds like these queues could be configured to have equal
priorities and thus be used as purely a TX multiqueue scheme
for better scalability.

I suspect that doing that alone will fix the qdisc cache line bouncing
you mention.