using huge numbers of queues

using huge numbers of queues

[Andrew Grover]

Hi Herbert,

At NetConf, you made a passing remark about wanting lots of queues,
even 1-per-socket. Have you thought further about how we would use so
many?

Thinking about this reminded me of VJ's 2006 netchannel concept, which
although not adopted, was pretty interesting. Would having 1 queue per
socket (or at least 1 per process) and hw that is able to filter
individual flows (I think the Intel 82599 can do this now for up to
128) perhaps make netchannels workable? At least this would get all
processing out of int/bh and into process context, if not userspace,
no?

Regards -- Andy

Re: using huge numbers of queues

[David Miller]

From: Andrew Grover <andy.grover@gmail.com>
Date: Wed, 7 Oct 2009 14:59:49 -0700

> At NetConf, you made a passing remark about wanting lots of queues,
> even 1-per-socket. Have you thought further about how we would use so
> many?

Classification.

Lots and lots of virtual queues, which map to a smaller number
of physical queues for delivery.

The virtual queue matched serves as a index and a classification
hint to things like GRO receive, etc.

Re: using huge numbers of queues

[Ben Hutchings]

On Wed, 2009-10-07 at 14:59 -0700, Andrew Grover wrote:
> Hi Herbert,
> 
> At NetConf, you made a passing remark about wanting lots of queues,
> even 1-per-socket. Have you thought further about how we would use so
> many?
> 
> Thinking about this reminded me of VJ's 2006 netchannel concept, which
> although not adopted, was pretty interesting. Would having 1 queue per
> socket (or at least 1 per process) and hw that is able to filter
> individual flows (I think the Intel 82599 can do this now for up to
> 128) perhaps make netchannels workable? At least this would get all
> processing out of int/bh and into process context, if not userspace,
> no?

Solarflare controllers already support 1000+ queues.  The OpenOnload
software <http://www.openonload.org> assigns one queue per process
(though this isn't quite 1:1 as sockets may be shared between processes)
and almost all protocol processing is done in user-space.  I'm not sure
quite how close this is to the netchannels concept.

Ben.

-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: using huge numbers of queues

[Mark Smith]

On Wed, 07 Oct 2009 15:38:11 -0700 (PDT)
David Miller <davem@davemloft.net> wrote:

> From: Andrew Grover <andy.grover@gmail.com>
> Date: Wed, 7 Oct 2009 14:59:49 -0700
> 
> > At NetConf, you made a passing remark about wanting lots of queues,
> > even 1-per-socket. Have you thought further about how we would use so
> > many?
> 
> Classification.
> 
> Lots and lots of virtual queues, which map to a smaller number
> of physical queues for delivery.
> 
> The virtual queue matched serves as a index and a classification
> hint to things like GRO receive, etc.

Is that similar to what is described in 

Trading Packet Headers for Packet Processing
http://www.sigcomm.org/sigcomm95/papers/chandranmenon.ps

?

My understanding of that paper is that when a packet enters the host it
is classified using various attributes e.g. Ip src/dest/etc, and then
assigned an unique identifier. Subsequent processing of the packet is
indexed by this identifier, rather than each processing stage
performing it's own packet classification and selection.

For packets that are forwarded, the packet is then tagged/labled with
an ID, so that subsequent hosts don't have to perform classification
either.

(This paper is pretty much the origins of MPLS, which is where my
original interest came from)


> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: using huge numbers of queues

[Herbert Xu]

On Wed, Oct 07, 2009 at 02:59:49PM -0700, Andrew Grover wrote:
> 
> Thinking about this reminded me of VJ's 2006 netchannel concept, which
> although not adopted, was pretty interesting. Would having 1 queue per
> socket (or at least 1 per process) and hw that is able to filter
> individual flows (I think the Intel 82599 can do this now for up to
> 128) perhaps make netchannels workable? At least this would get all
> processing out of int/bh and into process context, if not userspace,
> no?

This is exactly right.  Having one socket per queue would allow
the netchannel concept to become reality without compromising
features such as netfilter.

Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt