multiqueue interrupts...

multiqueue interrupts...

[David Miller]

During kernel summit I was speaking with Arjan van de Ven
about irqbalanced and networking card multiqueue interrupts.

In order for irqbalanaced to make smart decisions, what needs to
happen in drivers is that the individual interrupts need to be
named in such a way that he can tell by looking at /proc/interrupts
output that these interrupts are related.

I would suggest that people use something like:

	char buf[IFNAMSIZ+6];

	sprintf(buf, "%s-%s-%d",
	        netdev->name,
		(RX_INTERRUPT ? "rx" : "tx"),
		queue->index);

So on a multiqueue card with 2 RX queues and 2 TX queues we'd
have names like:

	eth0-rx-0
	eth0-rx-1
	eth0-tx-0
	eth0-tx-1

So let's make an effort to get this done right in 2.6.28 and meanwhile
Arjan can add the irqbalanced code.

Re: multiqueue interrupts...

[Ben Hutchings]

On Thu, 2008-09-18 at 19:38 -0700, David Miller wrote:
[...]
> So on a multiqueue card with 2 RX queues and 2 TX queues we'd
> have names like:
> 
> 	eth0-rx-0
> 	eth0-rx-1
> 	eth0-tx-0
> 	eth0-tx-1
> 
> So let's make an effort to get this done right in 2.6.28 and meanwhile
> Arjan can add the irqbalanced code.

What about the case where an interrupt is shared between RX and TX
completions?  Our hardware is very flexible in this regard, but based on
performance testing prior to the introduction of TX multiqueue we
currently allocate multiple interrupts for RX completions and share the
first with TX completions.

Ben.

-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: multiqueue interrupts...

[Brice Goglin]

Ben Hutchings wrote:
> On Thu, 2008-09-18 at 19:38 -0700, David Miller wrote:
> [...]
>   
>> So on a multiqueue card with 2 RX queues and 2 TX queues we'd
>> have names like:
>>
>> 	eth0-rx-0
>> 	eth0-rx-1
>> 	eth0-tx-0
>> 	eth0-tx-1
>>
>> So let's make an effort to get this done right in 2.6.28 and meanwhile
>> Arjan can add the irqbalanced code.
>>     
>
> What about the case where an interrupt is shared between RX and TX
> completions?  Our hardware is very flexible in this regard, but based on
> performance testing prior to the introduction of TX multiqueue we
> currently allocate multiple interrupts for RX completions and share the
> first with TX completions.
>   

myri10ge uses the same interrupts for TX and RX. The current name is
eth%d:slice-%d but we could change it if there's a consensus.

Brice

Re: multiqueue interrupts...

[Matthew Wilcox]

On Thu, Sep 18, 2008 at 7:38 PM, David Miller wrote:
> During kernel summit I was speaking with Arjan van de Ven
> about irqbalanced and networking card multiqueue interrupts.
> 
> In order for irqbalanaced to make smart decisions, what needs to
> happen in drivers is that the individual interrupts need to be
> named in such a way that he can tell by looking at /proc/interrupts
> output that these interrupts are related.
> 
> So on a multiqueue card with 2 RX queues and 2 TX queues we'd
> have names like:
> 
>        eth0-rx-0
>        eth0-rx-1
>        eth0-tx-0
>        eth0-tx-1
> 
> So let's make an effort to get this done right in 2.6.28 and meanwhile
> Arjan can add the irqbalanced code.

Instead of having magic names, how about we put something in
/proc/irq/nnn/ that lets us tell which interrupts are connected to which
queues?

Another idea I've been thinking about is a flag to tell irqbalance to
leave stuff alone, and we just set stuff up right the first time.

We were discussing various options around multiqueue at first the scsi
multiqueue BOF and later at the PCI MSI BOF.  There's a general feeling
that drivers should be given some guidance about how many queues they
should be enabling, and the sysadmin needs to be the one telling the
PCI layer, which drivers should then query.  The use cases vary wildly
depending whether you're doing routing or are an end node, whether
you're doing v12n or NUMA or both and on just how many cards and cpus
you have.

In a storage / NUMA configuration we really want to set up one queue per
cpu / package / node (depending on resource constraints) and know that
the interrupt is going to come back to the same cpu / package / node.
We definitely don't want irqbalanced moving the interrupt around.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: multiqueue interrupts...

[David Miller]

From: Ben Hutchings <bhutchings@solarflare.com>
Date: Fri, 19 Sep 2008 12:38:01 +0100

> On Thu, 2008-09-18 at 19:38 -0700, David Miller wrote:
> [...]
> > So on a multiqueue card with 2 RX queues and 2 TX queues we'd
> > have names like:
> > 
> > 	eth0-rx-0
> > 	eth0-rx-1
> > 	eth0-tx-0
> > 	eth0-tx-1
> > 
> > So let's make an effort to get this done right in 2.6.28 and meanwhile
> > Arjan can add the irqbalanced code.
> 
> What about the case where an interrupt is shared between RX and TX
> completions?  Our hardware is very flexible in this regard, but based on
> performance testing prior to the introduction of TX multiqueue we
> currently allocate multiple interrupts for RX completions and share the
> first with TX completions.

Probably it would be sufficient to purely use eth0-N, it's just so
that irqbalanced knows that the interrupts are related.

Re: multiqueue interrupts...

[David Miller]

From: Brice Goglin <brice@myri.com>
Date: Fri, 19 Sep 2008 14:29:07 +0200

> Ben Hutchings wrote:
> > On Thu, 2008-09-18 at 19:38 -0700, David Miller wrote:
> > [...]
> >   
> >> So on a multiqueue card with 2 RX queues and 2 TX queues we'd
> >> have names like:
> >>
> >> 	eth0-rx-0
> >> 	eth0-rx-1
> >> 	eth0-tx-0
> >> 	eth0-tx-1
> >>
> >> So let's make an effort to get this done right in 2.6.28 and meanwhile
> >> Arjan can add the irqbalanced code.
> >>     
> >
> > What about the case where an interrupt is shared between RX and TX
> > completions?  Our hardware is very flexible in this regard, but based on
> > performance testing prior to the introduction of TX multiqueue we
> > currently allocate multiple interrupts for RX completions and share the
> > first with TX completions.
> >   
> 
> myri10ge uses the same interrupts for TX and RX. The current name is
> eth%d:slice-%d but we could change it if there's a consensus.

You might be OK as-is.  If we make irqbalanced look for X-%d, you'll
likely be fine.

Re: multiqueue interrupts...

[David Miller]

From: Matthew Wilcox <matthew@wil.cx>
Date: Fri, 19 Sep 2008 12:18:41 -0600

> On Thu, Sep 18, 2008 at 7:38 PM, David Miller wrote:
> > During kernel summit I was speaking with Arjan van de Ven
> > about irqbalanced and networking card multiqueue interrupts.
> > 
> > In order for irqbalanaced to make smart decisions, what needs to
> > happen in drivers is that the individual interrupts need to be
> > named in such a way that he can tell by looking at /proc/interrupts
> > output that these interrupts are related.
> > 
> > So on a multiqueue card with 2 RX queues and 2 TX queues we'd
> > have names like:
> > 
> >        eth0-rx-0
> >        eth0-rx-1
> >        eth0-tx-0
> >        eth0-tx-1
> > 
> > So let's make an effort to get this done right in 2.6.28 and meanwhile
> > Arjan can add the irqbalanced code.
> 
> Instead of having magic names, how about we put something in
> /proc/irq/nnn/ that lets us tell which interrupts are connected to which
> queues?

More code for irqbalanced to write, to look for the files and
directories under there, etc.  Harder to make work with simple
individual driver backports, which you know dists are going to
want to do.

That's why.

No, this new subdirectory thing isn't a good idea.

RE: multiqueue interrupts...

[Brandeburg, Jesse]

Matthew Wilcox wrote:
>> So let's make an effort to get this done right in 2.6.28 and
>> meanwhile Arjan can add the irqbalanced code.
 
> Another idea I've been thinking about is a flag to tell irqbalance to
> leave stuff alone, and we just set stuff up right the first time.

There is already this flag called IRQF_NOBALANCING, at least I think
that's what we want.  irqbalanced's treatment of this flag is another
matter.
 
> We were discussing various options around multiqueue at first the scsi
> multiqueue BOF and later at the PCI MSI BOF.  There's a general
> feeling that drivers should be given some guidance about how many
> queues they should be enabling, and the sysadmin needs to be the one
> telling the PCI layer, which drivers should then query.  The use
> cases vary wildly depending whether you're doing routing or are an
> end node, whether you're doing v12n or NUMA or both and on just how
> many cards and cpus you have.

not a bad, idea, but I can appreciate why DaveM thinks this is
un-necessary.  However all we are left with right now is code changes or
module parameters when trying to configure the number of queues.

How about some new ethtool options having to do with multiqueue
configurations?  Here is a proposal.  I haven't spent much time thinking
about this before but here is an idea.

query multiqueue capabilities:
ethtool -q ethX

set multiqueue capabilities:
ethtool -Q tx N rx N int <fixedcpu|pairs|somethingelse?>

tx N and rx N are pretty self explanitory
int fixedcpu - each queue gets a cpu and is registered IRQF_NOBALANCING
int pairs - tx and rx queues are allocated per cpu and (probably) share
a vector
There should be others here, I'm not sure how/if we would want to make a
pluggable way to do this within ethtool's design without opening buffer
overlow kinds of holes.
 
> In a storage / NUMA configuration we really want to set up one queue
> per cpu / package / node (depending on resource constraints) and know
> that the interrupt is going to come back to the same cpu / package /
> node. We definitely don't want irqbalanced moving the interrupt
> around. 

ethtool doesn't help storage, but I read this on netdev anyway...

Jesse

Re: multiqueue interrupts...

[David Miller]

From: "Brandeburg, Jesse" <jesse.brandeburg@intel.com>
Date: Fri, 19 Sep 2008 13:57:58 -0700

> set multiqueue capabilities:
> ethtool -Q tx N rx N int <fixedcpu|pairs|somethingelse?>

No way.

Let irqbalanced implement the policy.  Just configure as many
queues as possible and let it figure out what to do.

Re: multiqueue interrupts...

[Matthew Wilcox]

On Fri, Sep 19, 2008 at 02:09:22PM -0700, David Miller wrote:
> Let irqbalanced implement the policy.  Just configure as many
> queues as possible and let it figure out what to do.

That isn't a great idea.  Some cards consume a significant amount of
resources per queue and the cost of each queue may be larger than the
benefit.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: multiqueue interrupts...

[Arjan van de Ven]

On Fri, 19 Sep 2008 12:18:41 -0600
Matthew Wilcox <matthew@wil.cx> wrote:
> 
> Another idea I've been thinking about is a flag to tell irqbalance to
> leave stuff alone, and we just set stuff up right the first time.

that's not a good answer. There are reasons for moving interrupts that
are a sysadmin choice (like power management policy that if the system
is seriously idle, that all interrupts go to one of the sockets so that
the others can stay in low power mode). Putting the policy in the kernel
to prohibit such admin choices sounds like a bad idea to me.

There are better ways to do what you want, for example by exposing a
"preferred cpu" somewhere so that irqbalance will place it there
"unless <...>". That is, if such kernel policy binding is right in the
first place

> In a storage / NUMA configuration we really want to set up one queue
> per cpu / package / node (depending on resource constraints) and know
> that the interrupt is going to come back to the same cpu / package /
> node. We definitely don't want irqbalanced moving the interrupt
> around.

irqbalance is NUMA aware and places a penalty on placing an interrupt
"wrongly". We can argue on how strong this penalty should be, but
thinking that irqbalance doesn't use the numa info the kernel exposes
is incorrect.



-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: multiqueue interrupts...

[Andy Fleming]

On Fri, Sep 19, 2008 at 5:11 PM, Arjan van de Ven <arjan@infradead.org> wrote:
> On Fri, 19 Sep 2008 12:18:41 -0600
> Matthew Wilcox <matthew@wil.cx> wrote:

>> In a storage / NUMA configuration we really want to set up one queue
>> per cpu / package / node (depending on resource constraints) and know
>> that the interrupt is going to come back to the same cpu / package /
>> node. We definitely don't want irqbalanced moving the interrupt
>> around.
>
> irqbalance is NUMA aware and places a penalty on placing an interrupt
> "wrongly". We can argue on how strong this penalty should be, but
> thinking that irqbalance doesn't use the numa info the kernel exposes
> is incorrect.
>

I'm only just now wading into this area, but I thought one of the
advantages of multiple hardware queues was that we don't have to worry
about multiple cpus trying to access the buffer rings at the same
time, thus eliminating locking.  If the driver can't rely on that,
don't we lose that advantage?

Andy

Re: multiqueue interrupts...

[Arjan van de Ven]

On Fri, 19 Sep 2008 17:24:00 -0500
"Andy Fleming" <afleming@gmail.com> wrote:

> On Fri, Sep 19, 2008 at 5:11 PM, Arjan van de Ven
> <arjan@infradead.org> wrote:
> > On Fri, 19 Sep 2008 12:18:41 -0600
> > Matthew Wilcox <matthew@wil.cx> wrote:
> 
> >> In a storage / NUMA configuration we really want to set up one
> >> queue per cpu / package / node (depending on resource constraints)
> >> and know that the interrupt is going to come back to the same
> >> cpu / package / node. We definitely don't want irqbalanced moving
> >> the interrupt around.
> >
> > irqbalance is NUMA aware and places a penalty on placing an
> > interrupt "wrongly". We can argue on how strong this penalty should
> > be, but thinking that irqbalance doesn't use the numa info the
> > kernel exposes is incorrect.
> >
> 
> I'm only just now wading into this area, but I thought one of the
> advantages of multiple hardware queues was that we don't have to worry
> about multiple cpus trying to access the buffer rings at the same
> time, thus eliminating locking.  If the driver can't rely on that,
> don't we lose that advantage?

that's only true if you have at least the amount of queues as you have
logical cpus. Ask SGI about how many cpus they have in 3 years, and
then ask your NIC vendor how many queues they have planned for ;-)

and a per-cpu lock isn't really all THAT expensive.
the really big advantage is that you no longer cacheline bounce to hell
and back... and that you have either way.


-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org