Receive steering and hash and cache misses

Receive steering and hash and cache misses

[Stephen Hemminger]

Although Receive Packet Steering can use a hardware generated receive hash
the device driver still causes an unnecessary cache miss on the interrupt
processing CPU.  The current Ethernet network device driver receive processing
has the device driver calling eth_type_trans() which causes a the
interrupt CPU to read the received frame header.

Is there some way the hardware receive hash value could be used to
steer to the receive CPU, then have the receive CPU find the Ethernet
type field (eth_type_trans)?

Re: Receive steering and hash and cache misses

[Tom Herbert]

On Fri, Apr 2, 2010 at 10:26 AM, Stephen Hemminger
<shemminger@vyatta.com> wrote:
>
> Although Receive Packet Steering can use a hardware generated receive hash
> the device driver still causes an unnecessary cache miss on the interrupt
> processing CPU.  The current Ethernet network device driver receive processing
> has the device driver calling eth_type_trans() which causes a the
> interrupt CPU to read the received frame header.
>

It should be possible to deduce the values set by eth_type_trans from
the RX descriptor along with the RX hash.  I'll post the patch getting
rxhash from bnx2x which does this.

> Is there some way the hardware receive hash value could be used to
> steer to the receive CPU, then have the receive CPU find the Ethernet
> type field (eth_type_trans)?
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Receive steering and hash and cache misses

[Stephen Hemminger]

On Fri, 2 Apr 2010 10:59:43 -0700
Tom Herbert <therbert@google.com> wrote:

> On Fri, Apr 2, 2010 at 10:26 AM, Stephen Hemminger
> <shemminger@vyatta.com> wrote:
> >
> > Although Receive Packet Steering can use a hardware generated receive hash
> > the device driver still causes an unnecessary cache miss on the interrupt
> > processing CPU.  The current Ethernet network device driver receive processing
> > has the device driver calling eth_type_trans() which causes a the
> > interrupt CPU to read the received frame header.
> >
> 
> It should be possible to deduce the values set by eth_type_trans from
> the RX descriptor along with the RX hash.  I'll post the patch getting
> rxhash from bnx2x which does this.
> 

On sky2, I get only RSS, Checksum, and length from descriptor info.

Re: Receive steering and hash and cache misses

[Eric Dumazet]

Le vendredi 02 avril 2010 à 11:54 -0700, Stephen Hemminger a écrit :
> On Fri, 2 Apr 2010 10:59:43 -0700
> Tom Herbert <therbert@google.com> wrote:
> 
> > On Fri, Apr 2, 2010 at 10:26 AM, Stephen Hemminger
> > <shemminger@vyatta.com> wrote:
> > >
> > > Although Receive Packet Steering can use a hardware generated receive hash
> > > the device driver still causes an unnecessary cache miss on the interrupt
> > > processing CPU.  The current Ethernet network device driver receive processing
> > > has the device driver calling eth_type_trans() which causes a the
> > > interrupt CPU to read the received frame header.
> > >
> > 
> > It should be possible to deduce the values set by eth_type_trans from
> > the RX descriptor along with the RX hash.  I'll post the patch getting
> > rxhash from bnx2x which does this.
> > 
> 
> On sky2, I get only RSS, Checksum, and length from descriptor info.

Doesnt sky2 also provide vlan id (OP_RXVLAN/OP_RXCHKSVLAN) ?

A future version of hardware could provide more info perhaps...

Must eth_type_trans() be done *before* netif_receive_skb() ?

If a device provides a rxhash, maybe we can delay eth_type_trans() too.
If not, we need to access IP header anyway in the first cpu.

Re: Receive steering and hash and cache misses

[Stephen Hemminger]

On Fri, 02 Apr 2010 21:36:31 +0200
Eric Dumazet <eric.dumazet@gmail.com> wrote:

> Le vendredi 02 avril 2010 à 11:54 -0700, Stephen Hemminger a écrit :
> > On Fri, 2 Apr 2010 10:59:43 -0700
> > Tom Herbert <therbert@google.com> wrote:
> > 
> > > On Fri, Apr 2, 2010 at 10:26 AM, Stephen Hemminger
> > > <shemminger@vyatta.com> wrote:
> > > >
> > > > Although Receive Packet Steering can use a hardware generated receive hash
> > > > the device driver still causes an unnecessary cache miss on the interrupt
> > > > processing CPU.  The current Ethernet network device driver receive processing
> > > > has the device driver calling eth_type_trans() which causes a the
> > > > interrupt CPU to read the received frame header.
> > > >
> > > 
> > > It should be possible to deduce the values set by eth_type_trans from
> > > the RX descriptor along with the RX hash.  I'll post the patch getting
> > > rxhash from bnx2x which does this.
> > > 
> > 
> > On sky2, I get only RSS, Checksum, and length from descriptor info.
> 
> Doesnt sky2 also provide vlan id (OP_RXVLAN/OP_RXCHKSVLAN) ?
> 
> A future version of hardware could provide more info perhaps...

I have only some information from Marvell and no idea what they might
do with future hardware. 

> Must eth_type_trans() be done *before* netif_receive_skb() ?

In current arch yes, because netif_receive_skb is used for multiple
hardware types and the backlog queue could theoretically contain
skb's of different hardware types.  Also GRO works against RPS
since it does lookup work on the initial CPU and dirties the skb.

This is mostly theoretical at this point the bigger performance bottlenecks
are farther down the packet processing chain.