[RFC] ixgbe: is DCA really that good ?

[RFC] ixgbe: is DCA really that good ?

[Eric Dumazet]

Hi guys


In a forwarding [or RPS/RFS] setup, why should we populate cpu caches
with full frames content ? We only need first cache line to perform
routing [or RPS/RFS] decision.

-> DCA should be a knob (ethtool ?) that an admin can switch off and on,
port by port, not a CONFIG_IXGBE_DCA thing.

Thanks

Re: [RFC] ixgbe: is DCA really that good ?

[Tom Herbert]

On Wed, Apr 6, 2011 at 6:44 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> Hi guys
>
>
> In a forwarding [or RPS/RFS] setup, why should we populate cpu caches
> with full frames content ? We only need first cache line to perform
> routing [or RPS/RFS] decision.
>
DCA + accelerated RFS might make good.  But I agree that it should be
configurable.

> -> DCA should be a knob (ethtool ?) that an admin can switch off and on,
> port by port, not a CONFIG_IXGBE_DCA thing.
>
> Thanks
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [RFC] ixgbe: is DCA really that good ?

[Eric Dumazet]

Le mercredi 06 avril 2011 à 08:24 -0700, Tom Herbert a écrit :
> On Wed, Apr 6, 2011 at 6:44 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> > Hi guys
> >
> >
> > In a forwarding [or RPS/RFS] setup, why should we populate cpu caches
> > with full frames content ? We only need first cache line to perform
> > routing [or RPS/RFS] decision.
> >
> DCA + accelerated RFS might make good.  But I agree that it should be
> configurable.

An IRQ affinity mismatch is fatal with dca-core current code, because

dca3_get_tag() ->
	dca_common_get_tag() ->
		spin_lock_irqsave(&dca_lock, flags);

Wei Gu hit this on a 64 cpus machine (but only 8 cpus servicing ixgbe
interrupts), and finding the problem took lot of time.

Re: [RFC] ixgbe: is DCA really that good ?

[Alexander Duyck]

On 4/6/2011 6:44 AM, Eric Dumazet wrote:
> Hi guys
>
>
> In a forwarding [or RPS/RFS] setup, why should we populate cpu caches
> with full frames content ? We only need first cache line to perform
> routing [or RPS/RFS] decision.
>
> ->  DCA should be a knob (ethtool ?) that an admin can switch off and on,
> port by port, not a CONFIG_IXGBE_DCA thing.
>
> Thanks

The problem is the implementation essentially makes it so that DCA is 
controlled by the availability of the DCA providers.  What would 
probably be the best solution would be for the DCA provider to have a 
means of shutting down specific devices.

The quick and dirty way to disable DCA is to rmmod the ioatdma module 
from the system.  With that removed it will remove the DCA provider and 
essentially turn off DCA.

Thanks,

Alex