Why is LRO off by default on ixgbe?

Why is LRO off by default on ixgbe?

[Ben Greear]

I just noticed that enabling LRO on ixgbe lets me reach about 9Gbps receive on two
NICs concurrently in an NFS test, where I was only getting about 6Gbps w/out it (1500 MTU).

Why is LRO disabled by default?

Thanks,
Ben
-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Why is LRO off by default on ixgbe?

[Stephen Hemminger]

On Wed, 23 Sep 2009 09:29:59 -0700
Ben Greear <greearb@candelatech.com> wrote:

> I just noticed that enabling LRO on ixgbe lets me reach about 9Gbps receive on two
> NICs concurrently in an NFS test, where I was only getting about 6Gbps w/out it (1500 MTU).
> 
> Why is LRO disabled by default?
> 
> Thanks,
> Ben

LRO is turned off if bridging or routing because of End to End requirements.

Re: Why is LRO off by default on ixgbe?

[Ben Greear]

On 09/23/2009 09:53 AM, Stephen Hemminger wrote:
> On Wed, 23 Sep 2009 09:29:59 -0700
> Ben Greear<greearb@candelatech.com>  wrote:
>
>> I just noticed that enabling LRO on ixgbe lets me reach about 9Gbps receive on two
>> NICs concurrently in an NFS test, where I was only getting about 6Gbps w/out it (1500 MTU).
>>
>> Why is LRO disabled by default?
>>
>> Thanks,
>> Ben
>
> LRO is turned off if bridging or routing because of End to End requirements.

That makes sense.

If I know that all interfaces in question can handle TSO and LRO,
I could manually enable LRO w/out risk, right?

Thanks,
Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Why is LRO off by default on ixgbe?

[Stephen Hemminger]

On Wed, 23 Sep 2009 09:57:51 -0700
Ben Greear <greearb@candelatech.com> wrote:

> On 09/23/2009 09:53 AM, Stephen Hemminger wrote:
> > On Wed, 23 Sep 2009 09:29:59 -0700
> > Ben Greear<greearb@candelatech.com>  wrote:
> >
> >> I just noticed that enabling LRO on ixgbe lets me reach about 9Gbps receive on two
> >> NICs concurrently in an NFS test, where I was only getting about 6Gbps w/out it (1500 MTU).
> >>
> >> Why is LRO disabled by default?
> >>
> >> Thanks,
> >> Ben
> >
> > LRO is turned off if bridging or routing because of End to End requirements.
> 
> That makes sense.
> 
> If I know that all interfaces in question can handle TSO and LRO,
> I could manually enable LRO w/out risk, right?
> 

The problem is that LRO merges TCP packets, this breaks the end-to-end
ack clocking and checksumming, and therefore is not enabled.
That is why GRO is the replacement solution (preserves packet boundaries)

Re: Why is LRO off by default on ixgbe?

[Ben Greear]

On 09/23/2009 10:07 AM, Stephen Hemminger wrote:
> On Wed, 23 Sep 2009 09:57:51 -0700
> Ben Greear<greearb@candelatech.com>  wrote:
>
>> On 09/23/2009 09:53 AM, Stephen Hemminger wrote:
>>> On Wed, 23 Sep 2009 09:29:59 -0700
>>> Ben Greear<greearb@candelatech.com>   wrote:
>>>
>>>> I just noticed that enabling LRO on ixgbe lets me reach about 9Gbps receive on two
>>>> NICs concurrently in an NFS test, where I was only getting about 6Gbps w/out it (1500 MTU).
>>>>
>>>> Why is LRO disabled by default?
>>>>
>>>> Thanks,
>>>> Ben
>>>
>>> LRO is turned off if bridging or routing because of End to End requirements.
>>
>> That makes sense.
>>
>> If I know that all interfaces in question can handle TSO and LRO,
>> I could manually enable LRO w/out risk, right?
>>
>
> The problem is that LRO merges TCP packets, this breaks the end-to-end
> ack clocking and checksumming, and therefore is not enabled.
> That is why GRO is the replacement solution (preserves packet boundaries)

Ok.  It seems GRO was enabled the whole time, but LRO is what gave me the
extra performance boost.

In this particular case, I'm not actually routing, though I do have ip-forward
enabled, so I guess LRO will be OK as long as I'm careful...

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Why is LRO off by default on ixgbe?

[Herbert Xu]

Ben Greear <greearb@candelatech.com> wrote:
>
> Ok.  It seems GRO was enabled the whole time, but LRO is what gave me the
> extra performance boost.
> 
> In this particular case, I'm not actually routing, though I do have ip-forward
> enabled, so I guess LRO will be OK as long as I'm careful...

What? A driver is either GRO or LRO, it can't be both.  What
kernel version was this?

Thanks,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Why is LRO off by default on ixgbe?

[Herbert Xu]

On Fri, Sep 25, 2009 at 02:37:27PM -0700, Herbert Xu wrote:
>
> > Ok.  It seems GRO was enabled the whole time, but LRO is what gave me the
> > extra performance boost.
> > 
> > In this particular case, I'm not actually routing, though I do have ip-forward
> > enabled, so I guess LRO will be OK as long as I'm careful...
> 
> What? A driver is either GRO or LRO, it can't be both.  What
> kernel version was this?

Oh perhaps you mean LRO performed by hardware? That would make
sense indeed.  It's also one of the reasons why we hope the hardware
folks would switch over to GRO so we can enable it for everyone :)

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Why is LRO off by default on ixgbe?

[Ben Greear]

On 09/25/2009 02:43 PM, Herbert Xu wrote:
> On Fri, Sep 25, 2009 at 02:37:27PM -0700, Herbert Xu wrote:
>>
>>> Ok.  It seems GRO was enabled the whole time, but LRO is what gave me the
>>> extra performance boost.
>>>
>>> In this particular case, I'm not actually routing, though I do have ip-forward
>>> enabled, so I guess LRO will be OK as long as I'm careful...
>>
>> What? A driver is either GRO or LRO, it can't be both.  What
>> kernel version was this?
>
> Oh perhaps you mean LRO performed by hardware? That would make
> sense indeed.  It's also one of the reasons why we hope the hardware
> folks would switch over to GRO so we can enable it for everyone :)

I assume it's in hardware.  I am using ixgbe driver, 82599 chipset,
2.6.31 kernel.

GRO was showing as enabled the whole time in ethtool.  Turning on LRO gave me an extra
6Gbps (from 12Gbps -> 18Gbps) receive throughput, so it certainly had some affect!

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com