[RFC PATCH] drivers/net/tg3.c: Raise Jumbo Frame MTU to 9216?

[RFC PATCH] drivers/net/tg3.c: Raise Jumbo Frame MTU to 9216?

[Joe Perches]

The TG3 apparently supports 9K frame sizes.

http://www.broadcom.com/collateral/pb/5704C-PB05-R.pdf

Is exactly 9000 a hardware limit?

Should the jumbo frame MTU be raised to 9216 or 9216
less the size of MAC, VLAN, IP and TCP headers?

diff --git a/drivers/net/tg3.c b/drivers/net/tg3.c
index 9f6ffff..3727070 100644
--- a/drivers/net/tg3.c
+++ b/drivers/net/tg3.c
@@ -95,7 +95,7 @@
 /* hardware minimum and maximum for a single frame's data payload */
 #define TG3_MIN_MTU			60
 #define TG3_MAX_MTU(tp)	\
-	((tp->tg3_flags & TG3_FLAG_JUMBO_CAPABLE) ? 9000 : 1500)
+	((tp->tg3_flags & TG3_FLAG_JUMBO_CAPABLE) ? 9216 : 1500)
 
 /* These numbers seem to be hard coded in the NIC firmware somehow.
  * You can't change the ring sizes, but you can change where you place

Re: [RFC PATCH] drivers/net/tg3.c: Raise Jumbo Frame MTU to 9216?

[Michael Chan]

On Wed, 2010-09-15 at 10:41 -0700, Joe Perches wrote:
> The TG3 apparently supports 9K frame sizes.
> 
> http://www.broadcom.com/collateral/pb/5704C-PB05-R.pdf
> 
> Is exactly 9000 a hardware limit?
> 
> Should the jumbo frame MTU be raised to 9216 or 9216
> less the size of MAC, VLAN, IP and TCP headers?

9000 has been the de facto standard, has it been changed recently?

Anyway, we've never done any testing on 9216.  As it uses up to 2 more
internal mbufs per packet, there may not be sufficient buffers inside
the chip for optimal operations.  At best, some water marks will need to
be tweaked.  The hardware statistics counters (ethtool -S) also may not
work for packets bigger than 9022 bytes.

> 
> diff --git a/drivers/net/tg3.c b/drivers/net/tg3.c
> index 9f6ffff..3727070 100644
> --- a/drivers/net/tg3.c
> +++ b/drivers/net/tg3.c
> @@ -95,7 +95,7 @@
>  /* hardware minimum and maximum for a single frame's data payload */
>  #define TG3_MIN_MTU			60
>  #define TG3_MAX_MTU(tp)	\
> -	((tp->tg3_flags & TG3_FLAG_JUMBO_CAPABLE) ? 9000 : 1500)
> +	((tp->tg3_flags & TG3_FLAG_JUMBO_CAPABLE) ? 9216 : 1500)
>  
>  /* These numbers seem to be hard coded in the NIC firmware somehow.
>   * You can't change the ring sizes, but you can change where you place
> 
> 
> 

Re: [RFC PATCH] drivers/net/tg3.c: Raise Jumbo Frame MTU to 9216?

[Joe Perches]

On Wed, 2010-09-15 at 10:57 -0700, Michael Chan wrote:
> On Wed, 2010-09-15 at 10:41 -0700, Joe Perches wrote:
> > The TG3 apparently supports 9K frame sizes.
> > http://www.broadcom.com/collateral/pb/5704C-PB05-R.pdf
> > Is exactly 9000 a hardware limit?
> > Should the jumbo frame MTU be raised to 9216 or 9216
> > less the size of MAC, VLAN, IP and TCP headers?
> 9000 has been the de facto standard, has it been changed recently?

I know of a performance lab that's trying to use 9216 as a "standard"
jumbo frame length.

Unrelated to the performance lab:
http://www.uoregon.edu/~joe/jumbo-clean-gear.html
9216 seems popular, especially with Cisco gear.
Contrary to that link, the Cisco 3750 does work with 9216 length
jumbo frames.

> Anyway, we've never done any testing on 9216.  As it uses up to 2 more
> internal mbufs per packet, there may not be sufficient buffers inside
> the chip for optimal operations.  At best, some water marks will need to
> be tweaked.  The hardware statistics counters (ethtool -S) also may not
> work for packets bigger than 9022 bytes.

Thanks, do you have pointers to where the tweaking needs to be done?

Is this define a hardware upper bound?
#define TG3_RX_JMB_DMA_SZ		9046

cheers, Joe

Re: [RFC PATCH] drivers/net/tg3.c: Raise Jumbo Frame MTU to 9216?

[Ben Hutchings]

On Wed, 2010-09-15 at 12:14 -0700, Joe Perches wrote:
> On Wed, 2010-09-15 at 10:57 -0700, Michael Chan wrote:
> > On Wed, 2010-09-15 at 10:41 -0700, Joe Perches wrote:
> > > The TG3 apparently supports 9K frame sizes.
> > > http://www.broadcom.com/collateral/pb/5704C-PB05-R.pdf
> > > Is exactly 9000 a hardware limit?
> > > Should the jumbo frame MTU be raised to 9216 or 9216
> > > less the size of MAC, VLAN, IP and TCP headers?
> > 9000 has been the de facto standard, has it been changed recently?
> 
> I know of a performance lab that's trying to use 9216 as a "standard"
> jumbo frame length.
[...]

We also use 9000 as the 'normal' jumbo MTU, though hard limit in sfc is
a bit higher than this.

The frame length limit is not defined only by the DMA engine and FIFOs
but also by the MAC and PHY.  On the receive side, the MAC and/or PHY
need to handle skew between the remote and local clocks which can grow
proportionally with frame length.  For a controller which is used with
several different PHYs, I would be hesitant to raise the driver's limit.

Ben.

-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: [RFC PATCH] drivers/net/tg3.c: Raise Jumbo Frame MTU to 9216?

[David Newman]

On 9/15/10 12:14 PM, Joe Perches wrote:
> On Wed, 2010-09-15 at 10:57 -0700, Michael Chan wrote:
>> On Wed, 2010-09-15 at 10:41 -0700, Joe Perches wrote:
>>> The TG3 apparently supports 9K frame sizes.
>>> http://www.broadcom.com/collateral/pb/5704C-PB05-R.pdf
>>> Is exactly 9000 a hardware limit?
>>> Should the jumbo frame MTU be raised to 9216 or 9216
>>> less the size of MAC, VLAN, IP and TCP headers?
>> 9000 has been the de facto standard, has it been changed recently?
> 
> I know of a performance lab that's trying to use 9216 as a "standard"
> jumbo frame length.
> 
> Unrelated to the performance lab:
> http://www.uoregon.edu/~joe/jumbo-clean-gear.html
> 9216 seems popular, especially with Cisco gear.
> Contrary to that link, the Cisco 3750 does work with 9216 length
> jumbo frames.
> 
>> Anyway, we've never done any testing on 9216. 

9216 is commonly used in Ethernet switch testing [1]. Most data center
switches support it, including some built around Broadcom silicon (e.g.,
Dell PowerConnect 8024, Extreme Summit X650).

There is no IEEE standard for jumbo lengths. The IETF's RFC 5180
recommends 9216 for switch performance testing.

Although 5180 specs this length as a SHOULD and not a MUST, its use does
represent widespread industry practice in switch performance benchmarking.

Regards,
David Newman
Network Test

[1] Length from first byte of DA to last byte of CRC absent any VLAN
header(s).