Re: RFC: MTU for serving NFS on Infiniband

[Stephen Hemminger <shemminger@vyatta.com>]

On Thu, 26 Aug 2010 08:43:42 -0600 (Mountain Daylight Time)
Marc Aurele La France <tsi@ualberta.ca> wrote:

> On Thu, 26 Aug 2010, Eric Dumazet wrote:
> > Le jeudi 26 août 2010 à 05:40 -0600, Marc Aurele La France a écrit :
> 
> >> Steady now.  There's no need to YELL nor be arrogant.  You and I both know
> >> there's a place for NFS over UDP.  That's not changing any time soon.  While
> >> I'm aware of the issue you brought up, it is separate from the one at hand in
> >> this discussion.
> 
> >> I do want to thank you, however, for reminding me of TCP.  It's something
> >> 20/20 hindsight says I should have checked out before starting this thread.
> >> Logistically, it'll be a few days before I can do so though.  If that allows
> >> me to increase the MTU all the way up to 65520, then this UDP thing will
> >> likely remain unresolved.
> 
> > Unfortunately, your infiniband device lacks NETIF_F_SG support.
> 
> Oh, the device itself probably has something similar, but ipoib 
> (IP-over-Infiniband) doesn't export that capability.
> 
> > MTU a bit larger than PAGE_SIZE-overhead will need high order
> > allocations ?
> 
> Right.  And a 65520 MTU allocates sk_buff's with 128K contiguous payloads.

Infiniband device driver needs to be fixed to do SG and checksum offload.
Otherwise it is insane to try and run large MTU over it. I even wonder if
the dev_change_mtu() function should reject > PAGESIZE mtu for devices
that don't do scatter/gather or at least a raise a warning.