Does IPv6 support Jumbograms ?

Does IPv6 support Jumbograms ?

[Francois WELLENREITER]

Hi there,

I've been recently running performance tests with the loopback interface
increasing the MTU over the 65535 byte limit.
I was really surprised to see that a simple scp onto the ::1 address
systematically blocked after transferring about 2,4 MB.
My interpretation of this behavior is that the current IPv6 kernel layer
does not support Jumbograms at all. Am I wrong ?
If that's not the case, what could then the right interpretation of this
issue ?
And whenever I'm right, is there any plan to support this feature in a
near future ?

Thanks in advance for any help.
Regards,

                  François

Re: Does IPv6 support Jumbograms ?

[Hannes Frederic Sowa]

Hi!

On Wed, Apr 09, 2014 at 09:42:43PM +0200, Francois WELLENREITER wrote:
> I've been recently running performance tests with the loopback interface
> increasing the MTU over the 65535 byte limit.
> I was really surprised to see that a simple scp onto the ::1 address
> systematically blocked after transferring about 2,4 MB.
> My interpretation of this behavior is that the current IPv6 kernel layer
> does not support Jumbograms at all. Am I wrong ?
> If that's not the case, what could then the right interpretation of this
> issue ?
> And whenever I'm right, is there any plan to support this feature in a
> near future ?

IPv6 should handle the reception of jumbo frames quite fine, even the UDP size
override IIRC. But there is no support for large outgoing UDP frames, and I
currently don't know about TCP-MSS override works.

I guess you should be able to send jumbos with rawsocket if you attach the
dsthop header manually, but I have never tried it (maybe it will get blocked
by some check in ip6_output, HDRINCL and H-b-H option setsockopt could work
also).

I currently don't see the need for that option if no one also takes care about
UDP and TCP paths, so I guess support is coming as soon as someone cares. ;)

Greetings,

  Hannes

Re: Does IPv6 support Jumbograms ?

[Eric Dumazet]

On Wed, 2014-04-09 at 21:42 +0200, Francois WELLENREITER wrote:
> Hi there,
> 
> I've been recently running performance tests with the loopback interface
> increasing the MTU over the 65535 byte limit.
> I was really surprised to see that a simple scp onto the ::1 address
> systematically blocked after transferring about 2,4 MB.
> My interpretation of this behavior is that the current IPv6 kernel layer
> does not support Jumbograms at all. Am I wrong ?
> If that's not the case, what could then the right interpretation of this
> issue ?
> And whenever I'm right, is there any plan to support this feature in a
> near future ?

What do you mean by blocked ?

Please give more details (kernel version, exact mtu...), because it
should not happen !

# ifconfig lo mtu 100000
# scp -6 vmlinux edumazet@ip6-localhost:/tmp
Password: 
vmlinux
100%   24MB  23.5MB/s   00:00    
# ls -l /tmp/vmlinux
-rwxr-xr-x 1 edumazet eng 24668915 Apr  9 14:37 /tmp/vmlinux

Re: Does IPv6 support Jumbograms ?

[Hannes Frederic Sowa]

On Wed, Apr 09, 2014 at 02:41:05PM -0700, Eric Dumazet wrote:
> On Wed, 2014-04-09 at 21:42 +0200, Francois WELLENREITER wrote:
> > Hi there,
> > 
> > I've been recently running performance tests with the loopback interface
> > increasing the MTU over the 65535 byte limit.
> > I was really surprised to see that a simple scp onto the ::1 address
> > systematically blocked after transferring about 2,4 MB.
> > My interpretation of this behavior is that the current IPv6 kernel layer
> > does not support Jumbograms at all. Am I wrong ?
> > If that's not the case, what could then the right interpretation of this
> > issue ?
> > And whenever I'm right, is there any plan to support this feature in a
> > near future ?
> 
> What do you mean by blocked ?
> 
> Please give more details (kernel version, exact mtu...), because it
> should not happen !
> 
> # ifconfig lo mtu 100000
> # scp -6 vmlinux edumazet@ip6-localhost:/tmp
> Password: 
> vmlinux
> 100%   24MB  23.5MB/s   00:00    
> # ls -l /tmp/vmlinux
> -rwxr-xr-x 1 edumazet eng 24668915 Apr  9 14:37 /tmp/vmlinux

I couldn't test it on my development system with a recent net-next kernel
yet, but on my laptop with a distribution 3.13.9 kernel this happened too.

I tested with two netcats and threw a dd from /dev/zero into it with a
1MB block size. ssh seems not to be aggressive enough to trigger this:

# tcpdump -i lo -vvv -nKS tcp and port 9988
# tcpdump -i lo -nKS -vvv tcp and port 9988
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
00:26:02.914511 IP6 (hlim 64, next-header TCP (6) payload length: 40) ::1.52141 > ::1.nsesrvr: Flags [S], seq 2750352628, win 43690, options [mss 65535,sackOK,TS val 43288646 ecr 0,nop,wscale 7], length 0
00:26:02.914593 IP6 (hlim 64, next-header TCP (6) payload length: 40) ::1.nsesrvr > ::1.52141: Flags [S.], seq 1418051855, ack 2750352629, win 43690, options [mss 65535,sackOK,TS val 43288647 ecr 43288646,nop,wscale 7], length 0
00:26:02.914659 IP6 (hlim 64, next-header TCP (6) payload length: 32) ::1.52141 > ::1.nsesrvr: Flags [.], seq 2750352629, ack 1418051856, win 342, options [nop,nop,TS val 43288647 ecr 43288647], length 0
00:26:02.915013 IP6 (hlim 64, next-header TCP (6) payload length: 8224) ::1.52141 > ::1.nsesrvr: Flags [P.], seq 2750352629:2750360821, ack 1418051856, win 342, options [nop,nop,TS val 43288647 ecr 43288647], length 8192
00:26:02.915089 IP6 (hlim 64, next-header TCP (6) payload length: 32) ::1.nsesrvr > ::1.52141: Flags [.], seq 1418051856, ack 2750360821, win 1366, options [nop,nop,TS val 43288647 ecr 43288647], length 0
00:26:02.915289 IP6 (hlim 64, next-header TCP (6) payload length: 8224) ::1.52141 > ::1.nsesrvr: Flags [P.], seq 2750360821:2750369013, ack 1418051856, win 342, options [nop,nop,TS val 43288647 ecr 43288647], length 8192
00:26:02.915342 IP6 (hlim 64, next-header TCP (6) payload length: 32) ::1.nsesrvr > ::1.52141: Flags [.], seq 1418051856, ack 2750369013, win 2389, options [nop,nop,TS val 43288647 ecr 43288647], length 0
00:26:02.915467 IP6 (hlim 64, next-header TCP (6) payload length: 8224) ::1.52141 > ::1.nsesrvr: Flags [P.], seq 2750369013:2750377205, ack 1418051856, win 342, options [nop,nop,TS val 43288647 ecr 43288647], length 8192
00:26:02.915558 IP6 (hlim 64, next-header TCP (6) payload length: 32) ::1.nsesrvr > ::1.52141: Flags [.], seq 1418051856, ack 2750377205, win 3413, options [nop,nop,TS val 43288648 ecr 43288647], length 0
00:26:02.915723 IP6 (hlim 64, next-header TCP (6) payload length: 8224) ::1.52141 > ::1.nsesrvr: Flags [P.], seq 2750377205:2750385397, ack 1418051856, win 342, options [nop,nop,TS val 43288648 ecr 43288648], length 8192
00:26:02.915778 IP6 (hlim 64, next-header TCP (6) payload length: 32) ::1.nsesrvr > ::1.52141: Flags [.], seq 1418051856, ack 2750385397, win 3639, options [nop,nop,TS val 43288648 ecr 43288648], length 0
00:26:02.915898 IP6 (hlim 64, next-header TCP (6) payload length: 8224) ::1.52141 > ::1.nsesrvr: Flags [P.], seq 2750385397:2750393589, ack 1418051856, win 342, options [nop,nop,TS val 43288648 ecr 43288648], length 8192
00:26:02.915952 IP6 (hlim 64, next-header TCP (6) payload length: 32) ::1.nsesrvr > ::1.52141: Flags [.], seq 1418051856, ack 2750393589, win 3639, options [nop,nop,TS val 43288648 ecr 43288648], length 0
00:26:02.916072 IP6 (hlim 64, next-header TCP (6) payload length: 8224) ::1.52141 > ::1.nsesrvr: Flags [P.], seq 2750393589:2750401781, ack 1418051856, win 342, options [nop,nop,TS val 43288648 ecr 43288648], length 8192
00:26:02.916126 IP6 (hlim 64, next-header TCP (6) payload length: 32) ::1.nsesrvr > ::1.52141: Flags [.], seq 1418051856, ack 2750401781, win 3639, options [nop,nop,TS val 43288648 ecr 43288648], length 0
00:26:02.916245 IP6 (hlim 64, next-header TCP (6) payload length: 8224) ::1.52141 > ::1.nsesrvr: Flags [P.], seq 2750401781:2750409973, ack 1418051856, win 342, options [nop,nop,TS val 43288648 ecr 43288648], length 8192

00:26:02.916850 IP6 (hlim 64, next-header TCP (6) payload length: 19) ::1.52141 > ::1.nsesrvr: [|tcp]
00:26:02.918682 IP6 (hlim 64, next-header TCP (6) payload length: 19) ::1.52141 > ::1.nsesrvr: [|tcp]
00:26:02.919852 IP6 (hlim 64, next-header TCP (6) payload length: 19) ::1.52141 > ::1.nsesrvr: [|tcp]
00:26:02.920966 IP6 (hlim 64, next-header TCP (6) payload length: 19) ::1.52141 > ::1.nsesrvr: [|tcp]
00:26:02.922092 IP6 (hlim 64, next-header TCP (6) payload length: 19) ::1.52141 > ::1.nsesrvr: [|tcp]
00:26:02.923194 IP6 (hlim 64, next-header TCP (6) payload length: 19) ::1.52141 > ::1.nsesrvr: [|tcp]
00:26:02.955634 IP6 (hlim 64, next-header TCP (6) payload length: 32) ::1.nsesrvr > ::1.52141: Flags [.], seq 1418051856, ack 2750409973, win 3639, options [nop,nop,TS val 43288688 ecr 43288648], length 0
00:26:02.955832 IP6 (hlim 64, next-header TCP (6) payload length: 19) ::1.52141 > ::1.nsesrvr: [|tcp]
00:26:03.160698 IP6 (hlim 64, next-header TCP (6) payload length: 19) ::1.52141 > ::1.nsesrvr: [|tcp]
00:26:03.572148 IP6 (hlim 64, next-header TCP (6) payload length: 19) ::1.52141 > ::1.nsesrvr: [|tcp]
00:26:04.394117 IP6 (hlim 64, next-header TCP (6) payload length: 19) ::1.52141 > ::1.nsesrvr: [|tcp]

Hmm... that is strange.

Bye,

  Hannes

Re: Does IPv6 support Jumbograms ?

[Eric Dumazet]

On Thu, 2014-04-10 at 00:35 +0200, Hannes Frederic Sowa wrote:
> On Wed, Apr 09, 2014 at 02:41:05PM -0700, Eric Dumazet wrote:
> > What do you mean by blocked ?
> > 
> > Please give more details (kernel version, exact mtu...), because it
> > should not happen !
> > 
> > # ifconfig lo mtu 100000
> > # scp -6 vmlinux edumazet@ip6-localhost:/tmp
> > Password: 
> > vmlinux
> > 100%   24MB  23.5MB/s   00:00    
> > # ls -l /tmp/vmlinux
> > -rwxr-xr-x 1 edumazet eng 24668915 Apr  9 14:37 /tmp/vmlinux
> 
> I couldn't test it on my development system with a recent net-next kernel
> yet, but on my laptop with a distribution 3.13.9 kernel this happened too.

Oh well, it seems ip6_mtu() needs to cap mtu to max mtu for non
jumbograms...

-	return mtu;
+	return min_t(unsigned int, mtu, IP6_MAX_MTU);

#define IP6_MAX_MTU (65535 + 40)

Re: Does IPv6 support Jumbograms ?

[Hannes Frederic Sowa]

On Wed, Apr 09, 2014 at 05:36:10PM -0700, Eric Dumazet wrote:
> On Thu, 2014-04-10 at 00:35 +0200, Hannes Frederic Sowa wrote:
> > On Wed, Apr 09, 2014 at 02:41:05PM -0700, Eric Dumazet wrote:
> > > What do you mean by blocked ?
> > > 
> > > Please give more details (kernel version, exact mtu...), because it
> > > should not happen !
> > > 
> > > # ifconfig lo mtu 100000
> > > # scp -6 vmlinux edumazet@ip6-localhost:/tmp
> > > Password: 
> > > vmlinux
> > > 100%   24MB  23.5MB/s   00:00    
> > > # ls -l /tmp/vmlinux
> > > -rwxr-xr-x 1 edumazet eng 24668915 Apr  9 14:37 /tmp/vmlinux
> > 
> > I couldn't test it on my development system with a recent net-next kernel
> > yet, but on my laptop with a distribution 3.13.9 kernel this happened too.
> 
> Oh well, it seems ip6_mtu() needs to cap mtu to max mtu for non
> jumbograms...
> 
> -	return mtu;
> +	return min_t(unsigned int, mtu, IP6_MAX_MTU);
> 
> #define IP6_MAX_MTU (65535 + 40)

I thought ip6_default_advmss will deal with this which correctly limits the
mtu. Capping dst_mtu would make it harder to implement jumbograms some day.

I'll look into it, thank you.

Re: Does IPv6 support Jumbograms ?

[Eric Dumazet]

On Thu, 2014-04-10 at 15:28 +0200, Hannes Frederic Sowa wrote:

> I thought ip6_default_advmss will deal with this which correctly limits the
> mtu. Capping dst_mtu would make it harder to implement jumbograms some day.

I doubt we'll do this (jumbo grams)

This is a terrible academic idea.

Re: Does IPv6 support Jumbograms ?

[Hannes Frederic Sowa]

On Wed, Apr 09, 2014 at 05:36:10PM -0700, Eric Dumazet wrote:
> On Thu, 2014-04-10 at 00:35 +0200, Hannes Frederic Sowa wrote:
> > On Wed, Apr 09, 2014 at 02:41:05PM -0700, Eric Dumazet wrote:
> > > What do you mean by blocked ?
> > > 
> > > Please give more details (kernel version, exact mtu...), because it
> > > should not happen !
> > > 
> > > # ifconfig lo mtu 100000
> > > # scp -6 vmlinux edumazet@ip6-localhost:/tmp
> > > Password: 
> > > vmlinux
> > > 100%   24MB  23.5MB/s   00:00    
> > > # ls -l /tmp/vmlinux
> > > -rwxr-xr-x 1 edumazet eng 24668915 Apr  9 14:37 /tmp/vmlinux
> > 
> > I couldn't test it on my development system with a recent net-next kernel
> > yet, but on my laptop with a distribution 3.13.9 kernel this happened too.
> 
> Oh well, it seems ip6_mtu() needs to cap mtu to max mtu for non
> jumbograms...
> 
> -	return mtu;
> +	return min_t(unsigned int, mtu, IP6_MAX_MTU);
> 
> #define IP6_MAX_MTU (65535 + 40)

                             ^ minus

I think we should do that. Otherwise we would have to start supporting
jumbograms in the output path generally.

Greetings,

  Hannes

Re: Does IPv6 support Jumbograms ?

[Eric Dumazet]

On Fri, 2014-04-11 at 01:54 +0200, Hannes Frederic Sowa wrote:

> > #define IP6_MAX_MTU (65535 + 40)
> 
>                              ^ minus
> 
> I think we should do that. Otherwise we would have to start supporting
> jumbograms in the output path generally.

Why 'minus' ?

hdr->payload_len = htons(seg_len);

seg_len doesn't include the IPv6 header.

So it seems we can use up to 65575 for the device mtu ?

Anyway I am testing the patch ;)

[PATCH] ipv6: Limit mtu to 65572 bytes

[Eric Dumazet]

From: Eric Dumazet <edumazet@google.com>

Francois reported that setting big mtu on loopback device could prevent
tcp sessions making progress.

We do not support (yet ?) IPv6 Jumbograms and cook corrupted packets.

We must limit the IPv6 MTU to (65535 + 40) bytes in theory.

In practice, its better to align to a multiple of 4 for optimal TCP
performance.

Tested:

ifconfig lo mtu 70000
netperf -H ::1

Before patch :

-> Throughput :   0.05 Mbits

After patch :

-> Throughput : 36624.24 Mbits

Reported-by: Francois WELLENREITER <f.wellenreiter@gmail.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
 include/net/ip6_route.h |    7 +++++++
 net/ipv6/route.c        |    5 +++--
 2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/include/net/ip6_route.h b/include/net/ip6_route.h
index 3c3bb184eb8f..8510a0c14b1d 100644
--- a/include/net/ip6_route.h
+++ b/include/net/ip6_route.h
@@ -32,6 +32,13 @@ struct route_info {
 #define RT6_LOOKUP_F_SRCPREF_PUBLIC	0x00000010
 #define RT6_LOOKUP_F_SRCPREF_COA	0x00000020
 
+/* We do not (yet ?) support IPv6 jumbograms (RFC 2675)
+ * Unlike IPv4, hdr->seg_len doesn't include the IPv6 header
+ * In theory, limit is 0xFFFF + 40, but we round-down to a multiple
+ * of 4 for better TCP performance.
+ */
+#define IP6_MAX_MTU (0xFFFC + sizeof(struct ipv6hdr))
+
 /*
  * rt6_srcprefs2flags() and rt6_flags2srcprefs() translate
  * between IPV6_ADDR_PREFERENCES socket option values
diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index 5015c50a5ba7..5ea462eacd9f 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -1338,7 +1338,7 @@ static unsigned int ip6_mtu(const struct dst_entry *dst)
 	unsigned int mtu = dst_metric_raw(dst, RTAX_MTU);
 
 	if (mtu)
-		return mtu;
+		goto out;
 
 	mtu = IPV6_MIN_MTU;
 
@@ -1348,7 +1348,8 @@ static unsigned int ip6_mtu(const struct dst_entry *dst)
 		mtu = idev->cnf.mtu6;
 	rcu_read_unlock();
 
-	return mtu;
+out:
+	return min_t(unsigned int, mtu, IP6_MAX_MTU);
 }
 
 static struct dst_entry *icmp6_dst_gc_list;

Re: [PATCH] ipv6: Limit mtu to 65572 bytes

[Eric Dumazet]

On Thu, 2014-04-10 at 18:42 -0700, Eric Dumazet wrote:
> From: Eric Dumazet <edumazet@google.com>
> 
> Francois reported that setting big mtu on loopback device could prevent
> tcp sessions making progress.
> 
> We do not support (yet ?) IPv6 Jumbograms and cook corrupted packets.
> 
> We must limit the IPv6 MTU to (65535 + 40) bytes in theory.
> 
> In practice, its better to align to a multiple of 4 for optimal TCP
> performance.

Same story for IPv4 btw : current lo mtu=65536 is suboptimal


lpq83:~# ifconfig lo mtu 65536
lpq83:~# ./netperf 
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
localhost () port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  16384    10.00    33020.76   


lpq83:~# ifconfig lo mtu 65533
lpq83:~# ./netperf 
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
localhost () port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  16384    10.00    39010.09   

Re: [PATCH] ipv6: Limit mtu to 65572 bytes

[YOSHIFUJI Hideaki]

Eric Dumazet wrote:

> We must limit the IPv6 MTU to (65535 + 40) bytes in theory.
> 
> In practice, its better to align to a multiple of 4 for optimal TCP
> performance.

It is a TCP issue.  We should not limit the mtu to 65532+40.
I am for 65535+40. Otherwise, other protocol such as UDP cannot
use full mtu as before.

--yoshfuji

Re: [PATCH] ipv6: Limit mtu to 65572 bytes

[Hannes Frederic Sowa]

On Fri, Apr 11, 2014 at 11:30:44AM +0900, YOSHIFUJI Hideaki wrote:
> Eric Dumazet wrote:
> 
> > We must limit the IPv6 MTU to (65535 + 40) bytes in theory.
> > 
> > In practice, its better to align to a multiple of 4 for optimal TCP
> > performance.
> 
> It is a TCP issue.  We should not limit the mtu to 65532+40.
> I am for 65535+40. Otherwise, other protocol such as UDP cannot
> use full mtu as before.

I have not seen problems with max ipv6 mtu limit of 65535+40 and tcp.

I agree this would be a better approach, and maybe & ~3 the mtu/mss
in tcp code? I assume people expect maximum udp packet sizes working
over loopback?

Greetings,

  Hannes

Re: [PATCH] ipv6: Limit mtu to 65572 bytes

[Eric Dumazet]

On Fri, 2014-04-11 at 04:57 +0200, Hannes Frederic Sowa wrote:
> On Fri, Apr 11, 2014 at 11:30:44AM +0900, YOSHIFUJI Hideaki wrote:
> > Eric Dumazet wrote:
> > 
> > > We must limit the IPv6 MTU to (65535 + 40) bytes in theory.
> > > 
> > > In practice, its better to align to a multiple of 4 for optimal TCP
> > > performance.
> > 
> > It is a TCP issue.  We should not limit the mtu to 65532+40.
> > I am for 65535+40. Otherwise, other protocol such as UDP cannot
> > use full mtu as before.
> 
> I have not seen problems with max ipv6 mtu limit of 65535+40 and tcp.
> 
> I agree this would be a better approach, and maybe & ~3 the mtu/mss
> in tcp code? I assume people expect maximum udp packet sizes working
> over loopback?

Yes, recent Intel cpus have really fast mem copy, even with different
alignments for source/dest.

But old cpus or other arches might have an issue here. Apparently nobody
really cares, so I wont argue and send a V2.

RE: [PATCH] ipv6: Limit mtu to 65572 bytes

[David Laight]

From: Eric Dumazet
> Yes, recent Intel cpus have really fast mem copy, even with different
> alignments for source/dest.

Yes, they've finally thrown a small amount of logic at it.
Probably a cache line sized barrel shifter.
'rep movsb' has also been optimised for short transfers (but is
slower for longer ones).

At least the zero length 'rep movsb' at the end of the inlined
memcpy will not take 40+ clocks on those cpus.

	David

RE: [PATCH] ipv6: Limit mtu to 65572 bytes

[David Laight]

From: Hannes Frederic Sowa
> On Fri, Apr 11, 2014 at 11:30:44AM +0900, YOSHIFUJI Hideaki wrote:
> > Eric Dumazet wrote:
> >
> > > We must limit the IPv6 MTU to (65535 + 40) bytes in theory.
> > >
> > > In practice, its better to align to a multiple of 4 for optimal TCP
> > > performance.
> >
> > It is a TCP issue.  We should not limit the mtu to 65532+40.
> > I am for 65535+40. Otherwise, other protocol such as UDP cannot
> > use full mtu as before.
> 
> I have not seen problems with max ipv6 mtu limit of 65535+40 and tcp.
> 
> I agree this would be a better approach, and maybe & ~3 the mtu/mss
> in tcp code? I assume people expect maximum udp packet sizes working
> over loopback?

The 'trick' would be to generate slightly short messages in the tcp
transmit code when doing so wouldn't generate an extra packet.
So (using very small numbers) if the max userdata was 63 bytes a 120
byte transmit would generate two 60 byte packets, but 121 to 126 byte
transmits would still generate two packets.

	David

Re: [PATCH] ipv6: Limit mtu to 65572 bytes

[Eric Dumazet]

From: Eric Dumazet <edumazet@google.com>

Francois reported that setting big mtu on loopback device could prevent
tcp sessions making progress.

We do not support (yet ?) IPv6 Jumbograms and cook corrupted packets.

We must limit the IPv6 MTU to (65535 + 40) bytes in theory.

Tested:

ifconfig lo mtu 70000
netperf -H ::1

Before patch : Throughput :   0.05 Mbits

After patch : Throughput : 35484 Mbits

Reported-by: Francois WELLENREITER <f.wellenreiter@gmail.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
v2: set the max limit, not round-down it.

 include/net/ip6_route.h |    5 +++++
 net/ipv6/route.c        |    5 +++--
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/include/net/ip6_route.h b/include/net/ip6_route.h
index 3c3bb184eb8f..6c4f5eac98e7 100644
--- a/include/net/ip6_route.h
+++ b/include/net/ip6_route.h
@@ -32,6 +32,11 @@ struct route_info {
 #define RT6_LOOKUP_F_SRCPREF_PUBLIC	0x00000010
 #define RT6_LOOKUP_F_SRCPREF_COA	0x00000020
 
+/* We do not (yet ?) support IPv6 jumbograms (RFC 2675)
+ * Unlike IPv4, hdr->seg_len doesn't include the IPv6 header
+ */
+#define IP6_MAX_MTU (0xFFFF + sizeof(struct ipv6hdr))
+
 /*
  * rt6_srcprefs2flags() and rt6_flags2srcprefs() translate
  * between IPV6_ADDR_PREFERENCES socket option values
diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index 5015c50a5ba7..5ea462eacd9f 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -1338,7 +1338,7 @@ static unsigned int ip6_mtu(const struct dst_entry *dst)
 	unsigned int mtu = dst_metric_raw(dst, RTAX_MTU);
 
 	if (mtu)
-		return mtu;
+		goto out;
 
 	mtu = IPV6_MIN_MTU;
 
@@ -1348,7 +1348,8 @@ static unsigned int ip6_mtu(const struct dst_entry *dst)
 		mtu = idev->cnf.mtu6;
 	rcu_read_unlock();
 
-	return mtu;
+out:
+	return min_t(unsigned int, mtu, IP6_MAX_MTU);
 }
 
 static struct dst_entry *icmp6_dst_gc_list;

Re: [PATCH] ipv6: Limit mtu to 65572 bytes

[YOSHIFUJI Hideaki]

Eric Dumazet wrote:
> From: Eric Dumazet <edumazet@google.com>
> 
> Francois reported that setting big mtu on loopback device could prevent
> tcp sessions making progress.
> 
> We do not support (yet ?) IPv6 Jumbograms and cook corrupted packets.
> 
> We must limit the IPv6 MTU to (65535 + 40) bytes in theory.
> 
> Tested:
> 
> ifconfig lo mtu 70000
> netperf -H ::1
> 
> Before patch : Throughput :   0.05 Mbits
> 
> After patch : Throughput : 35484 Mbits
> 
> Reported-by: Francois WELLENREITER <f.wellenreiter@gmail.com>
> Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: YOSHIFUJI Hideaki <yoshfuji@linux-ipv6.org>

--yoshfuji

Re: [PATCH] ipv6: Limit mtu to 65572 bytes

[YOSHIFUJI Hideaki]

Eric,

YOSHIFUJI Hideaki wrote:
> Eric Dumazet wrote:
>> From: Eric Dumazet <edumazet@google.com>
>>
>> Francois reported that setting big mtu on loopback device could prevent
>> tcp sessions making progress.
>>
>> We do not support (yet ?) IPv6 Jumbograms and cook corrupted packets.
>>
>> We must limit the IPv6 MTU to (65535 + 40) bytes in theory.
>>
>> Tested:
>>
>> ifconfig lo mtu 70000
>> netperf -H ::1
>>
>> Before patch : Throughput :   0.05 Mbits
>>
>> After patch : Throughput : 35484 Mbits
>>
>> Reported-by: Francois WELLENREITER <f.wellenreiter@gmail.com>
>> Signed-off-by: Eric Dumazet <edumazet@google.com>
> Acked-by: YOSHIFUJI Hideaki <yoshfuji@linux-ipv6.org>

Oops, you need to change the subject as well.
Otherwise, I am okay.

Thanks a lot.

--yoshfuji

Re: [PATCH] ipv6: Limit mtu to 65572 bytes

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Thu, 10 Apr 2014 20:20:27 -0700

> +/* We do not (yet ?) support IPv6 jumbograms (RFC 2675)
> + * Unlike IPv4, hdr->seg_len doesn't include the IPv6 header
> + */
> +#define IP6_MAX_MTU (0xFFFF + sizeof(struct ipv6hdr))

Hmmm, does this still match $Subj?

Re: [PATCH] ipv6: Limit mtu to 65572 bytes

[Eric Dumazet]

On Thu, 2014-04-10 at 23:30 -0400, David Miller wrote:
> From: Eric Dumazet <eric.dumazet@gmail.com>
> Date: Thu, 10 Apr 2014 20:20:27 -0700
> 
> > +/* We do not (yet ?) support IPv6 jumbograms (RFC 2675)
> > + * Unlike IPv4, hdr->seg_len doesn't include the IPv6 header
> > + */
> > +#define IP6_MAX_MTU (0xFFFF + sizeof(struct ipv6hdr))
> 
> Hmmm, does this still match $Subj?

Ugh, sorry, I had to run. I'll resend. 

[PATCH v3] ipv6: Limit mtu to 65575 bytes

[Eric Dumazet]

From: Eric Dumazet <edumazet@google.com>

Francois reported that setting big mtu on loopback device could prevent
tcp sessions making progress.

We do not support (yet ?) IPv6 Jumbograms and cook corrupted packets.

We must limit the IPv6 MTU to (65535 + 40) bytes in theory.

Tested:

ifconfig lo mtu 70000
netperf -H ::1

Before patch : Throughput :   0.05 Mbits

After patch : Throughput : 35484 Mbits

Reported-by: Francois WELLENREITER <f.wellenreiter@gmail.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: YOSHIFUJI Hideaki <yoshfuji@linux-ipv6.org>
---
v3: fix title... humpff...

 include/net/ip6_route.h |    5 +++++
 net/ipv6/route.c        |    5 +++--
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/include/net/ip6_route.h b/include/net/ip6_route.h
index 3c3bb184eb8f..6c4f5eac98e7 100644
--- a/include/net/ip6_route.h
+++ b/include/net/ip6_route.h
@@ -32,6 +32,11 @@ struct route_info {
 #define RT6_LOOKUP_F_SRCPREF_PUBLIC	0x00000010
 #define RT6_LOOKUP_F_SRCPREF_COA	0x00000020
 
+/* We do not (yet ?) support IPv6 jumbograms (RFC 2675)
+ * Unlike IPv4, hdr->seg_len doesn't include the IPv6 header
+ */
+#define IP6_MAX_MTU (0xFFFF + sizeof(struct ipv6hdr))
+
 /*
  * rt6_srcprefs2flags() and rt6_flags2srcprefs() translate
  * between IPV6_ADDR_PREFERENCES socket option values
diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index 5015c50a5ba7..5ea462eacd9f 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -1338,7 +1338,7 @@ static unsigned int ip6_mtu(const struct dst_entry *dst)
 	unsigned int mtu = dst_metric_raw(dst, RTAX_MTU);
 
 	if (mtu)
-		return mtu;
+		goto out;
 
 	mtu = IPV6_MIN_MTU;
 
@@ -1348,7 +1348,8 @@ static unsigned int ip6_mtu(const struct dst_entry *dst)
 		mtu = idev->cnf.mtu6;
 	rcu_read_unlock();
 
-	return mtu;
+out:
+	return min_t(unsigned int, mtu, IP6_MAX_MTU);
 }
 
 static struct dst_entry *icmp6_dst_gc_list;

Re: [PATCH v3] ipv6: Limit mtu to 65575 bytes

[Hannes Frederic Sowa]

On Thu, Apr 10, 2014 at 09:23:36PM -0700, Eric Dumazet wrote:
> From: Eric Dumazet <edumazet@google.com>
> 
> Francois reported that setting big mtu on loopback device could prevent
> tcp sessions making progress.
> 
> We do not support (yet ?) IPv6 Jumbograms and cook corrupted packets.
> 
> We must limit the IPv6 MTU to (65535 + 40) bytes in theory.
> 
> Tested:
> 
> ifconfig lo mtu 70000
> netperf -H ::1
> 
> Before patch : Throughput :   0.05 Mbits
> 
> After patch : Throughput : 35484 Mbits
> 
> Reported-by: Francois WELLENREITER <f.wellenreiter@gmail.com>
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> Acked-by: YOSHIFUJI Hideaki <yoshfuji@linux-ipv6.org>
> ---
> v3: fix title... humpff...

Acked-by: Hannes Frederic Sowa <hannes@stressinduktion.org>

Eric, the optimization for limiting the mtu in tcp case is still possible.
It is not that people don't care. ;)

Thanks,

  Hannes

Re: [PATCH v3] ipv6: Limit mtu to 65575 bytes

[Eric Dumazet]

On Fri, 2014-04-11 at 15:22 +0200, Hannes Frederic Sowa wrote:

> Acked-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
> 
> Eric, the optimization for limiting the mtu in tcp case is still possible.
> It is not that people don't care. ;)

I am not sure I want to add an additional test in TCP, as current
systems do not really have performance issues. I would 99.99% of the
time, MSS is already a multiple of 4.

I think we have more pressing problems ;)

Re: [PATCH v3] ipv6: Limit mtu to 65575 bytes

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Thu, 10 Apr 2014 21:23:36 -0700

> From: Eric Dumazet <edumazet@google.com>
> 
> Francois reported that setting big mtu on loopback device could prevent
> tcp sessions making progress.
> 
> We do not support (yet ?) IPv6 Jumbograms and cook corrupted packets.
> 
> We must limit the IPv6 MTU to (65535 + 40) bytes in theory.
> 
> Tested:
> 
> ifconfig lo mtu 70000
> netperf -H ::1
> 
> Before patch : Throughput :   0.05 Mbits
> 
> After patch : Throughput : 35484 Mbits
> 
> Reported-by: Francois WELLENREITER <f.wellenreiter@gmail.com>
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> Acked-by: YOSHIFUJI Hideaki <yoshfuji@linux-ipv6.org>

Applied and queued up for -stable, thanks Eric.

Re: [PATCH] ipv6: Limit mtu to 65572 bytes

[Hannes Frederic Sowa]

On Thu, Apr 10, 2014 at 06:42:10PM -0700, Eric Dumazet wrote:
> From: Eric Dumazet <edumazet@google.com>
> 
> Francois reported that setting big mtu on loopback device could prevent
> tcp sessions making progress.
> 
> We do not support (yet ?) IPv6 Jumbograms and cook corrupted packets.
> 
> We must limit the IPv6 MTU to (65535 + 40) bytes in theory.
> 
> In practice, its better to align to a multiple of 4 for optimal TCP
> performance.
> 
> Tested:
> 
> ifconfig lo mtu 70000
> netperf -H ::1
> 
> Before patch :
> 
> -> Throughput :   0.05 Mbits
> 
> After patch :
> 
> -> Throughput : 36624.24 Mbits
> 
> Reported-by: Francois WELLENREITER <f.wellenreiter@gmail.com>
> Signed-off-by: Eric Dumazet <edumazet@google.com>

Acked-by: Hannes Frederic Sowa <hannes@stressinduktion.org>

Sorry about my wrong statement earlier in this thread, you're right.

Btw. this patch fixes a flaw in the UDP output path, too, where we didn't
generate EMSGSIZE error correctly before in all cases. udp_sendmsg only checks
for INT_MAX - sizeof(udphdr) and defers further checking to ip6_append_data
which seems to be prepared for jumbograms. ;)

Very cool catch with the possible ipv4 performance improvment, too. ;)

Thanks,

  Hannes