[PATCH net-next 06/10] tcp: set MSS correctly for PMTU above 65535

[Mariusz Klimek <maklimek97@gmail.com>]

This patch fixes __tcp_mtu_to_mss to support MTUs above 65535. According to
RFC2675 and RFC6691, an advertised MSS of 65535 means "any MSS, rely only
on PMTU discovery". ip6_default_advmss already adheres to this behavior,
but __tcp_mtu_to_mss doesn't. __tcp_mtu_to_mss instead clamps the MSS to
65535.

Note that MTU probing also doesn't currently support PMTU > 65535 because
icsk_mtup.search_high is set to 65535. This commit doesn't add support for
PMTU > 65535 when MTU probing is enabled because it is not obvious what
icsk_mtup.search_high should be. A possible solution is to only set
icsk_mtup.search_low and to exponentially raise it until a probe becomes
too big (and then set icsk_mtup.search_high) but since this is a
non-trivial change, delegate it to a separate patch series.

Signed-off-by: Mariusz Klimek <maklimek97@gmail.com>
---
 net/ipv4/tcp_output.c | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index a66a3622006d..2d87b9cacd12 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -1990,19 +1990,30 @@ static inline int __tcp_mtu_to_mss(struct sock *sk, int pmtu)
 {
 	const struct tcp_sock *tp = tcp_sk(sk);
 	const struct inet_connection_sock *icsk = inet_csk(sk);
+	unsigned int ext_hdr_len;
 	int mss_now;
 
+	ext_hdr_len = icsk->icsk_ext_hdr_len;
+
+	/* Take into account added jumbogram HBH header. */
+	if (unlikely(pmtu - sizeof(struct ipv6hdr) > IPV6_MAXPLEN))
+		ext_hdr_len += sizeof(struct hop_jumbo_hdr);
+
 	/* Calculate base mss without TCP options:
 	   It is MMS_S - sizeof(tcphdr) of rfc1122
 	 */
 	mss_now = pmtu - icsk->icsk_af_ops->net_header_len - sizeof(struct tcphdr);
 
-	/* Clamp it (mss_clamp does not include tcp options) */
-	if (mss_now > tp->rx_opt.mss_clamp)
+	/* Clamp it (mss_clamp does not include tcp options).
+	 * An mss of 65535 means we should rely entirely on PMTU discovery
+	 * (RFC2675, RFC6691).
+	 */
+	if (mss_now > tp->rx_opt.mss_clamp &&
+	    likely(tp->rx_opt.mss_clamp < IPV6_MAXPLEN))
 		mss_now = tp->rx_opt.mss_clamp;
 
 	/* Now subtract optional transport overhead */
-	mss_now -= icsk->icsk_ext_hdr_len;
+	mss_now -= ext_hdr_len;
 
 	/* Then reserve room for full set of TCP options and 8 bytes of data */
 	mss_now = max(mss_now,
-- 
2.47.3