mss to pmtu clamping partially broken?

mss to pmtu clamping partially broken?

[Andreas Steinmetz]

There seems to be a problem with mss to pmtu clamping for incoming syn
packets on reply to an outgoing connection on a ppp interface. The mss
of the outgoing syn packets is always always clamped to the pmtu, I did
check this with a target host I do have access to. The incoming syn
reply to such a packet, however, is mss clamped only sometimes and this
seems to depend on the DSL line used.

The kernels tested were 2.6.20.1, 2.6.20.3 and 2.6.22rc6.

Test setup: Two DSL lines, otherwise identical setup (same masquerading
linux gateway, same DSL account, same DSL modem, same DSL line provider,
same target host, request from and tcpdump on the same client).

Linux Client<->Masquerading Linux Gateway<->DSL Modem<->DSL Line<->...

DSL line 1, working:

22:26:39.319281 IP (tos 0x0, ttl  64, id 22377, offset 0, flags [DF],
length: 48
) 192.168.0.253.1164 > 64.34.165.170.80: S [tcp sum ok]
1465827859:1465827859(0)
 win 5840 <mss 1460,nop,nop,sackOK>
22:26:39.459314 IP (tos 0x0, ttl  51, id 0, offset 0, flags [DF],
length: 48) 64
.34.165.170.80 > 192.168.0.253.1164: S [tcp sum ok]
3667852791:3667852791(0) ack
 1465827860 win 5840 <mss 1452,nop,nop,sackOK>

The tcpdump on the client shows that the mss of the incoming syn reply
packet is clamped to the ppp interface mtu.

DSL line 2, not working:

22:03:57.725998 IP (tos 0x0, ttl  64, id 55984, offset 0, flags [DF],
length: 48
) 192.168.0.253.1600 > 64.34.165.170.80: S [tcp sum ok]
36968258:36968258(0) win
 5840 <mss 1460,nop,nop,sackOK>
22:03:57.866966 IP (tos 0x0, ttl  51, id 0, offset 0, flags [DF],
length: 48) 64
.34.165.170.80 > 192.168.0.253.1600: S [tcp sum ok]
2226854208:2226854208(0) ack
 36968259 win 5840 <mss 1460,nop,nop,sackOK>

The tcpdump on the client shows that the mss of the incoming syn reply
packet is *NOT* clamped to the ppp interface mtu.

-- 
Andreas Steinmetz                       SPAMmers use robotrap@domdv.de

Re: mss to pmtu clamping partially broken?

[Patrick McHardy]

Andreas Steinmetz wrote:
> There seems to be a problem with mss to pmtu clamping for incoming syn
> packets on reply to an outgoing connection on a ppp interface. The mss
> of the outgoing syn packets is always always clamped to the pmtu, I did
> check this with a target host I do have access to. The incoming syn
> reply to such a packet, however, is mss clamped only sometimes and this
> seems to depend on the DSL line used.
> 
> The kernels tested were 2.6.20.1, 2.6.20.3 and 2.6.22rc6.
> 
> Test setup: Two DSL lines, otherwise identical setup (same masquerading
> linux gateway, same DSL account, same DSL modem, same DSL line provider,
> same target host, request from and tcpdump on the same client).
> 
> Linux Client<->Masquerading Linux Gateway<->DSL Modem<->DSL Line<->...
> 
> DSL line 1, working:
> 
> 22:26:39.319281 IP (tos 0x0, ttl  64, id 22377, offset 0, flags [DF],
> length: 48
> ) 192.168.0.253.1164 > 64.34.165.170.80: S [tcp sum ok]
> 1465827859:1465827859(0)
>  win 5840 <mss 1460,nop,nop,sackOK>
> 22:26:39.459314 IP (tos 0x0, ttl  51, id 0, offset 0, flags [DF],
> length: 48) 64
> .34.165.170.80 > 192.168.0.253.1164: S [tcp sum ok]
> 3667852791:3667852791(0) ack
>  1465827860 win 5840 <mss 1452,nop,nop,sackOK>
> 
> The tcpdump on the client shows that the mss of the incoming syn reply
> packet is clamped to the ppp interface mtu.
> 
> DSL line 2, not working:
> 
> 22:03:57.725998 IP (tos 0x0, ttl  64, id 55984, offset 0, flags [DF],
> length: 48
> ) 192.168.0.253.1600 > 64.34.165.170.80: S [tcp sum ok]
> 36968258:36968258(0) win
>  5840 <mss 1460,nop,nop,sackOK>
> 22:03:57.866966 IP (tos 0x0, ttl  51, id 0, offset 0, flags [DF],
> length: 48) 64
> .34.165.170.80 > 192.168.0.253.1600: S [tcp sum ok]
> 2226854208:2226854208(0) ack
>  36968259 win 5840 <mss 1460,nop,nop,sackOK>
> 
> The tcpdump on the client shows that the mss of the incoming syn reply
> packet is *NOT* clamped to the ppp interface mtu.


You forgot to mention *how* you're clamping the MSS. Using
TCPMSS? Do you have a rule for incoming packets?

Re: mss to pmtu clamping partially broken?

[Andreas Steinmetz]

Patrick McHardy wrote:
> Andreas Steinmetz wrote:
>> There seems to be a problem with mss to pmtu clamping for incoming syn
>> packets on reply to an outgoing connection on a ppp interface. The mss
>> of the outgoing syn packets is always always clamped to the pmtu, I did
>> check this with a target host I do have access to. The incoming syn
>> reply to such a packet, however, is mss clamped only sometimes and this
>> seems to depend on the DSL line used.
>>
>> The kernels tested were 2.6.20.1, 2.6.20.3 and 2.6.22rc6.
>>
>> Test setup: Two DSL lines, otherwise identical setup (same masquerading
>> linux gateway, same DSL account, same DSL modem, same DSL line provider,
>> same target host, request from and tcpdump on the same client).
>>
>> Linux Client<->Masquerading Linux Gateway<->DSL Modem<->DSL Line<->...
>>
>> DSL line 1, working:
>>
>> 22:26:39.319281 IP (tos 0x0, ttl  64, id 22377, offset 0, flags [DF],
>> length: 48
>> ) 192.168.0.253.1164 > 64.34.165.170.80: S [tcp sum ok]
>> 1465827859:1465827859(0)
>>  win 5840 <mss 1460,nop,nop,sackOK>
>> 22:26:39.459314 IP (tos 0x0, ttl  51, id 0, offset 0, flags [DF],
>> length: 48) 64
>> .34.165.170.80 > 192.168.0.253.1164: S [tcp sum ok]
>> 3667852791:3667852791(0) ack
>>  1465827860 win 5840 <mss 1452,nop,nop,sackOK>
>>
>> The tcpdump on the client shows that the mss of the incoming syn reply
>> packet is clamped to the ppp interface mtu.
>>
>> DSL line 2, not working:
>>
>> 22:03:57.725998 IP (tos 0x0, ttl  64, id 55984, offset 0, flags [DF],
>> length: 48
>> ) 192.168.0.253.1600 > 64.34.165.170.80: S [tcp sum ok]
>> 36968258:36968258(0) win
>>  5840 <mss 1460,nop,nop,sackOK>
>> 22:03:57.866966 IP (tos 0x0, ttl  51, id 0, offset 0, flags [DF],
>> length: 48) 64
>> .34.165.170.80 > 192.168.0.253.1600: S [tcp sum ok]
>> 2226854208:2226854208(0) ack
>>  36968259 win 5840 <mss 1460,nop,nop,sackOK>
>>
>> The tcpdump on the client shows that the mss of the incoming syn reply
>> packet is *NOT* clamped to the ppp interface mtu.
> 
> 
> You forgot to mention *how* you're clamping the MSS. Using
> TCPMSS? Do you have a rule for incoming packets?
> 

The relevant iptables commands I do use for masquerading and clamping are:

iptables -t nat -A POSTROUTING -o ppp0 -j MASQUERADE
iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS \
	--clamp-mss-to-pmtu

-- 
Andreas Steinmetz                       SPAMmers use robotrap@domdv.de

Re: mss to pmtu clamping partially broken?

[Patrick McHardy]

Andreas Steinmetz wrote:
> Patrick McHardy wrote:
> 
>>Andreas Steinmetz wrote:
>>
>>>[...]
>>>The tcpdump on the client shows that the mss of the incoming syn reply
>>>packet is *NOT* clamped to the ppp interface mtu.
>>
>>
>>You forgot to mention *how* you're clamping the MSS. Using
>>TCPMSS? Do you have a rule for incoming packets?
>>
> 
> 
> The relevant iptables commands I do use for masquerading and clamping are:
> 
> iptables -t nat -A POSTROUTING -o ppp0 -j MASQUERADE
> iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS \
> 	--clamp-mss-to-pmtu


Two things here:

- tcpdumps on ppp0 will show unclamped packets since they haven't
been forwarded yet

- assuming you have ethernet internally, the PMTU from your router
to the internal hosts is 1500, so it won't do any clamping.

Does that explain it?

A useful thing for TCPMSS for routers would be to clamp to the
minimum of the PMTU of both directions. But thats not supported
so far.

Re: mss to pmtu clamping partially broken?

[Andreas Steinmetz]

Patrick McHardy wrote:
> Andreas Steinmetz wrote:
>> Patrick McHardy wrote:
>>
>>> Andreas Steinmetz wrote:
>>>
>>>> [...]
>>>> The tcpdump on the client shows that the mss of the incoming syn reply
>>>> packet is *NOT* clamped to the ppp interface mtu.
>>>
>>> You forgot to mention *how* you're clamping the MSS. Using
>>> TCPMSS? Do you have a rule for incoming packets?
>>>
>>
>> The relevant iptables commands I do use for masquerading and clamping are:
>>
>> iptables -t nat -A POSTROUTING -o ppp0 -j MASQUERADE
>> iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS \
>> 	--clamp-mss-to-pmtu
> 
> 
> Two things here:
> 
> - tcpdumps on ppp0 will show unclamped packets since they haven't
> been forwarded yet
> 

That is true, I know this.

> - assuming you have ethernet internally, the PMTU from your router
> to the internal hosts is 1500, so it won't do any clamping.
> 

Yep, internal PMTU is 1500, still the incoming packets are clamped to
1452 on the one line and not clamped on the other.

> Does that explain it?
> 
> A useful thing for TCPMSS for routers would be to clamp to the
> minimum of the PMTU of both directions. But thats not supported
> so far.
> 

I wonder, as somteimes it gets clamped. If it would never have been
clamped I wouldn't have asked.

-- 
Andreas Steinmetz                       SPAMmers use robotrap@domdv.de

Re: mss to pmtu clamping partially broken?

[Patrick McHardy]

Andreas Steinmetz wrote:
> Patrick McHardy wrote:
> 
>>- assuming you have ethernet internally, the PMTU from your router
>>to the internal hosts is 1500, so it won't do any clamping.
>>
> 
> 
> Yep, internal PMTU is 1500, still the incoming packets are clamped to
> 1452 on the one line and not clamped on the other.
> 
> 
>>Does that explain it?
>>
>>A useful thing for TCPMSS for routers would be to clamp to the
>>minimum of the PMTU of both directions. But thats not supported
>>so far.
>>
> 
> 
> I wonder, as somteimes it gets clamped. If it would never have been
> clamped I wouldn't have asked.


Its possible that one of your ISPs is doing clamping. You could
check on ppp0 if thats the case. Or maybe for some reason the
PMTU value for the internal host is smaller than 1500. You can
check that by doing "ip route get <internal host>".

Re: mss to pmtu clamping partially broken?

[Andreas Steinmetz]

Patrick McHardy wrote:
> Andreas Steinmetz wrote:
>> Patrick McHardy wrote:
>>
>>> - assuming you have ethernet internally, the PMTU from your router
>>> to the internal hosts is 1500, so it won't do any clamping.
>>>
>>
>> Yep, internal PMTU is 1500, still the incoming packets are clamped to
>> 1452 on the one line and not clamped on the other.
>>
>>
>>> Does that explain it?
>>>
>>> A useful thing for TCPMSS for routers would be to clamp to the
>>> minimum of the PMTU of both directions. But thats not supported
>>> so far.
>>>
>>
>> I wonder, as somteimes it gets clamped. If it would never have been
>> clamped I wouldn't have asked.
> 
> 
> Its possible that one of your ISPs is doing clamping. You could

This would be fun as it is the same ISP for both lines. I'll check next
week as the lines are located 40km away.

> check on ppp0 if thats the case. Or maybe for some reason the
> PMTU value for the internal host is smaller than 1500. You can
> check that by doing "ip route get <internal host>".
> 

No. Unmodified internal network in both test cases.

> 


-- 
Andreas Steinmetz                       SPAMmers use robotrap@domdv.de

Re: mss to pmtu clamping partially broken?

[Jan Engelhardt]

On Jun 29 2007 13:09, Andreas Steinmetz wrote:
>
>There seems to be a problem with mss to pmtu clamping for incoming syn
>packets on reply to an outgoing connection on a ppp interface. The mss
>of the outgoing syn packets is always always clamped to the pmtu, I did
>check this with a target host I do have access to. The incoming syn
>reply to such a packet, however, is mss clamped only sometimes and this
>seems to depend on the DSL line used.

Do you really need clamping? It's a hack, since TCP should do MSS negotiation
itself. (Of course it may happen that some routers are broken.) But usually not
for incoming packets.



	Jan
-- 

Re: mss to pmtu clamping partially broken?

[Andreas Steinmetz]

Patrick McHardy wrote:
> Its possible that one of your ISPs is doing clamping. You could
> check on ppp0 if thats the case. Or maybe for some reason the
> PMTU value for the internal host is smaller than 1500. You can
> check that by doing "ip route get <internal host>".
> 
> 

Oh well, thew fun with ISPs. Same provider, clamping on one line but not
the other. This is fun :-(

-- 
Andreas Steinmetz                       SPAMmers use robotrap@domdv.de

Re: mss to pmtu clamping partially broken?

[Andreas Steinmetz]

Jan Engelhardt wrote:
> Do you really need clamping? It's a hack, since TCP should do MSS negotiation
> itself. (Of course it may happen that some routers are broken.) But usually not
> for incoming packets.

You never know when you hit ICMP blackholes, broken routers and other
evil things. Better safe than sorry so clamping is the way to go for me.

-- 
Andreas Steinmetz                       SPAMmers use robotrap@domdv.de

Re: mss to pmtu clamping partially broken?

[Phil Dibowitz]

[-- Attachment #1: Type: text/plain, Size: 1164 bytes --]

On Mon, Jul 02, 2007 at 07:04:12PM +0200, Andreas Steinmetz wrote:
> Jan Engelhardt wrote:
> > Do you really need clamping? It's a hack, since TCP should do MSS negotiation
> > itself. (Of course it may happen that some routers are broken.) But usually not
> > for incoming packets.
> 
> You never know when you hit ICMP blackholes, broken routers and other
> evil things. Better safe than sorry so clamping is the way to go for me.

I encourage you to report PMTUD Blackholes to the MSS Initiative at
http://www.phildev.net/mss/

We'll notify them, and if we can't get them to fix it, blacklist them. We have
more fixed sites than blacklisted sites, so it's at least somewhat successful.

-- 
Phil Dibowitz                             phil@ipom.com
Open Source software and tech docs        Insanity Palace of Metallica
http://www.phildev.net/                   http://www.ipom.com/

"Never write it in C if you can do it in 'awk';
 Never do it in 'awk' if 'sed' can handle it;
 Never use 'sed' when 'tr' can do the job;
 Never invoke 'tr' when 'cat' is sufficient;
 Avoid using 'cat' whenever possible" -- Taylor's Laws of Programming


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: mss to pmtu clamping partially broken?

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 701 bytes --]



On Mon, 2 Jul 2007, Phil Dibowitz wrote:

> On Mon, Jul 02, 2007 at 07:04:12PM +0200, Andreas Steinmetz wrote:
>> Jan Engelhardt wrote:
>>> Do you really need clamping? It's a hack, since TCP should do MSS negotiation
>>> itself. (Of course it may happen that some routers are broken.) But usually not
>>> for incoming packets.
>>
>> You never know when you hit ICMP blackholes, broken routers and other
>> evil things. Better safe than sorry so clamping is the way to go for me.
>
> I encourage you to report PMTUD Blackholes to the MSS Initiative at
> http://www.phildev.net/mss/

Any chances for similar initiative for "SACK vandals"? ;)

Best regards,

 				Krzysztof Olędzki

Re: mss to pmtu clamping partially broken?

[Phil Dibowitz]

[-- Attachment #1: Type: text/plain, Size: 1624 bytes --]

On Mon, Jul 02, 2007 at 09:16:57PM +0200, Krzysztof Oledzki wrote:
>
>
> On Mon, 2 Jul 2007, Phil Dibowitz wrote:
>
>> On Mon, Jul 02, 2007 at 07:04:12PM +0200, Andreas Steinmetz wrote:
>>> Jan Engelhardt wrote:
>>>> Do you really need clamping? It's a hack, since TCP should do MSS 
>>>> negotiation
>>>> itself. (Of course it may happen that some routers are broken.) But 
>>>> usually not
>>>> for incoming packets.
>>>
>>> You never know when you hit ICMP blackholes, broken routers and other
>>> evil things. Better safe than sorry so clamping is the way to go for me.
>>
>> I encourage you to report PMTUD Blackholes to the MSS Initiative at
>> http://www.phildev.net/mss/
>
> Any chances for similar initiative for "SACK vandals"? ;)

There's already a counterpart for ECN blackholes, so I'm not opposed to it.
However, keeping up with new reports, re-testing existing offenders, etc.
takes up a good chunk of time, so I don't have the time to do it myself. I'm
happy to reference such a site, however.

Though - I'm not familiar with the problem of SACK vandals either. There
appears to be a thread on here, I'll go read it...

-- 
Phil Dibowitz                             phil@ipom.com
Open Source software and tech docs        Insanity Palace of Metallica
http://www.phildev.net/                   http://www.ipom.com/

"Never write it in C if you can do it in 'awk';
 Never do it in 'awk' if 'sed' can handle it;
 Never use 'sed' when 'tr' can do the job;
 Never invoke 'tr' when 'cat' is sufficient;
 Avoid using 'cat' whenever possible" -- Taylor's Laws of Programming


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: mss to pmtu clamping partially broken?

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1262 bytes --]



On Mon, 2 Jul 2007, Phil Dibowitz wrote:

> On Mon, Jul 02, 2007 at 09:16:57PM +0200, Krzysztof Oledzki wrote:
>>
>>
>> On Mon, 2 Jul 2007, Phil Dibowitz wrote:
>>
>>> On Mon, Jul 02, 2007 at 07:04:12PM +0200, Andreas Steinmetz wrote:
>>>> Jan Engelhardt wrote:
>>>>> Do you really need clamping? It's a hack, since TCP should do MSS
>>>>> negotiation
>>>>> itself. (Of course it may happen that some routers are broken.) But
>>>>> usually not
>>>>> for incoming packets.
>>>>
>>>> You never know when you hit ICMP blackholes, broken routers and other
>>>> evil things. Better safe than sorry so clamping is the way to go for me.
>>>
>>> I encourage you to report PMTUD Blackholes to the MSS Initiative at
>>> http://www.phildev.net/mss/
>>
>> Any chances for similar initiative for "SACK vandals"? ;)
>
> There's already a counterpart for ECN blackholes, so I'm not opposed to it.
> However, keeping up with new reports, re-testing existing offenders, etc.
> takes up a good chunk of time, so I don't have the time to do it myself. I'm
> happy to reference such a site, however.

Indeed and it seems there are more important issues, like similar window 
scaling problem for example. :(

Best regards,

 				Krzysztof Olędzki