Strange problem with HTTPS POST (maybe) through router from Linux

Strange problem with HTTPS POST (maybe) through router from Linux

[Tvrtko Ursulin]

Hi all,

Not really sure it is appropriate for this mailing list but I have a hunch it 
could be netfilter related, or at least people who could know something about 
it are likely to visit this place. :)

I have a small home network behind a cheap ADSL router and a bizarre problem 
where I am not getting responses from some web sites in some situations. I 
suspect it is when a POST needs to go over the connection because it is always 
when I need to log in somewhere over HTTPS. Also it happens when a number of 
unrelated sites.

This only happens from Linux! Just today I tried four different distributions 
on two different machines and it is a total pattern. From Windows it all works 
fine. Also I tried three or four browsers on Linux and all behave exactly the 
same.

I had a look at the iptables setup on the router (see below) and it looks 
reasonable (at least short) to me, but it have been some years since I last 
used it so my knowledge is a bit thin here.

Or could it be that something has changed in recent kernels which could make 
the router unhappy and lose packets?

Anything more I could try to diagnose this? Any hints are appreciated!

Regards,

Tvrtko

 > iptables -L -n -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            224.0.0.0/3         
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:*** 
to:192.168.1.2:22 
DNAT       udp  --  0.0.0.0/0            192.168.1.1         udp dpt:53 
to:212.139.132.44    

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  192.168.1.0/24       0.0.0.0/0           

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
 > iptables -L -n       
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     2    --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state 
RELATED,ESTABLISHED 
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0           udp dpt:500               
ACCEPT     esp  --  0.0.0.0/0            0.0.0.0/0                                     
ACCEPT    !esp  --  0.0.0.0/0            0.0.0.0/0           MARK match 
0x10000000/0x10000000 
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:30005                    
LOG        tcp  --  0.0.0.0/0            0.0.0.0/0           tcp 
flags:0x17/0x02 limit: avg 6/hour burst 5 LOG flags 0 level 1 prefix `Intrusion -
> '                                                                                       
DROP       all  --  0.0.0.0/0            0.0.0.0/0                                                                    

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
DROP       all  --  0.0.0.0/0            224.0.0.22
DROP       all  --  0.0.0.0/0            224.0.0.2
ACCEPT     all  --  0.0.0.0/0            224.0.0.251
ACCEPT     all  --  0.0.0.0/0            239.255.255.250
TCPMSS     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp 
flags:0x06/0x02 TCPMSS set 1452
TCPMSS     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp 
flags:0x06/0x02 TCPMSS set 1452
ACCEPT     all  --  0.0.0.0/0            224.0.0.0/3
ACCEPT     tcp  --  0.0.0.0/0            192.168.1.2         tcp dpt:22
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state 
RELATED,ESTABLISHED
ACCEPT    !esp  --  0.0.0.0/0            0.0.0.0/0           MARK match 
0x10000000/0x10000000
LOG        tcp  --  0.0.0.0/0            0.0.0.0/0           tcp 
flags:0x17/0x02 limit: avg 6/hour burst 5 LOG flags 0 level 1 prefix `Intrusion -
> '
DROP       all  --  0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
DROP       all  --  0.0.0.0/0            239.255.255.250

Re: Strange problem with HTTPS POST (maybe) through router from Linux

[Jan Engelhardt]

On Tuesday 2010-06-08 23:03, Tvrtko Ursulin wrote:
>
>I have a small home network behind a cheap ADSL router and a bizarre problem 
>where I am not getting responses from some web sites in some situations.

 - Select one web page/URL where this has happened, preferably one
   that has a small output (like Apache's ingenious "It works!")
 - Is it 100% reproducible on every page reload?

If yes: check your firewall. Don't block ICMP errors.
Name the URL so that people can confirm.

If not: SACK/DSACK/FACK is broken in 2.6.18 (dunno when it was fixed,
but 2.6.25 looks good), and if either client or server make use
of it, things can hang once SACKs are exchanged.

Re: Strange problem with HTTPS POST (maybe) through router from Linux

[Thanasis]

on 06/09/2010 12:03 AM Tvrtko Ursulin wrote the following:
>
> Hi all,
>
> Not really sure it is appropriate for this mailing list but I have a hunch it 
> could be netfilter related, or at least people who could know something about 
> it are likely to visit this place. :)
>
> I have a small home network behind a cheap ADSL router and a bizarre problem 
> where I am not getting responses from some web sites in some situations. I 
> suspect it is when a POST needs to go over the connection because it is always 
> when I need to log in somewhere over HTTPS. Also it happens when a number of 
> unrelated sites.
>
> This only happens from Linux! Just today I tried four different distributions 
> on two different machines and it is a total pattern. From Windows it all works 
> fine. Also I tried three or four browsers on Linux and all behave exactly the 
> same.
>
> I had a look at the iptables setup on the router (see below) and it looks 
> reasonable (at least short) to me, but it have been some years since I last 
> used it so my knowledge is a bit thin here.
>
> Or could it be that something has changed in recent kernels which could make 
> the router unhappy and lose packets?
>
> Anything more I could try to diagnose this? Any hints are appreciated!
>
> Regards,
>
> Tvrtko
This usually happens when the firewall(s) does not allow path MTU
discovery because it is configured to filter/drop ICMP.
Take a look here:
http://www.znep.com/~marcs/mtu/
http://www.netheaven.com/pmtu.html
Try setting the MTU to a small value like:
ifconfig eth0 mtu 1000
and if that solves the problem, then fix your firewall(s)

Re: Strange problem with HTTPS POST (maybe) through router from Linux

[Tvrtko Ursulin]

On Tuesday 08 Jun 2010 22:46:33 Thanasis wrote:
> on 06/09/2010 12:03 AM Tvrtko Ursulin wrote the following:
> > Hi all,
> >
> > Not really sure it is appropriate for this mailing list but I have a
> > hunch it could be netfilter related, or at least people who could know
> > something about it are likely to visit this place. :)
> >
> > I have a small home network behind a cheap ADSL router and a bizarre
> > problem where I am not getting responses from some web sites in some
> > situations. I suspect it is when a POST needs to go over the connection
> > because it is always when I need to log in somewhere over HTTPS. Also it
> > happens when a number of unrelated sites.
> >
> > This only happens from Linux! Just today I tried four different
> > distributions on two different machines and it is a total pattern. From
> > Windows it all works fine. Also I tried three or four browsers on Linux
> > and all behave exactly the same.
> >
> > I had a look at the iptables setup on the router (see below) and it looks
> > reasonable (at least short) to me, but it have been some years since I
> > last used it so my knowledge is a bit thin here.
> >
> > Or could it be that something has changed in recent kernels which could
> > make the router unhappy and lose packets?
> >
> > Anything more I could try to diagnose this? Any hints are appreciated!
> >
> > Regards,
> >
> > Tvrtko
> 
> This usually happens when the firewall(s) does not allow path MTU
> discovery because it is configured to filter/drop ICMP.
> Take a look here:
> http://www.znep.com/~marcs/mtu/
> http://www.netheaven.com/pmtu.html
> Try setting the MTU to a small value like:
> ifconfig eth0 mtu 1000
> and if that solves the problem, then fix your firewall(s)

You are right, MTU of 1000 indeed helps.

I have read both links and thought I understand what is happening there, but 
obviosuly not because I was expecting "echo 1 > 
/proc/sys/net/ipv4/ip_no_pmtu_disc" to be another workaround but it is not in 
practice.

To fix the firewall I will have to talk to the manufacturer, there is currently 
no way that I can see to enable ICMP.

Tvrtko

 

Re: Strange problem with HTTPS POST (maybe) through router from Linux

[Tvrtko Ursulin]

On Tuesday 08 Jun 2010 22:36:28 Jan Engelhardt wrote:
> On Tuesday 2010-06-08 23:03, Tvrtko Ursulin wrote:
> >I have a small home network behind a cheap ADSL router and a bizarre
> > problem where I am not getting responses from some web sites in some
> > situations.
> 
>  - Select one web page/URL where this has happened, preferably one
>    that has a small output (like Apache's ingenious "It works!")
>  - Is it 100% reproducible on every page reload?
> 
> If yes: check your firewall. Don't block ICMP errors.
> Name the URL so that people can confirm.

Firewall rules do not mention ICMP and I can ping outside world so I guess 
that means it is not blocked?

There is this:
TCPMSS     tcp  --  anywhere             anywhere            tcp 
flags:SYN,RST/SYN TCPMSS set 1452 
TCPMSS     tcp  --  anywhere             anywhere            tcp 
flags:SYN,RST/SYN TCPMSS set 1452 

In the forward chain at the firewall which is the only thing which stands out. 
If I understand it correctly it sets MTU to 1452 for outgoing packets. Why it 
is specified twice I have no idea. 

I tried doing "ifconfig eth0 mtu 1452" on the client and that did not help. I 
had to go all the way down to 1400 for one site and that also worked for the 
other one. 

Site I was testing with is http://www.tesco.com/superstore/ . This page does 
not load unless MTU 1400 is set on the client.
 
> If not: SACK/DSACK/FACK is broken in 2.6.18 (dunno when it was fixed,
> but 2.6.25 looks good), and if either client or server make use
> of it, things can hang once SACKs are exchanged.
 
My clients are 2.6.31 - 2.6.34, but the router/firewall is running 2.6.21.5. 

Tvrtko

Re: Strange problem with HTTPS POST (maybe) through router from Linux

[Jan Engelhardt]

On Wednesday 2010-06-09 10:00, Tvrtko Ursulin wrote:
>
>Firewall rules do not mention ICMP and I can ping outside world so I guess 
>that means it is not blocked?

ICMP is not just ping, there is more like PMTUD and others.
If PMTUD works on your side, you don't need TCPMSS.

>I tried doing "ifconfig eth0 mtu 1452" on the client and that did not help.

Just for the record, do _not_ use ifconfig, but ip in the future.

>Site I was testing with is http://www.tesco.com/superstore/ . This page does 
>not load unless MTU 1400 is set on the client.

>> If not: SACK/DSACK/FACK is broken in 2.6.18 (dunno when it was fixed,
>> but 2.6.25 looks good), and if either client or server make use
>> of it, things can hang once SACKs are exchanged.
> 
>My clients are 2.6.31 - 2.6.34, but the router/firewall is running 2.6.21.5. 

Well try deactivating sack/dsack/fack then (that's in sysctl).

Re: Strange problem with HTTPS POST (maybe) through router from Linux

[Tvrtko Ursulin]

On 9 June 2010 11:02, Jan Engelhardt <jengelh@medozas.de> wrote:
>
> On Wednesday 2010-06-09 10:00, Tvrtko Ursulin wrote:
>>
>>Firewall rules do not mention ICMP and I can ping outside world so I guess
>>that means it is not blocked?
>
> ICMP is not just ping, there is more like PMTUD and others.
> If PMTUD works on your side, you don't need TCPMSS.

Is there a way to check that across the link? If my router has no ICMP
rules in iptables than should I suspect the ISP?

>>I tried doing "ifconfig eth0 mtu 1452" on the client and that did not help.
>
> Just for the record, do _not_ use ifconfig, but ip in the future.

Ok, will look at it.

>>Site I was testing with is http://www.tesco.com/superstore/ . This page does
>>not load unless MTU 1400 is set on the client.
>
>>> If not: SACK/DSACK/FACK is broken in 2.6.18 (dunno when it was fixed,
>>> but 2.6.25 looks good), and if either client or server make use
>>> of it, things can hang once SACKs are exchanged.
>>
>>My clients are 2.6.31 - 2.6.34, but the router/firewall is running 2.6.21.5.
>
> Well try deactivating sack/dsack/fack then (that's in sysctl).

On the router? Will try in the evening.

What is puzzling me is that Windows clients work fine, even though
they also have MTU set to 1500. All I am reading about his issues
cannot explain this to me.

Tvrtko

Re: Strange problem with HTTPS POST (maybe) through router from Linux

[Jan Engelhardt]

On Wednesday 2010-06-09 15:41, Tvrtko Ursulin wrote:
>>
>> ICMP is not just ping, there is more like PMTUD and others.
>> If PMTUD works on your side, you don't need TCPMSS.
>
>Is there a way to check that across the link? If my router has no ICMP
>rules in iptables than should I suspect the ISP?

	ping -M do -s 9000 target

From <router> icmp_seq=1 Frag needed and DF set (mtu = 1412)

Then you retry with

	ping -M do -s $[1412-28] target

and do that as long as Frag needed is outputted.
That's basically manual PMTUD and allows you to see where
MTU reduction along the route occurs.

>>>> If not: SACK/DSACK/FACK is broken in 2.6.18 (dunno when it was fixed,
>>>> but 2.6.25 looks good), and if either client or server make use
>>>> of it, things can hang once SACKs are exchanged.
>>>
>>>My clients are 2.6.31 - 2.6.34, but the router/firewall is running 2.6.21.5.
>>
>> Well try deactivating sack/dsack/fack then (that's in sysctl).
>
>On the router? Will try in the evening.

No, on at least one of the end host(s).
(Since you have control over your client, that shouldn't be a problem.)

>What is puzzling me is that Windows clients work fine, even though
>they also have MTU set to 1500. All I am reading about his issues
>cannot explain this to me.

That's why I suspected SACK issues. (Because SACK is too smart a
technology to be usable in Windows ;-)

Re: Strange problem with HTTPS POST (maybe) through router from Linux

[Tvrtko Ursulin]

On Wednesday 09 Jun 2010 16:02:19 Jan Engelhardt wrote:
> On Wednesday 2010-06-09 15:41, Tvrtko Ursulin wrote:
> >> ICMP is not just ping, there is more like PMTUD and others.
> >> If PMTUD works on your side, you don't need TCPMSS.
> >
> >Is there a way to check that across the link? If my router has no ICMP
> >rules in iptables than should I suspect the ISP?
> 
> 	ping -M do -s 9000 target
> 
> From <router> icmp_seq=1 Frag needed and DF set (mtu = 1412)
> 
> Then you retry with
> 
> 	ping -M do -s $[1412-28] target
> 
> and do that as long as Frag needed is outputted.
> That's basically manual PMTUD and allows you to see where
> MTU reduction along the route occurs.

Starting from mtu=1500 and testing with "ping -M do -s $[$mtu-28] 
secure.tesco.com
", first value which does not need fragmentation is 1492 which is what the MTU 
is set to the PPPoA interface on the router. Would that look like there is no 
problem?

Sidenote - if I change the  PPPoA MTU on the router to 1462, which is 
allegedly optimal for ATM, then the above ping test starts to pass only with 
mtu=1462.

Does this make any sense? secure.tesco.com is the host browsers are waiting a 
response from forever.. Am I misunderstanding the results of the ping test?

> >>>> If not: SACK/DSACK/FACK is broken in 2.6.18 (dunno when it was fixed,
> >>>> but 2.6.25 looks good), and if either client or server make use
> >>>> of it, things can hang once SACKs are exchanged.
> >>>
> >>>My clients are 2.6.31 - 2.6.34, but the router/firewall is running
> >>> 2.6.21.5.
> >>
> >> Well try deactivating sack/dsack/fack then (that's in sysctl).
> >
> >On the router? Will try in the evening.
> 
> No, on at least one of the end host(s).
> (Since you have control over your client, that shouldn't be a problem.)
> 
> >What is puzzling me is that Windows clients work fine, even though
> >they also have MTU set to 1500. All I am reading about his issues
> >cannot explain this to me.
> 
> That's why I suspected SACK issues. (Because SACK is too smart a
> technology to be usable in Windows ;-)

Unfortunately disabling all three on the client did not help. Plot thickens. 
:)

Thanks for your help so far - I already learned a lot, including that I know 
little. :)

Tvrtko
 

Re: Strange problem with HTTPS POST (maybe) through router from Linux

[Tvrtko Ursulin]

On Wednesday 09 Jun 2010 18:51:27 Tvrtko Ursulin wrote:
> On Wednesday 09 Jun 2010 16:02:19 Jan Engelhardt wrote:
> > On Wednesday 2010-06-09 15:41, Tvrtko Ursulin wrote:
> > >> ICMP is not just ping, there is more like PMTUD and others.
> > >> If PMTUD works on your side, you don't need TCPMSS.
> > >
> > >Is there a way to check that across the link? If my router has no ICMP
> > >rules in iptables than should I suspect the ISP?
> >
> > 	ping -M do -s 9000 target
> >
> > From <router> icmp_seq=1 Frag needed and DF set (mtu = 1412)
> >
> > Then you retry with
> >
> > 	ping -M do -s $[1412-28] target
> >
> > and do that as long as Frag needed is outputted.
> > That's basically manual PMTUD and allows you to see where
> > MTU reduction along the route occurs.
> 
> Starting from mtu=1500 and testing with "ping -M do -s $[$mtu-28]
> secure.tesco.com
> ", first value which does not need fragmentation is 1492 which is what the
>  MTU is set to the PPPoA interface on the router. Would that look like
>  there is no problem?
> 
> Sidenote - if I change the  PPPoA MTU on the router to 1462, which is
> allegedly optimal for ATM, then the above ping test starts to pass only
>  with mtu=1462.
> 
> Does this make any sense? secure.tesco.com is the host browsers are waiting
>  a response from forever.. Am I misunderstanding the results of the ping
>  test?

Yes I was misunderstanding the results, or to better say I was not thinking! 
:) Point is for "Frag needed" message to go away _AND_ replies starting to 
come back. 

So I tracerouted it and found which hop stops replying to this specific pings. 
Guess I can either find out whose router is it and see if they want to fix it, 
or I can setup an explicit route with smaller MTU for problematic web sites I 
care about.

Well this was one pretty educational exercise, thanks again for your help!

Tvrtko

Re: Strange problem with HTTPS POST (maybe) through router from Linux

[Tvrtko Ursulin]

On Wednesday 09 Jun 2010 23:03:32 Tvrtko Ursulin wrote:
> Yes I was misunderstanding the results, or to better say I was not
>  thinking!
> 
> :) Point is for "Frag needed" message to go away _AND_ replies starting to
> 
> come back.
> 
> So I tracerouted it and found which hop stops replying to this specific
>  pings. Guess I can either find out whose router is it and see if they want
>  to fix it, or I can setup an explicit route with smaller MTU for
>  problematic web sites I care about.
> 
> Well this was one pretty educational exercise, thanks again for your help!

And I forgot this last point, this does not explain why Windows clients are 
not affected.

Tvrtko