Re: Linux forwarding Win XP hosts VERY slowly

Re: Linux forwarding Win XP hosts VERY slowly

[itd.nam]

----- Original Message -----
From: Alistair Tonner <Alistair@nerdnet.ca>
Date: Sunday, May 1, 2005 5:10 am
Subject: Re: Linux forwarding Win XP hosts VERY slowly
> On April 30, 2005 01:41 pm, Dave Cinege wrote: 
> > I've built an advanced rotuign appliance, and I'm having 2 
> outstanding> problems, that I'm being to think are related to the 
> linux ip/netfilter 
> > stack, choking on XP traffic (possiblity XP-SP2) hosts that are 
> on the LAN. 
> > I'm running 2.4.30 at the moment. 
> > 
> > The 2 problems I'm seeing: 
> > 
> > 1) Forwarded traffic (most notably web) is VERY slow with XP 
> clients.> 
> > Example: Saw this last 2 nights ago: Appliance has a linksys 
> Wifi bridge 
> > attached to a NIC. Customer browses through the appliance to the 
> Linksys> config page. It moves like molasses. He browse to the 
> local Zope hosted 
> > made page. Slow as hell. I unplug his machine, and plug my linux 
> laptop> into same switch port. Linksys and Zope pages load adn 
> reload instantly. 
> > Plug his machine in....slow again. 
> > 
> > 2) Zope serves user interface pages for the appliance. Zope has been 
> > locking solid for no apparent reason, but only when and Windows 
> host is 
> > attached. The trick is SOME windows machine don't seem to cause 
> a problem. 
> > Example: 
> > I worked with a unit for 3 days using a customers XP desktop. 
> Not a hiccup. 
> > My partner came in and attached to the network and starting 
> connect to our 
> > appliance with his XP laptop. Within 15 minutes Zope was hung. 
> > 
> 
> I would strongly suspect the XP box has a b0rken TCP stack 
> happening. One 
> thing that some windows systems will do is flat out ignore the TCP 
> MTU/window 
> size settings on the network, especially if you've a) set them up 
> for modem 
> dialup b) installed anything that is supposed to automatically 
> improve your 
> internet speed c) hard wired (EVER) the MTU settings. I had a 
> win2k box that 
> had this sort of issue once, and even though it was at that time 
> set to use 
> default settings for MTU/max recieve window and the like, I had to 
> completely 
> uninstall the tcp stack, the network card driver, all modem bits 
> etc and 
> reinstall em from scratch to get it to behave normally. For the 
> record, 
> Ethereal dumps of the communications CLEARLY show that the windows 
> box is 
> using bad MTU settings and bad TCP window sizes, if this is the case. 
> 
> I *still* believe that there are settings left on that box (still 
> in use these 
> days) that cause issues periodically. Cant wait to get it out of 
> service 
> later in May. 
> 
> Hmm ... Grant T has the same drift (darn ... mail filters are 
> busy tonight -- 
> must be lots of spam in this round ... took Grants mail 6 minutes 
> to get into 
> the box after yours....) 
> 
> Alistair Tonner 
> 
> > I'm really lost. ANY ideas out there? 
> 
>
Have a look at the WXP computer if patch KB893066 is installed as this one changes some default TCP window size. I had a whole LAN down before stumbling across this is one. Once removed everything worked fine. Here is the KB article from MS :

http://support.microsoft.com/default.aspx?scid=kb;en-us;893066 

Hope this helps you ...

Enjoy your day,

Bernd Lippert

Re: Linux forwarding Win XP hosts VERY slowly

[Dave Cinege]

On Wednesday 31 December 1969 18:59, netfilter-request@lists.netfilter.org 
wrote:

> I'm going to presume that the different subnet's from your eth1 and your
> iptables rule 172.27.1.0/30 vs 172.27.0.0/30 was a typo.

I'm going to presume you saying 172.27.1.0/30 vs 172.27.0.0/30 was a 
typo.  ;  ) I'm lazy in the code that generates these rules and I just use 
ip/mask. iptables seems to be intelligent enough to round down to the network 
address. It does seem to work....

# /sbin/iptables -t nat -I POSTROUTING -o eth1 -d 172.27.0.2/255.255.255.252 \
-j SNAT --to-source 172.27.0.2

# iptables -t nat -L -n
...
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
SNAT       all  --  0.0.0.0/0            172.27.0.0/30      to:172.27.0.2

> I'm not sure what to say.  I'm at a loss as to what might be causing
> problems.  At this point I would start sniffing as much traffic as I could
> on all the networks going in and out of devices that you are working with.
>  What you are trying to do looks reasonable to me.  I don't see any
> inherent flaws in what you are trying to do.  But that is just my eyes, see
> what the rest of the list has to say.

Well I will see if adding a 'clamp-mss-to-pmtu' rule makes any difference. 
Beyond that the next step is to build a unit and XP box and leave them 
connected a few days until problems start. If problems start. I've yet to
see these issues in-shop at all.

Dave

Re: Linux forwarding Win XP hosts VERY slowly

[Dave Cinege]

> > Yes. In my last example of the slow browing to an attached linksys, I was
> > using Firefox on XP. Previously I had thought it was all IE, but it's
> > not.
>
> Am I reading this correctly?  Are you experiencing slow downs in
> connections to a Linksys's (device) management web page that is connected
> to the same network on the same subnet?  Translated are you experiencing
> slow downs in connections to a equipment when the traffic is not passing
> through your router?

No. To get very specific:

Local net
    eth0: 192.168.1.1 - net/mask 192.168.1.0/24
Configuration network (special type of network in my appliance)
    eth1: 172.27.1.2 - net/mask 172.27.0.0/30

The configuration network Masq's:
/sbin/iptables -t nat -I POSTROUTING -o eth1 -d 172.27.0.2/255.255.255.252 \
  -j SNAT --to-source 172.27.0.2

The SNAT to configration networks is to simplify life, as we don't have to 
deal with playing with routes back to the local network in the external 
devices, (wifi bridges, satellite terminals, etc)

And we normally browse to these devices via IP not hostname.

So to clarify my situtation:
 Depending on what mood Windows is in, browering from a 192.168.1.0/24 host to 
172.27.1.1 (linksys bridge in this case) or browsing to 192.168.1.1 (the 
appliance itself, running Zope) will become ungodly slow. SOMETIMES it is 
just fine.

I am also seeing zope deadlock, which I now believe is related to this 
performance problem. When zope locks all other processes are fine, and the 
box pings. 

To expand further on the zope issue:
A few months ago I was remotly ssh'ed in to an appliance with my partner on 
site. Zope had locked. I restarted the process. He connected a few times and 
it locked again. I could not get a head reply back when it locked. 
   (telnet 127.0.0.1 < HEAD / HTTP/1.0)
We restarted it a dozen times with the same thing. It did not stop locking 
until we rebooted the machine. All of this was via SSL. 

It was at this point I figured it was SSL in zope. I turned off SSL access. 
Seemed to help but now we're seeing locking, though maybe less frequently. 
I was wondering if it was hardware, IRQ, etc. all but ruled that out. As I 
mentioned, I just saw everything run perfectly with my laptop, then I plug 
the XP machine in, and it dies.

If you can't tell this is really driving us crazy. I've got the simplest of 
networks configs, barely any traffic, and it's running like shit with 
direclty attached hosts. 

Dave

Re: Linux forwarding Win XP hosts VERY slowly

[Taylor, Grant]

> Local net
>     eth0: 192.168.1.1 - net/mask 192.168.1.0/24
> Configuration network (special type of network in my appliance)
>     eth1: 172.27.1.2 - net/mask 172.27.0.0/30
> 
> The configuration network Masq's:
> /sbin/iptables -t nat -I POSTROUTING -o eth1 -d 172.27.0.2/255.255.255.252 \
>   -j SNAT --to-source 172.27.0.2

I'm going to presume that the different subnet's from your eth1 and your iptables rule 172.27.1.0/30 vs 172.27.0.0/30 was a typo.

> The SNAT to configration networks is to simplify life, as we don't have to 
> deal with playing with routes back to the local network in the external 
> devices, (wifi bridges, satellite terminals, etc)
> 
> And we normally browse to these devices via IP not hostname.
> 
> So to clarify my situtation:
>  Depending on what mood Windows is in, browering from a 192.168.1.0/24 host to 
> 172.27.1.1 (linksys bridge in this case) or browsing to 192.168.1.1 (the 
> appliance itself, running Zope) will become ungodly slow. SOMETIMES it is 
> just fine.

I'm not sure what to say.  I'm at a loss as to what might be causing problems.  At this point I would start sniffing as much traffic as I could on all the networks going in and out of devices that you are working with.  What you are trying to do looks reasonable to me.  I don't see any inherent flaws in what you are trying to do.  But that is just my eyes, see what the rest of the list has to say.

> I am also seeing zope deadlock, which I now believe is related to this 
> performance problem. When zope locks all other processes are fine, and the 
> box pings. 
> 
> To expand further on the zope issue:
> A few months ago I was remotly ssh'ed in to an appliance with my partner on 
> site. Zope had locked. I restarted the process. He connected a few times and 
> it locked again. I could not get a head reply back when it locked. 
>    (telnet 127.0.0.1 < HEAD / HTTP/1.0)
> We restarted it a dozen times with the same thing. It did not stop locking 
> until we rebooted the machine. All of this was via SSL. 
> 
> It was at this point I figured it was SSL in zope. I turned off SSL access. 
> Seemed to help but now we're seeing locking, though maybe less frequently. 
> I was wondering if it was hardware, IRQ, etc. all but ruled that out. As I 
> mentioned, I just saw everything run perfectly with my laptop, then I plug 
> the XP machine in, and it dies.

I'd be willing to bet that there is an underlying issue that is messing with things that is causing the SSL slowdown as well as the newly occurring non-SSL slowdown.  What that underlying issue is I have no clue though.

> If you can't tell this is really driving us crazy. I've got the simplest of 
> networks configs, barely any traffic, and it's running like shit with 
> direclty attached hosts. 

I bet.  I wish that I could be more help, but alas I can not.



Grant. . . .

Re: Linux forwarding Win XP hosts VERY slowly

[Mogens Valentin]

Dave Cinege wrote:
> To expand further on the zope issue:
> A few months ago I was remotly ssh'ed in to an appliance with my partner on 
> site. Zope had locked. I restarted the process. He connected a few times and 
> it locked again. I could not get a head reply back when it locked. 
>    (telnet 127.0.0.1 < HEAD / HTTP/1.0)
> We restarted it a dozen times with the same thing. It did not stop locking 
> until we rebooted the machine. All of this was via SSL. 
> 
> It was at this point I figured it was SSL in zope. I turned off SSL access. 
> Seemed to help but now we're seeing locking, though maybe less frequently. 
> I was wondering if it was hardware, IRQ, etc. all but ruled that out. As I 
> mentioned, I just saw everything run perfectly with my laptop, then I plug 
> the XP machine in, and it dies.

Sorry I have no way of directly helping, but since zope freezes over, 
could that be a stupid HW problem? i've seen a few services lock up, 
where the rest went on just fine.
Sometimes it has been temperature, sometimes a ramproblem, maybe 
nearly-ok ram, just operating at too close timings.
I don't expect to hear it's HW; your skills there are well reputed :p

A few ideas to isolate whether it's network problems or...
Can you keep zope running without locks with no connections?
Does it run without locks when connecting from the local box itself?
It it possible to create some local testapp to put load on zope?
And yes, I did read it works with one client. Doesn't make it better.

> If you can't tell this is really driving us crazy.

Get your drift...

-- 
Kind regards,
Mogens Valentin

Re: Linux forwarding Win XP hosts VERY slowly

[Dave Cinege]

On Wednesday 31 December 1969 18:59, netfilter-request@lists.netfilter.org 
wrote:

Grant,

>   1)  Check to make sure that it is not a ""windowing issue, i.e. the
> MTU/MRU, MSS, etc. 

Thanks for the hint. Looks like I should have some default 'clamp' rules as a 
start.

iptables -t filter -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS 
--clamp-mss-to-pmtu 

Correct? What about the traffic going to the appliance? This rule fails on 
INPUT....

> 2)  I know that XP SP 2 changes the number of TCP/IP 
> connections that a computer can have open at any given time.  I think the
> default prior to XP SP 2 was 50 or there abouts.  Now the default for XP SP

This is a product I'm building. It's not a single Firewall in a shop. Once 
it's out of my hands, I have no control over what is connecting to it. None of 
these people care their machine is broken. So, I need to do what i can in 
Linux on the appliance.   : (

Dave

Re: Linux forwarding Win XP hosts VERY slowly

[Dave Cinege]

On Wednesday 31 December 1969 18:59, netfilter-request@lists.netfilter.org 
wrote:

Grant,

>   1)  Check to make sure that it is not a ""windowing issue, i.e. the
> MTU/MRU, MSS, etc. 

Thanks for the hint. Looks like I should have some default 'clamp' rules as a 
start.

iptables -t filter -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS 
--clamp-mss-to-pmtu 

Correct? What about the traffic going to the appliance? This rule fails on 
INPUT....

> 2)  I know that XP SP 2 changes the number of TCP/IP 
> connections that a computer can have open at any given time.  I think the
> default prior to XP SP 2 was 50 or there abouts.  Now the default for XP SP

This is a product I'm building. It's not a single Firewall in a shop. Once 
it's out of my hands, I have no control over what is connecting to it. None of 
these people care their machine is broken. So, I need to do what i can in 
Linux on the appliance.   : (

Dave

Linux forwarding Win XP hosts VERY slowly

[Dave Cinege]

I've built an advanced rotuign appliance, and I'm having 2 outstanding 
problems, that I'm being to think are related to the linux ip/netfilter 
stack, choking on XP traffic (possiblity XP-SP2) hosts that are on the LAN. 
I'm running 2.4.30 at the moment. 

The 2 problems I'm seeing:

1) Forwarded traffic (most notably web) is VERY slow with XP clients. 

Example: Saw this last 2 nights ago: Appliance has a linksys Wifi bridge 
attached to a NIC. Customer browses through the appliance to the Linksys 
config page. It moves like molasses. He browse to the local Zope hosted made 
page. Slow as hell. I unplug his machine, and plug my linux laptop into same 
switch port. Linksys and Zope pages load adn reload instantly. Plug his 
machine in....slow again.

2) Zope serves user interface pages for the appliance. Zope has been locking 
solid for no apparent reason, but only when and Windows host is attached. 
The trick is SOME windows machine don't seem to cause a problem.
Example: 
I worked with a unit for 3 days using a customers XP desktop. Not a hiccup. My 
partner came in and attached to the network and starting connect to our 
appliance with his XP laptop. Within 15 minutes Zope was hung. 

I'm really lost. ANY ideas out there?

Re: Linux forwarding Win XP hosts VERY slowly

[Taylor, Grant]

> I've built an advanced rotuign appliance, and I'm having 2 outstanding 
> problems, that I'm being to think are related to the linux ip/netfilter 
> stack, choking on XP traffic (possiblity XP-SP2) hosts that are on the LAN. 
> I'm running 2.4.30 at the moment. 
> 
> The 2 problems I'm seeing:
> 
> 1) Forwarded traffic (most notably web) is VERY slow with XP clients. 
> 
> Example: Saw this last 2 nights ago: Appliance has a linksys Wifi bridge 
> attached to a NIC. Customer browses through the appliance to the Linksys 
> config page. It moves like molasses. He browse to the local Zope hosted made 
> page. Slow as hell. I unplug his machine, and plug my linux laptop into same 
> switch port. Linksys and Zope pages load adn reload instantly. Plug his 
> machine in....slow again.
> 
> 2) Zope serves user interface pages for the appliance. Zope has been locking 
> solid for no apparent reason, but only when and Windows host is attached. 
> The trick is SOME windows machine don't seem to cause a problem.
> Example: 
> I worked with a unit for 3 days using a customers XP desktop. Not a hiccup. My 
> partner came in and attached to the network and starting connect to our 
> appliance with his XP laptop. Within 15 minutes Zope was hung. 
> 
> I'm really lost. ANY ideas out there?

I don't know if this is related or not, but a couple of this come to mind.

  1)  Check to make sure that it is not a ""windowing issue, i.e. the MTU/MRU, MSS, etc.
  2)  I know that XP SP 2 changes the number of TCP/IP connections that a computer can have open at any given time.  I think the default prior to XP SP 2 was 50 or there abouts.  Now the default for XP SP 2 is 10, or so I have read.  Apparently this is a registry (?) setting that can be changed.  From what I have read is that this was / is an effort by M$ to make sure that infected systems can only infect other systems at a controlled rate.  This problem has primarily shown up on VoIP mailing lists as VoIP needs LOTS of small packets at any given time exceeding the amount that XP SP 2 will allow by default.



Grant. . . .

Re: Linux forwarding Win XP hosts VERY slowly

[Alistair Tonner]

On April 30, 2005 01:41 pm, Dave Cinege wrote:
> I've built an advanced rotuign appliance, and I'm having 2 outstanding
> problems, that I'm being to think are related to the linux ip/netfilter
> stack, choking on XP traffic (possiblity XP-SP2) hosts that are on the LAN.
> I'm running 2.4.30 at the moment.
>
> The 2 problems I'm seeing:
>
> 1) Forwarded traffic (most notably web) is VERY slow with XP clients.
>
> Example: Saw this last 2 nights ago: Appliance has a linksys Wifi bridge
> attached to a NIC. Customer browses through the appliance to the Linksys
> config page. It moves like molasses. He browse to the local Zope hosted
> made page. Slow as hell. I unplug his machine, and plug my linux laptop
> into same switch port. Linksys and Zope pages load adn reload instantly.
> Plug his machine in....slow again.
>
> 2) Zope serves user interface pages for the appliance. Zope has been
> locking solid for no apparent reason, but only when and Windows host is
> attached. The trick is SOME windows machine don't seem to cause a problem.
> Example:
> I worked with a unit for 3 days using a customers XP desktop. Not a hiccup.
> My partner came in and attached to the network and starting connect to our
> appliance with his XP laptop. Within 15 minutes Zope was hung.
>

	I would strongly suspect the XP box has a b0rken TCP stack happening.  One 
thing that some windows systems will do is flat out ignore the TCP MTU/window 
size settings on the network, especially if you've a) set them up for modem 
dialup b) installed anything that is supposed to automatically improve your 
internet speed c) hard wired (EVER) the MTU settings.  I had a win2k box that 
had this sort of issue once, and even though it was at that time set to use 
default settings for MTU/max recieve window and the like, I had to completely 
uninstall the tcp stack, the network card driver, all modem bits etc and 
reinstall em from scratch to get it to behave normally.  For the record, 
Ethereal dumps of the communications CLEARLY show that the windows box is 
using bad MTU settings and bad TCP window sizes, if this is the case.

I *still* believe that there are settings left on that box (still in use these 
days) that cause issues periodically.  Cant wait to get it out of service 
later in May.

	Hmm ... Grant T has the same drift (darn ... mail filters are busy tonight -- 
must be lots of spam in this round ... took Grants mail 6 minutes to get into 
the box after yours....)

	Alistair Tonner

> I'm really lost. ANY ideas out there?

Re: Linux forwarding Win XP hosts VERY slowly

[Raphael Jacquot]

Dave Cinege wrote:
> I've built an advanced rotuign appliance, and I'm having 2 outstanding 
> problems, that I'm being to think are related to the linux ip/netfilter 
> stack, choking on XP traffic (possiblity XP-SP2) hosts that are on the LAN. 
> I'm running 2.4.30 at the moment. 
> 
> The 2 problems I'm seeing:
> 
> 1) Forwarded traffic (most notably web) is VERY slow with XP clients. 

have you tried with a proper web browser, such as firefox ? IE is known
as not respecting the RFC for TCP and sending the HTTP request as data
in the first "SYN" packet (which is only recognised by IIS, of course)
so as to 'make' IE appear faster than it really is..

> Example: Saw this last 2 nights ago: Appliance has a linksys Wifi bridge 
> attached to a NIC. Customer browses through the appliance to the Linksys 
> config page. It moves like molasses. He browse to the local Zope hosted made 
> page. Slow as hell. I unplug his machine, and plug my linux laptop into same 
> switch port. Linksys and Zope pages load adn reload instantly. Plug his 
> machine in....slow again.
> 
> 2) Zope serves user interface pages for the appliance. Zope has been locking 
> solid for no apparent reason, but only when and Windows host is attached. 
> The trick is SOME windows machine don't seem to cause a problem.
> Example: 
> I worked with a unit for 3 days using a customers XP desktop. Not a hiccup. My 
> partner came in and attached to the network and starting connect to our 
> appliance with his XP laptop. Within 15 minutes Zope was hung. 
> 
> I'm really lost. ANY ideas out there?
> 

guess it's one more example of how badly Windows' TCP/IP stack is broken
is so many ways...

Re: Linux forwarding Win XP hosts VERY slowly

[Taylor, Grant]

> have you tried with a proper web browser, such as firefox ? IE is known
> as not respecting the RFC for TCP and sending the HTTP request as data
> in the first "SYN" packet (which is only recognised by IIS, of course)
> so as to 'make' IE appear faster than it really is..

Would it be possible / likely / practical to mangle such traffic to remove any data from the payload of the SYN packet?  Ultimately creating a cleaner connection to the server and causing IE to timeout on the packet and resend it?

> guess it's one more example of how badly Windows' TCP/IP stack is broken
> is so many ways...

*nod*



Grant. . . .