[LARTC] MTU problem with simple router?

[LARTC] MTU problem with simple router?

[Ken Price]

My colo provider provides IP addresses and expects routing much like a T-1
data provider.  I get 4-5 "WAN" IP addresses and 32 "PUBLIC ROUTABLE" IP
addresses.  Their enterprise router expects another router (customer
provided) to handle this forwarding.  In this role, I currently use a RedHat
7.3 box with 2 NICs, simply with IP forwarding enabled.  No special rules or
shaping.  Straight forward enough, and it's always worked without a hitch.
Behind this router, I have another RedHat box acting as a NAT Firewall which
protects my server farm.  Now my problem.  We've recently developed an
application that makes outgoing requests to other websites and returns data.
I'm noticing a serious lag in the amount of time it takes for this data to
return to the server vs. our development environment in the office which
uses a much slower internet link.

If I test from the production RH7.3 "router", all data is returned extremely
fast.  If I step back to the NAT firewall, or further back into the server
farm, I get serious delays.  ICMP does not seem to reflect this problem, I'm
assuming because of it's small packet size.  Could MTU size be an issue
here?  All of my firewalls and routers use the default 1500 MTU size and the
network is all 100Mbps up to the OC-48 internet backbone.  Am I missing some
router configuration?  I've tried adjusting the MTU size on the router with
no change in results.

Any suggestions on where to go with this? 

-Ken

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] MTU problem with simple router?

[Stef Coene]

On Friday 25 October 2002 17:38, Ken Price wrote:
> My colo provider provides IP addresses and expects routing much like a T-1
> data provider.  I get 4-5 "WAN" IP addresses and 32 "PUBLIC ROUTABLE" IP
> addresses.  Their enterprise router expects another router (customer
> provided) to handle this forwarding.  In this role, I currently use a
> RedHat 7.3 box with 2 NICs, simply with IP forwarding enabled.  No special
> rules or shaping.  Straight forward enough, and it's always worked without
> a hitch. Behind this router, I have another RedHat box acting as a NAT
> Firewall which protects my server farm.  Now my problem.  We've recently
> developed an application that makes outgoing requests to other websites and
> returns data. I'm noticing a serious lag in the amount of time it takes for
> this data to return to the server vs. our development environment in the
> office which uses a much slower internet link.
>
> If I test from the production RH7.3 "router", all data is returned
> extremely fast.  If I step back to the NAT firewall, or further back into
> the server farm, I get serious delays.  ICMP does not seem to reflect this
> problem, I'm assuming because of it's small packet size.  Could MTU size be
> an issue here?  All of my firewalls and routers use the default 1500 MTU
> size and the network is all 100Mbps up to the OC-48 internet backbone.  Am
> I missing some router configuration?  I've tried adjusting the MTU size on
> the router with no change in results.
>
> Any suggestions on where to go with this?
If I have a tcp delay, I always check the dns config.  In many cases, there is 
no reverse dns lookup of the ip-address so the other hosts waits for the 
dns-timeout before allowing the connection.  So, has your ip-address a 
reverse dns entry?

Stef

-- 

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

RE: [LARTC] MTU problem with simple router?

[Ken Price]

All of our public IPs are reverse mapped.  The initial connection to the
site is fast.  The delay happens when data starts comming back.  A way to
visualize this problem is using a browser.  You hit "Go" and the target site
immediately returns text, but like a low-bandwidth or overloaded site,
graphics trickle back.  This problem is not limited to a single site ...
it's all of them.  And isn't limited to a single router, I have two
different production evironments setup with different loadbalancer/firewall
combos.  What they both have in common is the RedHat router doing simple
forwarding.  One in each environments.

-Ken


>> Any suggestions on where to go with this?
>If I have a tcp delay, I always check the dns config.  In many cases, there
is 
>no reverse dns lookup of the ip-address so the other hosts waits for the 
>dns-timeout before allowing the connection.  So, has your ip-address a 
>reverse dns entry?

>Stef
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] MTU problem with simple router?

[Stef Coene]

On Friday 25 October 2002 18:53, Ken Price wrote:
> All of our public IPs are reverse mapped.  The initial connection to the
> site is fast.  The delay happens when data starts comming back.  A way to
> visualize this problem is using a browser.  You hit "Go" and the target
> site immediately returns text, but like a low-bandwidth or overloaded site,
> graphics trickle back.  This problem is not limited to a single site ...
> it's all of them.  And isn't limited to a single router, I have two
> different production evironments setup with different loadbalancer/firewall
> combos.  What they both have in common is the RedHat router doing simple
> forwarding.  One in each environments.
Stange.  Have you tried to dump the packets with tcpdump so you can analyse 
what happens ?

Stef

-- 

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

RE: [LARTC] MTU problem with simple router?

[Ken Price]

Yes.  Nothing out of the ordinary.  I'm seeing packets being sent at 1460
and returned at 1448 - both under my interface's MTU of 1500.  I even tried
with MTU sizes down to 500 at the firewall.  No luck.  The TCPDUMP actually
looks just like our development firewall's TCPDUMP in the office - and
development works fine.  So I'm lost.  The routers in question are Dell
350's ... PIII-850s with 256Mb RAM with Intel Pro100 NICS, plus one has a
4-port Znyx card (tulip drivers).  Page data (just the HTML text) is
returned within 1-2 seconds in development or on an outer production router,
8-10 seconds within production.  That's a considerable difference.

Leaving firewalling out of it, if you were to setup a Linux router to simply
bridge two subnets, after assigning the correct IPs to the interfaces,
setting the default gateway to the enterprise router of the ISP, and

# echo 1 > /proc/sys/net/ipv4/ip_forward

what needs to be done?

Ken

-----Original Message-----
From: Stef Coene [mailto:stef.coene@docum.org]
Sent: Friday, October 25, 2002 1:22 PM
To: Ken Price; lartc@mailman.ds9a.nl
Subject: Re: [LARTC] MTU problem with simple router?


On Friday 25 October 2002 18:53, Ken Price wrote:
> All of our public IPs are reverse mapped.  The initial connection to the
> site is fast.  The delay happens when data starts comming back.  A way to
> visualize this problem is using a browser.  You hit "Go" and the target
> site immediately returns text, but like a low-bandwidth or overloaded
site,
> graphics trickle back.  This problem is not limited to a single site ...
> it's all of them.  And isn't limited to a single router, I have two
> different production evironments setup with different
loadbalancer/firewall
> combos.  What they both have in common is the RedHat router doing simple
> forwarding.  One in each environments.
Stange.  Have you tried to dump the packets with tcpdump so you can analyse 
what happens ?

Stef

-- 

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

RE: [LARTC] MTU problem with simple router?

[Ken Price]

Ok everyone.  I want to apologize for this post.  The culprit was a poorly
configured Cisco 2950 switch (2 of them to be precise).  On VLAN creation,
STP (spanning tree) is enabled by default.  Disabling this feature
completely eliminated all the funky latency issues I've been experiencing.
Doh!

-Ken

-----Original Message-----
From: Ken Price [mailto:kprice@agentware.net]
Sent: Friday, October 25, 2002 2:27 PM
To: 'Stef Coene'; lartc@mailman.ds9a.nl
Subject: RE: [LARTC] MTU problem with simple router?


Yes.  Nothing out of the ordinary.  I'm seeing packets being sent at 1460
and returned at 1448 - both under my interface's MTU of 1500.  I even tried
with MTU sizes down to 500 at the firewall.  No luck.  The TCPDUMP actually
looks just like our development firewall's TCPDUMP in the office - and
development works fine.  So I'm lost.  The routers in question are Dell
350's ... PIII-850s with 256Mb RAM with Intel Pro100 NICS, plus one has a
4-port Znyx card (tulip drivers).  Page data (just the HTML text) is
returned within 1-2 seconds in development or on an outer production router,
8-10 seconds within production.  That's a considerable difference.

Leaving firewalling out of it, if you were to setup a Linux router to simply
bridge two subnets, after assigning the correct IPs to the interfaces,
setting the default gateway to the enterprise router of the ISP, and

# echo 1 > /proc/sys/net/ipv4/ip_forward

what needs to be done?

Ken

-----Original Message-----
From: Stef Coene [mailto:stef.coene@docum.org]
Sent: Friday, October 25, 2002 1:22 PM
To: Ken Price; lartc@mailman.ds9a.nl
Subject: Re: [LARTC] MTU problem with simple router?


On Friday 25 October 2002 18:53, Ken Price wrote:
> All of our public IPs are reverse mapped.  The initial connection to the
> site is fast.  The delay happens when data starts comming back.  A way to
> visualize this problem is using a browser.  You hit "Go" and the target
> site immediately returns text, but like a low-bandwidth or overloaded
site,
> graphics trickle back.  This problem is not limited to a single site ...
> it's all of them.  And isn't limited to a single router, I have two
> different production evironments setup with different
loadbalancer/firewall
> combos.  What they both have in common is the RedHat router doing simple
> forwarding.  One in each environments.
Stange.  Have you tried to dump the packets with tcpdump so you can analyse 
what happens ?

Stef

-- 

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/