TCP delay, SMTP errors

TCP delay, SMTP errors

[Nathan Cassano]

Hi folks,
	Our website (on an internal NAT'ed network) connects our email
server (on an external network). The website uses a SMTP PHP class to
send mail using socket connections. The problem is a delay in connecting
to the mail server that causes the class to quit prematurely. I fixed
the problem by adding a function call to sleep(1) after fsockopen() but
this is a hack. Is there a better way to fix this TCP delay? I tried
setting the TOS value to Minimize-Delay but I didn't see any difference.
Any suggestions are welcome. 

Nathan

PHP script 
<?php

$this->connection = fsockopen("mail.server.tld", 25, $errno, $errstr,
30);

socket_set_timeout($this->connection, 10, 0);

$smtp_greeting = fgets($this->connection, 512);

?>

$IPTABLES -A OUTPUT -t mangle -p tcp --dport 25 -j TOS --set-tos
Minimize-Delay

Re: TCP delay, SMTP errors

[Ramin Alidousti]

Hi Nathan,

First you have to find out what the reason of this delay is. Eg, is this
because of the load on the server or the client or is it because of the
poor layer 2 medium (lots of collisions...) or any other reason.

Are you here implying that the delay is because of the NAT'ed architecture?
What does a simple ping show delay-wise? What is the routing to and from
that external host? Is it assymetric? What happens if you connect to this
external host from the external interface of the firewall itself? Are you
still experiencing delays? Is this only happening with TCP or all types of
packets would have the same delay?

You see, it's not very easy to diagnose your delay problem without answering
to these and (apparently) many other questions.

Ramin

On Mon, Jun 10, 2002 at 01:45:50PM -0700, Nathan Cassano wrote:

> 
> Hi folks,
> 	Our website (on an internal NAT'ed network) connects our email
> server (on an external network). The website uses a SMTP PHP class to
> send mail using socket connections. The problem is a delay in connecting
> to the mail server that causes the class to quit prematurely. I fixed
> the problem by adding a function call to sleep(1) after fsockopen() but
> this is a hack. Is there a better way to fix this TCP delay? I tried
> setting the TOS value to Minimize-Delay but I didn't see any difference.
> Any suggestions are welcome. 
> 
> Nathan
> 
> PHP script 
> <?php
> 
> $this->connection = fsockopen("mail.server.tld", 25, $errno, $errstr,
> 30);
> 
> socket_set_timeout($this->connection, 10, 0);
> 
> $smtp_greeting = fgets($this->connection, 512);
> 
> ?>
> 
> $IPTABLES -A OUTPUT -t mangle -p tcp --dport 25 -j TOS --set-tos
> Minimize-Delay
> 

Re: TCP delay, SMTP errors

[Doug Monroe]

Nathan Cassano wrote:
> 
> Hi folks,
>         Our website (on an internal NAT'ed network) connects our email
> server (on an external network). The website uses a SMTP PHP class to
> send mail using socket connections. The problem is a delay in connecting
> to the mail server that causes the class to quit prematurely. I fixed
> the problem by adding a function call to sleep(1) after fsockopen() but
> this is a hack. Is there a better way to fix this TCP delay? I tried
> setting the TOS value to Minimize-Delay but I didn't see any difference.
> Any suggestions are welcome.

what happens when you use telnet from the website box to the mail server's
port 25? do you get a delay rec'ving the SMTP banner? Likely has nothing to do
with netfilter...my money's on a paranoid SMTP trying to do a reverse DNS
lookup (or identd?) causing the delay. Maybe there's no PTR record for the NAT
(public) IP of the website? Maybe your SMTP server has a big-ish list of DNS
hosts to try before it finally timesout?
--
Doug

Re: TCP delay, SMTP errors

[Simon A. Boggis]

[-- Attachment #1: Type: text/plain, Size: 1576 bytes --]

On Mon, 2002-06-10 at 23:35, Doug Monroe wrote:
> Nathan Cassano wrote:
> >         Our website (on an internal NAT'ed network) connects our email
> > server (on an external network). The website uses a SMTP PHP class to
> > send mail using socket connections. The problem is a delay in connecting
> > to the mail server that causes the class to quit prematurely. I fixed
>
> what happens when you use telnet from the website box to the mail server's
> port 25? do you get a delay rec'ving the SMTP banner? Likely has nothing to do
> with netfilter...my money's on a paranoid SMTP trying to do a reverse DNS
> lookup (or identd?) causing the delay. Maybe there's no PTR record for the NAT
> (public) IP of the website? Maybe your SMTP server has a big-ish list of DNS
> hosts to try before it finally timesout?

My first suspicion would be a timing out ident from the mail server.

I always send rejects on port 113/tcp (ident) for this reason. Some MTAs
absolutely insist on doing an ident (or timing out trying) before they
accept your mail.

It could be useful to tcpdump at your firewall/router, or on the machine
doing the smtp connection (if no border filtering) and see what is going
on.

Simon

-- 
----------------------------------------------------------------------
Dr Simon A. Boggis                                  Systems Programmer
Department of Computer Science,                     Tel. 020 7882 7522
Queen Mary, University of London, London E1 4NS UK. 
---- GPG public key <http://www.dcs.qmul.ac.uk/~simon/#publickey> ----

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 232 bytes --]

RE: TCP delay, SMTP errors

[Nathan Cassano]

Ramin wrote:

> First you have to find out what the reason of this delay is. Eg, is
this
> because of the load on the server or the client or is it because of
the
> poor layer 2 medium (lots of collisions...) or any other reason.
The load on the our switch is low. 99% idle on the firewall, and 90% on
the internal web server, both dual PIII's.

> Are you here implying that the delay is because of the NAT'ed
architecture?
That is my suspicion. I recall Linux NAT doing this type of thing
before.

> What does a simple ping show delay-wise?
Nothing irregular, all the ping times are the same.

> What is the routing to and from that external host? Is it asymmetric?
Static routing, one external network device.

> What happens if you connect to this external host from the external
interface of the firewall itself?
> Are you still experiencing delays?
Yes, the same delay is present connecting from the firewall.

> Is this only happening with TCP or all types of packets would have the
same delay?
Doing DNS queries using the dig program to our external DNS server is
fast. It seems
to be a delay in sending initial TCP data. 

Doug wrote:

> what happens when you use telnet from the website box to the mail
server's port 25?
> Do you get a delay rec'ving the SMTP banner?
Here is a delay of about a five seconds, then the SMTP banner displays. 

> Likely has nothing to do with netfilter...my money's on a paranoid
SMTP trying to
> do a reverse DNS lookup (or identd?) causing the delay. Maybe there's
no PTR record
> for the NAT (public) IP of the website?
I don't think so, try it for yourself (www4.cjhunter.com, 63.174.37.3).
We have no ident checks.

> Maybe your SMTP server has a big-ish list of DNS hosts to try before
it finally timesout?
Connecting to the mail server in quite fast outside the NAT'ed network.


Thanks for your good questions that help define the situation better.
Some extra info. 

Inbound connections are only allowed through --state
ESTABLISHED,RELATED.

Is there any way to explain these delays?

Re: TCP delay, SMTP errors

[Ramin Alidousti]

Nathan,

I also saw two replied to your question, one was about the reverse
lookup of your client and the other was the ident query. Both are
valid possibilities. What does tcpdump show?

BTW, the NAT is not doing anything weird here as the same NAT would
be applied to your non-TCP packets which, you say, are not showing
any delay..

Ramin

On Tue, Jun 11, 2002 at 08:54:25AM -0700, Nathan Cassano wrote:

> 
> Ramin wrote:
> 
> > First you have to find out what the reason of this delay is. Eg, is
> this
> > because of the load on the server or the client or is it because of
> the
> > poor layer 2 medium (lots of collisions...) or any other reason.
> The load on the our switch is low. 99% idle on the firewall, and 90% on
> the internal web server, both dual PIII's.
> 
> > Are you here implying that the delay is because of the NAT'ed
> architecture?
> That is my suspicion. I recall Linux NAT doing this type of thing
> before.
> 
> > What does a simple ping show delay-wise?
> Nothing irregular, all the ping times are the same.
> 
> > What is the routing to and from that external host? Is it asymmetric?
> Static routing, one external network device.
> 
> > What happens if you connect to this external host from the external
> interface of the firewall itself?
> > Are you still experiencing delays?
> Yes, the same delay is present connecting from the firewall.
> 
> > Is this only happening with TCP or all types of packets would have the
> same delay?
> Doing DNS queries using the dig program to our external DNS server is
> fast. It seems
> to be a delay in sending initial TCP data. 

RE: TCP delay

[Nathan Cassano]

Okay folks,
	I've narrowed this problem down a bit. There is a delay between
the firewall and the external network when I telnet to ports 23 or 25 on
any host on the external network. The weird thing is that port 22 (ssh)
will give me a quick response. My only thought is that the state
tracking is slow when allowing packets back in.

I have confirmed that DNS is working A and PTR on all hosts tested.

Below is the relevant parts of the firewall boot script.

/sbin/modprobe ip_tables
/sbin/modprobe ip_conntrack
/sbin/modprobe iptable_filter
/sbin/modprobe iptable_mangle
/sbin/modprobe iptable_nat
/sbin/modprobe ipt_LOG
/sbin/modprobe ipt_state
/sbin/modprobe ipt_REJECT
/sbin/modprobe ipt_MASQUERADE

# Disable Smurf amplifier attacks
/bin/echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts 

# Disable ICMP redirect acceptance. ICMP redirects can be used to alter
your routing 
# tables, possibly to a bad end. 
/bin/echo 0 > /proc/sys/net/ipv4/conf/all/accept_redirects 

# Don't accept source routed packets. Attackers can use source routing
to generate 
# traffic pretending to be from inside your network, but which is routed
back along 
# the path from which it came, namely outside, so attackers can
compromise your 
# network. Source routing is rarely used for legitimate purposes. 
/bin/echo 0 > /proc/sys/net/ipv4/conf/all/accept_source_route 

# Enable bad error message protection. 
/bin/echo 1 > /proc/sys/net/ipv4/icmp_ignore_bogus_error_responses 

###########################################
#
# Setup iptables rules
#

# Set all default Chain Policies to DROP

$IPTABLES -P INPUT DROP
$IPTABLES -P OUTPUT DROP
$IPTABLES -P FORWARD DROP

# Allow unlimited traffic on the loopback interface. 

$IPTABLES -A INPUT  -i lo -j ACCEPT 
$IPTABLES -A OUTPUT -o lo -j ACCEPT 

# Setup packet mangling for services

$IPTABLES -A OUTPUT -t mangle -p tcp --dport 22 -j TOS --set-tos
Minimize-Delay
$IPTABLES -A OUTPUT -t mangle -p tcp --dport 23 -j TOS --set-tos
Minimize-Delay
$IPTABLES -A OUTPUT -t mangle -p tcp --dport 80 -j TOS --set-tos
Minimize-Delay
$IPTABLES -A OUTPUT -t mangle -p tcp --dport 25 -j TOS --set-tos
Minimize-Delay

# Create chain targets segregated by the ethernet device's INPUTs and
OUTPUTs

$IPTABLES -N EXT_INPUT
$IPTABLES -N EXT_OUTPUT

# Direct traffic to go to their matching device input or output chain
targets

$IPTABLES -A INPUT -i $EXT_IFACE -j EXT_INPUT
$IPTABLES -A OUTPUT -o $EXT_IFACE -j EXT_OUTPUT

# Allow inbound connections to external device that are already
established
$IPTABLES -A EXT_INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow outbound traffic from external device
$IPTABLES -A EXT_OUTPUT -s $EXT_IP -j ACCEPT