Problem with sendmail server behind firewall...

Problem with sendmail server behind firewall...

[Pradeep Bhomia]

We are facing problem with the sendmail server when it is put behind the 
iptables firewall.
The setup is:
Sendmail 8.12.9
IPTables 1.2.7
Shorewall 1.3.14

Earlier Sendmail server was connected directly to the Internet with a valid 
Internet IP. At that time the number of sendmail processes never exceeded 10 
at any given poing of time. (Checked with ps -ef | grep sendmail)
Now we have implemented the Mandrake Linux 9.1 based firewall using IPTables 
and Shorewall. NATting was configured on the firewall. After doing this, we 
have observed that the number of sendmail process keeps on increasing and 
goes upto nearly 170 processes. It seems that the problem is with the 
incoming message requests. It process remains open for nearly 2 hours. This 
is observed for random connections or some particular sites.
As such sendmail server is receiving and sending mails without any problem to 
the users.
After I remove the firewall and put the server directly on the internet the 
sendmail processes remain at less than 10.

I have been working on this problem for last two days without success. I 
cannot understand whether the problem is with the implementation of firewall 
(NATting) or with sendmail server. I have checked the firewall with only 
NATting (removed all the rules)

Kindly help.

Thanks and regards, 
 
Pradeep Bhomia 

RE: Problem with sendmail server behind firewall...

[Mark E. Donaldson]

It sounds to me, based on the number of processes being generated, that you
may have the classic IDENT (AUTH) problem when sendmail is NATed behind a
firewall.  The IDENT (AUTH) protocol is often necessary for the smooth
performance and functioning of certain services such as mail, POP3, ftp, and
IRC.  This is particularly the case for sendmail in its default
configuration.  For instance, when sending outgoing mail, the receiving mail
server attempt to IDENT the sending mail server on TCP 113 SYN. Problems
occur when the incoming IDENT SYN packets are dropped at the firewall, or
not DNATED to the mail server, as the receiving mail server will wait for a
SYN/ACK reply to the TCP 113 connection request until it times out. IDENT is
a security concern so IDENT requests should normally not be answered, and
yet they cannot be dropped if service is to function properly.
SOLUTIONS:
1) If the firewall has the capability, which netfilter does, REJECT instead
of DROP IDENT requests at firewall, and return a RESET/ACK
back to the requesting server.  The server will then be
satisfied and continue processing the delivery.
2) If the firewall does not have this capability, a TCP 113
hole must be poked through the firewall for INBOUND IDENT requests. Then
direct the TCP 113 requests to the mail relay server wherein the IDENT
service has been disabled. The non-listening IDENT port on the server with
then cause the TCP stack to return a RESET/ACK packet to the requesting
server.
3) OUTBOUND IDENT requests (TCP 113) must also be allowed  through, as well
as inbound IDENT replies, high port RESET/ACKS,
and ICMP "port unreachables".

-----Original Message-----
From: netfilter-admin@lists.netfilter.org
[mailto:netfilter-admin@lists.netfilter.org]On Behalf Of Pradeep Bhomia
Sent: Thursday, September 04, 2003 11:40 PM
To: netfilter@lists.netfilter.org
Subject: Problem with sendmail server behind firewall...


We are facing problem with the sendmail server when it is put behind the
iptables firewall.
The setup is:
Sendmail 8.12.9
IPTables 1.2.7
Shorewall 1.3.14

Earlier Sendmail server was connected directly to the Internet with a valid
Internet IP. At that time the number of sendmail processes never exceeded 10
at any given poing of time. (Checked with ps -ef | grep sendmail)
Now we have implemented the Mandrake Linux 9.1 based firewall using IPTables
and Shorewall. NATting was configured on the firewall. After doing this, we
have observed that the number of sendmail process keeps on increasing and
goes upto nearly 170 processes. It seems that the problem is with the
incoming message requests. It process remains open for nearly 2 hours. This
is observed for random connections or some particular sites.
As such sendmail server is receiving and sending mails without any problem
to
the users.
After I remove the firewall and put the server directly on the internet the
sendmail processes remain at less than 10.

I have been working on this problem for last two days without success. I
cannot understand whether the problem is with the implementation of firewall
(NATting) or with sendmail server. I have checked the firewall with only
NATting (removed all the rules)

Kindly help.

Thanks and regards,

Pradeep Bhomia

Re: Problem with sendmail server behind firewall...

[Jim Carter]

On Fri, 5 Sep 2003, Pradeep Bhomia wrote:
> Earlier Sendmail server was connected directly to the Internet with a valid
> Internet IP. At that time the number of sendmail processes never exceeded 10
> at any given poing of time. (Checked with ps -ef | grep sendmail)
> Now we have implemented the Mandrake Linux 9.1 based firewall using IPTables
> and Shorewall. NATting was configured on the firewall. After doing this, we
> have observed that the number of sendmail process keeps on increasing and
> goes upto nearly 170 processes. It seems that the problem is with the
> incoming message requests. It process remains open for nearly 2 hours. This
> is observed for random connections or some particular sites.
> As such sendmail server is receiving and sending mails without any problem to
> the users.

Given that the problem starts and stops when you turn on and off your
firewall, it must have something to do with the firewall rules.  But... At
UCLA-Mathnet, we saw sendmail (8.12.5) on Solaris behave similarly, on a MX
in a pseudo-DMZ, i.e. porous firewall.  Remote MTA's, obviously sending
spam, would connect and hang through multi-hour timeouts at the start of
data collection (we think), and the server exceeded MaxDaemonChildren and
refused connections.  Legitimate users had no problem, except when
connections were being refused.

We cured it by shortening a number of timeouts and of course raising
MaxDaemonChildren (contact me if you want to see our sendmail.cf, with the
changed timeouts).

However, we had an odd occurrence that seems to tie in with this whole can
of worms: a certain rival school blocked port 25 on the more preferred MX
of their math department (I never found out why).  No TCP reset, no
connection refused, they just dropped the packets.  When sending, sendmail
on Solaris would time out, defer, and later retry the same MX.  But both
Postfix and Sendmail (same version) on Linux would time out, and
immediately try the less preferred MX, which was not blocked.

Conclusion #1: the TCP stacks on Solaris and Linux behave differently.
Conclusion #2: certain MTA's, favored by spammers, possibly part of "bots"
running on victim PC's, get into this hanging behavior when interacting
with Sendmail in certain environments.  Possibly there are answer packets,
most likely ICMP something or other, which your firewall is blocking, and
which our TCP stack is not producing.

If this rings any bells, and if using tcpdump - ethereal - snort - etc. you
ever discover what's going on, I would be very interested to hear the
answer.

Sorry to list members who don't want to hear about Sendmail, but clearly
<pradeepbohmia>'s firewall is killing something, and I wouldn't be too
surprised to find that Mathnet's porous firewall might also be blocking too
much from our rival.

James F. Carter          Voice 310 825 2897    FAX 310 206 6673
UCLA-Mathnet;  6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA 90095-1555
Email: jimc@math.ucla.edu  http://www.math.ucla.edu/~jimc (q.v. for PGP key)

Firewall performance querry

[Pradeep Bhomia]

Hello Friends,

I have recently setup an iptables based firewall and put a Sendmail server 
behind firewall (although after facing some problems and with help from 
friends in this mailling list). Now I want to put a proxy server behind this 
same firewall. Right now the proxy server is connected directly to the 
internet. Before moving ahead I want to know what will be the load on the 
firewall. The configuration of firewall box is P4, ~1.8GHz, 256MB RAM, 
Mandrake Linux 9.1, IPTables 1.2.7 and Shorewall 1.3.14. I will be having 
aroung 300-400 concurrent users. I plan to setup NATting. Can anybody help me 
in this regard. Whether NATting will be sufficient to take care about this 
load or some other method can be used ( Total load on firewall will be some 
1000 email accounts on sendmail server and around 400 clients for web 
browsing). If anybody can direct me to some websites having some sort of 
statistics for iptables.

Thanks a lot,

Pradeep Bhomia

Re: Firewall performance querry

[Dharmendra.T]

[-- Attachment #1: Type: text/plain, Size: 1433 bytes --]

On Tue, 2003-09-09 at 11:21, Pradeep Bhomia wrote:

    Hello Friends,
    
    I have recently setup an iptables based firewall and put a Sendmail server 
    behind firewall (although after facing some problems and with help from 
    friends in this mailling list). Now I want to put a proxy server behind this 
    same firewall. Right now the proxy server is connected directly to the 
    internet. Before moving ahead I want to know what will be the load on the 
    firewall. The configuration of firewall box is P4, ~1.8GHz, 256MB RAM, 
    Mandrake Linux 9.1, IPTables 1.2.7 and Shorewall 1.3.14. I will be having 
    aroung 300-400 concurrent users. I plan to setup NATting. Can anybody help me 
    in this regard. Whether NATting will be sufficient to take care about this 
    load or some other method can be used ( Total load on firewall will be some 
    1000 email accounts on sendmail server and around 400 clients for web 
    browsing). If anybody can direct me to some websites having some sort of 
    statistics for iptables.
    
    Thanks a lot,
    
    Pradeep Bhomia
    

See basically the firewall will not process much of your traffic, which
simply forwards the packets. So the currnet configuration for
firewalling is sufficient for iptables.
And you have to use natting if you are placing the proxy server before
the firewall. 
-- 
Regards
Dharmendra.T
dharmu@nsecure.net
Linux Security and Admin

[-- Attachment #2: Type: text/html, Size: 2930 bytes --]

Re: Firewall performance querry

[Chris Brenton]

Pradeep Bhomia wrote:
> 
>  Before moving ahead I want to know what will be the load on the
> firewall. The configuration of firewall box is P4, ~1.8GHz, 256MB RAM, 
> Mandrake Linux 9.1, IPTables 1.2.7 and Shorewall 1.3.14. I will be having 
> aroung 300-400 concurrent users.

You didn't mention link speed. Really it comes down to concurrent 
sessions rather than total users that defines the amount of throughput 
you need. Knowing the link speed will help you figure out if the 
performance gate will be the firewall or the link.

With that said, I've pushed as much as 300 Mb though a similar box (1.2 
GHz w/512 MB RAM) and its worked great, ***provided*** you are not 
trying to log too much of the traffic. If you do, the log entries start 
getting truncated with only partial information getting recorded, and 
latency across the firewall starts to climb drastically. With this in 
mind, hardware with good disk and bus I/O becomes more important than 
CPU. I've found that Compaq/HP Proliant boxes outfitted with RAID seems 
to do the best job in keeping up, your mileage may vary.

<soapbox>
I *think* part of the reason why this is a problem is that iptables 
records more information about passing packets than any other packet 
filtering firewall (this is one of its strengths in my book). Recording 
more info means more of a performance hit. It would be nice if '-j LOG' 
supported a '--verbose' setting or something similar so the user could 
choose between detailed info like it products now, and minimal info more 
in lines with commercial products. That way you could at least be 
logging something at high speeds rather than nothing at all.
</soapbox>

> I plan to setup NATting. Can anybody help me 
> in this regard. Whether NATting will be sufficient to take care about this 
> load or some other method can be used ( Total load on firewall will be some 
> 1000 email accounts on sendmail server and around 400 clients for web 
> browsing).

SMTP is not too bad as its pretty efficient. HTTP is the one that kills 
you as it spawns a unique concurrent session for every object on a web 
page you try to view. For example a user visiting www.netfilter.org 
generates 6 unique sessions to view the homepage. Microsoft's homepage 
is 10 sessions, Security Focus is 70, you get the idea.

Max concurrent sessions on iptables is around 16K, but you can tweak 
this up if needed. Again, keep an eye on how much logging you are doing 
and life should be cool.

HTH,
C

Re: Firewall performance querry

[Julian Gomez]

On Tue, Sep 09, 2003 at 06:54:24AM -0400, Chris Brenton spoke thusly:

(snip lots)

>SMTP is not too bad as its pretty efficient. 

I think DJB would disagree with you ? :-)

>HTTP is the one that kills you as it spawns a unique concurrent session
>for every object on a web page you try to view. For example a user
>visiting www.netfilter.org generates 6 unique sessions to view the
>homepage. Microsoft's homepage is 10 sessions, Security Focus is 70, you
>get the idea.

If I understand you correctly, that is with HTTP/1.0 which opens up a
unique tuple per object, HTTP/1.1 uses a single tuple only.

(snip again)