ESTABILISHED connections are not that estabilished

ESTABILISHED connections are not that estabilished

[Gioele Barabucci]

In my logs I often find reports of dropped input packets from my DNS:53 or
dropped output packets generated from localhost:25 to other mail servers.

They look like these:
  iptables INPUT DROP IN=eth0 OUT= SRC=69.93.28.254 DST=myIP LEN=70 TOS=0x00
PREC=0x00 TTL=54 ID=0 DF PROTO=UDP SPT=53 DPT=4156 LEN=50

  iptables OUTPUT DROP IN= OUT=eth0 SRC=myIP DST=219.136.64.239 LEN=87
TOS=0x00 PREC=0x00 TTL=64 ID=44757 DF PROTO=TCP SPT=25 DPT=3062 WINDOW=5840
RES=0x00 ACK PSH FIN URGP=0

I thought these connections should be handled by

iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

and not dropped by the default policy. The logs show that this is not true.
Why are these packets dropped?

-- 
Gioele <dev@gioelebarabucci.com>

Re: ESTABILISHED connections are not that estabilished

[Gioele Barabucci]

Gioele Barabucci wrote:
> In my logs I often find reports of dropped input packets from my DNS:53 or
> dropped output packets generated from localhost:25 to other mail servers.
An example of the DNS issue. I think that the SMTP issue is identical.

This is what happens when I launch 

  $ dig -x 66.70.178.103
  ;; Warning: Message parser reports malformed message packet.
  ;; Truncated, retrying in TCP mode.
  ;; Connection to 209.51.143.76#53(209.51.143.76) for
103.178.70.66.in-addr.arpa. failed: connection refused.
  ;; Connection to 69.93.28.254#53(69.93.28.254) for
103.178.70.66.in-addr.arpa. failed: connection refused.

  iptables OUTPUT DROP IN= OUT=eth0 SRC=myIP DST=209.51.143.76 LEN=60
TOS=0x00 PREC=0x00 TTL=64 ID=2747 DF PROTO=TCP SPT=1025 DPT=53 WINDOW=5840
RES=0x00 SYN URGP=0 OPT (020405B40402080A00008F8F0000000001030301)
  iptables OUTPUT DROP IN= OUT=eth0 SRC=myIP DST=69.93.28.254 LEN=60
TOS=0x00 PREC=0x00 TTL=64 ID=8196 DF PROTO=TCP SPT=1026 DPT=53 WINDOW=5840
RES=0x00 SYN URGP=0 OPT (020405B40402080A00008FF40000000001030301)
  iptables OUTPUT DROP IN= OUT=eth0 SRC=myIP DST=209.51.143.76 LEN=60
TOS=0x00 PREC=0x00 TTL=64 ID=2749 DF PROTO=TCP SPT=1025 DPT=53 WINDOW=5840
RES=0x00 SYN URGP=0 OPT (020405B40402080A000090BB0000000001030301)
  iptables OUTPUT DROP IN= OUT=eth0 SRC=myIP DST=69.93.28.254 LEN=60
TOS=0x00 PREC=0x00 TTL=64 ID=8198 DF PROTO=TCP SPT=1026 DPT=53 WINDOW=5840
RES=0x00 SYN URGP=0 OPT (020405B40402080A000091200000000001030301)


but then
  $ dig netfilter.org
  ;; QUESTION SECTION:
  ;netfilter.org.                 IN      A
  
  ;; ANSWER SECTION:
  netfilter.org.          43200   IN      A       213.95.27.115

and iptables don't drop any packets.

Obviously I have these broad rules for output

iptables -A OUTPUT -p udp --dport nameserver -j ACCEPT
iptables -A OUTPUT -p tcp --dport nameserver -j ACCEPT

-- 
Gioele <dev@gioelebarabucci.com>

Re: ESTABILISHED connections are not that estabilished

[curby .]

Since no one replied yet, I'll try a few stabs at debugging.

DNS by default uses UDP for most things, so your DNS servers might
simply be rejecting TCP requests.  That said, why are they even
getting to the server and being refused there if the firewall is
dropping the packets?

You might try starting with a very simple ruleset and see if you can
pinpoint where the problem occurs, especially if this is a personal
computer and not a large installation.  For example, just allow DNS in
a stateless fashion, then introduce stateful rules.  Keep track of
packet counters in iptables as you test to see which rules fire. 
Hopefully this helps, though I'm being rather vague because I don't
know too many details and am rather new myself.

Re: ESTABILISHED connections are not that estabilished

[Gioele Barabucci]

Gioele Barabucci wrote:
> In my logs I often find reports of dropped input packets from my DNS:53 or
> dropped output packets generated from localhost:25 to other mail servers.
> Why are these packets dropped?
I attach my simple iptables rules

  iptables -F
  iptables -X
  
  echo "Default policies"
  iptables -P INPUT DROP
  iptables -P OUTPUT DROP
  iptables -P FORWARD DROP # just for fun, I don't do any routing
  
  echo "Exceptions for OUTPUT"
  iptables -A OUTPUT -o lo -j ACCEPT
  iptables -A OUTPUT -p udp --dport nameserver -j ACCEPT
  iptables -A OUTPUT -p tcp --dport nameserver -j ACCEPT
  iptables -A OUTPUT -p tcp --dport smtp -j ACCEPT
  iptables -A OUTPUT -p icmp -j ACCEPT
  iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
  
  echo "Exceptions for INPUT"
  iptables -A INPUT -i lo -j ACCEPT
  iptables -A INPUT -p tcp --dport ssh -j ACCEPT
  iptables -A INPUT -p tcp --dport smtp -j ACCEPT
  iptables -A INPUT -p tcp --dport pop3 -j ACCEPT
  iptables -A INPUT -p tcp --dport imap -j ACCEPT
  iptables -A INPUT -p tcp --dport http -j ACCEPT
  iptables -A INPUT -p icmp -j ACCEPT
  iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
  
  echo "Logging"
  iptables -A INPUT -m limit --limit 3/second --limit-burst 5 -i ! lo -j LOG
--log-level "debug" --log-ip-options --log-tcp-options --log-prefix
'iptables INPUT DROP '
  iptables -A INPUT -m limit --limit 3/second --limit-burst 5 -i ! lo -j
ULOG --ulog-prefix 'iptables INPUT DROP '
  iptables -A OUTPUT -m limit --limit 3/second --limit-burst 5 -o ! lo -j
LOG --log-level "debug" --log-ip-options --log-tcp-options --log-prefix
'iptables OUTPUT DROP '

  echo "REJECT for outgoing packets"
  iptables -A OUTPUT -j REJECT # reject, don't DROP outgoing packets


-- 
Gioele <dev@gioelebarabucci.com>

Re: ESTABILISHED connections are not that estabilished

[/dev/rob0]

Gioele Barabucci wrote:
>>In my logs I often find reports of dropped input packets from my DNS:53 or
>>dropped output packets generated from localhost:25 to other mail servers.
>>Why are these packets dropped?

I have a theory and a suggestion.

>   iptables -P OUTPUT DROP

Bad idea (see below.)

>   iptables -P FORWARD DROP # just for fun, I don't do any routing

Good idea. Even if not routing it doesn't hurt to take these out at the 
firewall level.

>   iptables -A OUTPUT -p udp --dport nameserver -j ACCEPT
>   iptables -A OUTPUT -p tcp --dport nameserver -j ACCEPT

Theory:
# iptables -vA OUTPUT -p tcp --dport nameserver
   tcp opt -- in * out *  0.0.0.0/0  -> 0.0.0.0/0  tcp dpt:42
# iptables -vA OUTPUT -p tcp --dport domain
   tcp opt -- in * out *  0.0.0.0/0  -> 0.0.0.0/0  tcp dpt:53
I cannot see into your /etc/services, but mine resolves "nameserver" to 
"42". (R.I.P., D. Adams.)

>   echo "Logging"
>   iptables -A INPUT -m limit --limit 3/second --limit-burst 5 -i ! lo -j LOG
> --log-level "debug" --log-ip-options --log-tcp-options --log-prefix
> 'iptables INPUT DROP '
>   iptables -A INPUT -m limit --limit 3/second --limit-burst 5 -i ! lo -j
> ULOG --ulog-prefix 'iptables INPUT DROP '
>   iptables -A OUTPUT -m limit --limit 3/second --limit-burst 5 -o ! lo -j
> LOG --log-level "debug" --log-ip-options --log-tcp-options --log-prefix
> 'iptables OUTPUT DROP '

Whoa, that is a lot of logging! What do you expect to find in all that? 
Why LOG and ULOG both?

>   echo "REJECT for outgoing packets"
>   iptables -A OUTPUT -j REJECT # reject, don't DROP outgoing packets

It's cleaner to limit this to "-p tcp".


Suggestion: OUTPUT filtering is a bad idea. Carefully crafted rules to 
limit what your daemons can do with -m owner might slightly enhance 
security. General OUTPUT filtering only guarantees that things won't 
work well.

Why are you doing it? What is the benefit you think you are getting? 
This sounds like a single user machine, from the fact that there's no 
routing. Who are you limiting with OUTPUT rules? Yourself?

I would do OUTPUT -p ACCEPT and eliminate the OUTPUT rules. I'd also do 
away with the logging, or at least tightly curtail it. I usually only 
log for specific short-term reasons (troubleshooting a problem.) I'd 
include a "-m state --state INVALID -j DROP" rule for good measure. 
Finally, I'd move your --state rules to the top.

At that point everything would be working as you expect, and you would 
have a very tight firewall.
-- 
     mail to this address is discarded unless "/dev/rob0"
     or "not-spam" is in Subject: header

Re: ESTABILISHED connections are not that estabilished

[Gioele Barabucci]

/dev/rob0 wrote:
> Gioele Barabucci wrote:
> Theory:
> # iptables -vA OUTPUT -p tcp --dport nameserver
>    tcp opt -- in * out *  0.0.0.0/0  -> 0.0.0.0/0  tcp dpt:42
> # iptables -vA OUTPUT -p tcp --dport domain
>    tcp opt -- in * out *  0.0.0.0/0  -> 0.0.0.0/0  tcp dpt:53
> I cannot see into your /etc/services, but mine resolves "nameserver" to
> "42". (R.I.P., D. Adams.)
Yes, you're right. Lucky nameserver/udp is an alias for domain/udp so some
connections went without problem.
Now that I fixed these rules, the "dig -x" problem is solved, but I can
still see DROPped packed for INPUT and OUTPUT.

>>   echo "Logging"
> Whoa, that is a lot of logging! What do you expect to find in all that?
> Why LOG and ULOG both?
Debugging stage :)

> Suggestion: OUTPUT filtering is a bad idea.
>
> Why are you doing it? What is the benefit you think you are getting?
> This sounds like a single user machine, from the fact that there's no
> routing. Who are you limiting with OUTPUT rules? Yourself?
This is a public server and I fear that some of its daemons could be used to
gain user access to the machine. None of my daemon runs as root, so if
someone get in, it will get only user privs.
I think that DROPping OUTPUT ensures that nobody will ever open the server
to the outside or use the server to connect somewhere. Am I wrong? I'm
pretty new to server management so I want to be sure that I am playing
safe.

-- 
Gioele <dev@gioelebarabucci.com>

Re: ESTABILISHED connections are not that estabilished

[/dev/rob0]

Gioele Barabucci wrote:
>>Suggestion: OUTPUT filtering is a bad idea.
>>
>>Why are you doing it? What is the benefit you think you are getting?
>>This sounds like a single user machine, from the fact that there's no
>>routing. Who are you limiting with OUTPUT rules? Yourself?
> 
> This is a public server and I fear that some of its daemons could be used to
> gain user access to the machine. None of my daemon runs as root, so if
> someone get in, it will get only user privs.
> I think that DROPping OUTPUT ensures that nobody will ever open the server
> to the outside or use the server to connect somewhere. Am I wrong? I'm

Not entirely. I did say (in text you snipped) this: "Carefully crafted 
rules to limit what your daemons can do with -m owner might slightly 
enhance security."

And yes, you're going to slow down an attacker with that if he's gotten 
shell access. Will it stop him from getting out? I doubt it. You allow 
ICMP out, and RELATED,ESTABLISHED in and out. I'm not sure how RELATED 
is figured (I hope someone will fill us in on the details) but I bet a 
"ping -q host.fqdn &> /dev/null" would allow outbound access to any 
services on host.fqdn. And anything they might need from the host they 
rode in on is accessible: if not RELATED, through ssh.

Furthermore, having a hostile user with shell access doesn't prevent 
privilege escalation exploits. Twice I've been asked by machine owners 
(with shell access) to restore their lost root access. Each time it was 
simple and straightforward. Once, I guessed the root password. The other 
time, I dropped to a shell from a SUID binary.

That said, your garden variety script kiddie isn't going to work that 
hard at it, and attack slowdowns often equate to security. I once left a 
guessable password on a guessable account name on a small virtual host 
(user-mode Linux.) Sure enough, the ssh bots got in. The lastlog caused 
the limited disk space to fill up and the kernel process crashed. I 
examined the system from the host and determined that no privilege 
escalation had occurred. They logged in and didn't know what to do 
because there was no apache, PHP, sendmail: the commonly used tools.

Sure, go ahead and work on it. Eventually I might try experimenting with 
some OUTPUT rules too. It won't hurt if you can allow your daemons to do 
only what they need -- but allow them *everything* they need. You might 
want to use a -m owner rule to allow your own user account out, at least 
while you're logged in.

But a better approach overall ... and this is what I do ... is to 
restrict services which might conceivably (or directly) provide a shell. 
SSH should only allow in secret keys, no password authentication. IMAP 
and POP3 should use virtual accounts and not provide access to the $HOME 
of the system user for the virtual accounts. (And do require secure 
authentication ... TLS, or force the use of IMAPS/POP3S.) It might not 
hurt to disable your apache userdir_module. (There might be other ways 
to harden apache too ... I don't specialise in that.)
-- 
     mail to this address is discarded unless "/dev/rob0"
     or "not-spam" is in Subject: header