Rate-limiting to halt brute-force attack

Rate-limiting to halt brute-force attack

[Dimitri Yioulos]

Hi, folks.

 A few days ago, a major brute-force attack was launched 
against our (sendmail) mail server. It looks like a bot is 
aiming lots of zombies at us. Here's how OSSEC hids reports 
an attempt from one of the zombies:

OSSEC HIDS Notification.
2012 Nov 13 09:08:16

Received From: (plymouth) 192.168.1.2->/var/log/messages
Rule: 40111 fired (level 10) -> "Multiple authentication 
failures."
Portion of the log(s):

Nov 13 09:07:44 plymouth ipop3d[29926]: Login failed 
user=hod auth=hod host=201-93-132-240.dsl.telesp.net.br 
[201.93.132.240]
Nov 13 09:07:44 plymouth ipop3d[29925]: Login failed 
user=lee auth=lee host=201-93-132-240.dsl.telesp.net.br 
[201.93.132.240]
~
~

To remediate, I've put fail2ban in place on the mail server, 
and it's working. However, the attacks are still beating at 
the door, and it's significantly increased the load on the 
mail server . I'm now thinking of adding rules to our 
iptables/Netfilter firewall to rate-limit the brute-force 
connections. The rules I'd add are these:

iptables -A INPUT -p tcp --dport 110 -m state --state NEW -m 
recent --set

iptables -A INPUT -p tcp --dport 110 -m state --state NEW -m 
recent --update --seconds 15 --hitcount 3 -j DROP

As the mail server sits in a DMZ, and packets are forwarded 
to it, is the INPUT chain the best place to put these 
rules, or should they go in the FORWARD chain (with 
appropriate modifications)?

Of course, I don't want to stop legitimate mail. Is this the 
best course of action?

Thanks.

Dimitri

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

Re: Rate-limiting to halt brute-force attack

[/dev/rob0]

On Fri, Nov 16, 2012 at 09:22:35AM -0500, Dimitri Yioulos wrote:
>  A few days ago, a major brute-force attack was launched 
> against our (sendmail) mail server. It looks like a bot is 
> aiming lots of zombies at us. Here's how OSSEC hids reports 
> an attempt from one of the zombies:
> 
> OSSEC HIDS Notification.
> 2012 Nov 13 09:08:16
> 
> Received From: (plymouth) 192.168.1.2->/var/log/messages
> Rule: 40111 fired (level 10) -> "Multiple authentication 
> failures."
> Portion of the log(s):
> 
> Nov 13 09:07:44 plymouth ipop3d[29926]: Login failed 
> user=hod auth=hod host=201-93-132-240.dsl.telesp.net.br 
> [201.93.132.240]

This is not Sendmail. This is ipop3d.

> Nov 13 09:07:44 plymouth ipop3d[29925]: Login failed 
> user=lee auth=lee host=201-93-132-240.dsl.telesp.net.br 
> [201.93.132.240]

Yep, this is a spammer looking for credentials for SMTP AUTH, 
probably, so it eventually could be targeted at Sendmail.

> To remediate, I've put fail2ban in place on the mail server, 
> and it's working. However, the attacks are still beating at 
> the door, and it's significantly increased the load on the 
> mail server . I'm now thinking of adding rules to our 
> iptables/Netfilter firewall to rate-limit the brute-force 
> connections. The rules I'd add are these:
> 
> iptables -A INPUT -p tcp --dport 110 -m state --state NEW -m 
> recent --set
> 
> iptables -A INPUT -p tcp --dport 110 -m state --state NEW -m 
> recent --update --seconds 15 --hitcount 3 -j DROP

First, I would not apply this only to POP3 (a protocol which should 
have died out 10+ years ago! IMAP can do it all, and better!) I'd 
apply this to "-m multiport --dports 110,143,465,587,993,995" to 
cover all the potential attacks, omitting any of those that you're 
not providing service on.

No, 25 is not in that list. I'm not sure how safely to limit 
connections on 25, since MTAs are automated processes. Normal 
non-spam MUAs are driven by a human.

Second, I'm not sure that 3 in 15 seconds is safe. MUAs are semi- 
automated, and it's possible that a real user could trigger that. I 
might try something like 5 in 10 seconds, and a secondary check for 
more, maybe 10 in 30 seconds.

> As the mail server sits in a DMZ, and packets are forwarded 
> to it, is the INPUT chain the best place to put these 
> rules, or should they go in the FORWARD chain (with 
> appropriate modifications)?

The mailing list is not a substitute for the man page. Packets which 
are forwarded through a host do not hit that host's INPUT chain. 
Please do take the time to learn what each built-in chain is for.

> Of course, I don't want to stop legitimate mail. Is this the 
> best course of action?

Mail exchange is done with SMTP on the "smtp" port, 25. POP3 is a 
protocol for users' access to a mailbox.

http://en.wikipedia.org/wiki/Post_Office_Protocol
-- 
  http://rob0.nodns4.us/ -- system administration and consulting
  Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:

Re: Rate-limiting to halt brute-force attack

[Dimitri Yioulos]

On Friday 16 November 2012 1:01:50 pm /dev/rob0 wrote:
> On Fri, Nov 16, 2012 at 09:22:35AM -0500, Dimitri Yioulos 
wrote:
> >  A few days ago, a major brute-force attack was
> > launched against our (sendmail) mail server. It looks
> > like a bot is aiming lots of zombies at us. Here's how
> > OSSEC hids reports an attempt from one of the zombies:
> >
> > OSSEC HIDS Notification.
> > 2012 Nov 13 09:08:16
> >
> > Received From: (plymouth)
> > 192.168.1.2->/var/log/messages Rule: 40111 fired (level
> > 10) -> "Multiple authentication failures."
> > Portion of the log(s):
> >
> > Nov 13 09:07:44 plymouth ipop3d[29926]: Login failed
> > user=hod auth=hod host=201-93-132-240.dsl.telesp.net.br
> > [201.93.132.240]
>
> This is not Sendmail. This is ipop3d.
>
> > Nov 13 09:07:44 plymouth ipop3d[29925]: Login failed
> > user=lee auth=lee host=201-93-132-240.dsl.telesp.net.br
> > [201.93.132.240]
>
> Yep, this is a spammer looking for credentials for SMTP
> AUTH, probably, so it eventually could be targeted at
> Sendmail.
>
> > To remediate, I've put fail2ban in place on the mail
> > server, and it's working. However, the attacks are
> > still beating at the door, and it's significantly
> > increased the load on the mail server . I'm now
> > thinking of adding rules to our iptables/Netfilter
> > firewall to rate-limit the brute-force connections. The
> > rules I'd add are these:
> >
> > iptables -A INPUT -p tcp --dport 110 -m state --state
> > NEW -m recent --set
> >
> > iptables -A INPUT -p tcp --dport 110 -m state --state
> > NEW -m recent --update --seconds 15 --hitcount 3 -j
> > DROP
>
> First, I would not apply this only to POP3 (a protocol
> which should have died out 10+ years ago! IMAP can do it
> all, and better!) I'd apply this to "-m multiport
> --dports 110,143,465,587,993,995" to cover all the
> potential attacks, omitting any of those that you're not
> providing service on.
>
> No, 25 is not in that list. I'm not sure how safely to
> limit connections on 25, since MTAs are automated
> processes. Normal non-spam MUAs are driven by a human.
>
> Second, I'm not sure that 3 in 15 seconds is safe. MUAs
> are semi- automated, and it's possible that a real user
> could trigger that. I might try something like 5 in 10
> seconds, and a secondary check for more, maybe 10 in 30
> seconds.
>
> > As the mail server sits in a DMZ, and packets are
> > forwarded to it, is the INPUT chain the best place to
> > put these rules, or should they go in the FORWARD chain
> > (with appropriate modifications)?
>
> The mailing list is not a substitute for the man page.
> Packets which are forwarded through a host do not hit
> that host's INPUT chain. Please do take the time to learn
> what each built-in chain is for.
>
> > Of course, I don't want to stop legitimate mail. Is
> > this the best course of action?
>
> Mail exchange is done with SMTP on the "smtp" port, 25.
> POP3 is a protocol for users' access to a mailbox.
>
> http://en.wikipedia.org/wiki/Post_Office_Protocol
> --
>   http://rob0.nodns4.us/ -- system administration and
> consulting Offlist GMX mail is seen only if "/dev/rob0"
> is in the Subject: --
> To unsubscribe from this list: send the line "unsubscribe
> netfilter" in the body of a message to
> majordomo@vger.kernel.org More majordomo info at 
> http://vger.kernel.org/majordomo-info.html


Thanks very much.  Very helpful information.  And, apologies 
for not using the man page.  I wasn't trying to be lazy.

Dimitri

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

Re: Rate-limiting to halt brute-force attack

[/dev/rob0]

On Fri, Nov 16, 2012 at 01:13:23PM -0500, Dimitri Yioulos wrote:
[snip]
> Thanks very much.  Very helpful information.  And, apologies 
> for not using the man page.  I wasn't trying to be lazy.

No problem, you're welcome. I will be interested to hear your 
results, such as if you accidentally block any of your users.
-- 
  http://rob0.nodns4.us/ -- system administration and consulting
  Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:

Re: Rate-limiting to halt brute-force attack

[Emilio Lazo Zaia]

In order to block 3 SSH attempts in 40 seconds, I'm doing the following:

First, this will syslog all silently dropped SSH attempts (after third 
in 40 seconds):
# iptables -A LOG-BlockdSSH -m recent --rcheck --seconds 40 --hitcount 3 
--name BlockdSSH --rsource -j LOG --log-prefix "iptables/BlockdSSH: " 
--log-tcp-options --log-ip-options

But LOG chain passes the packet to the following rule, so we must drop it:
# iptables -A LOG-BlockdSSH -m recent --rcheck --seconds 40 --hitcount 3 
--name BlockdSSH --rsource -j DROP

This registers the IP address in a set of recent module:
# iptables -A BlockdSSH -m recent --set --name BlockdSSH --rsource -j DROP

Any new SSH attempt from that IP address will reset the counter to 10 
days (--update), and also log that packet:
# iptables -A NoJodan -p tcp -m tcp --dport 22 -m recent --update 
--seconds 864000 --name BlockdSSH --rsource -j LOG-BlockdSSH

This creates the set "SSH" containing all SSH connections, regardless 
they are "good" or not:
# iptables -A NoJodan -p tcp -m tcp --dport 22 -m recent --set --name 
SSH --rsource

If the limit is reached, this rule chains to BlockdSSH; that will LOG 
and block any connection from that IP address:
# iptables -A NoJodan -p tcp -m tcp --dport 22 -m recent --update 
--seconds 40 --hitcount 3 --name SSH --rsource -j BlockdSSH

You must create these chains and make all new packets enters into 
NoJodan chain.

For example, the first SSH packet will not match first rule of NoJodan 
because the recent extension hasn't that IP address in the BlockdSSH 
set, so the packet will continue and will not be diverted to 
LOG-BlockdSSH. Second rule will match and the IP address is registered 
in SSH set in recent. Third rule will not match because is the first 
packet and not the third in 40 sec.

Second packet (seconds later) also will not match first NoJodan rule, 
will be registered in the SSH set, and also third rule does not affect him.

Third packet will not match first rule because it isn't registered in 
BlockdSSH. This packet will match the second rule but also the third 
rule in which the limit is 3/40sec. That packet will go to iptables 
BlockdSSH chain. In this chain, that will be registered in the set named 
"BlockdSSH" because it reached the limit (I did choose the same name as 
the iptables chain: BlockdSSH), so now this IP address is the the black 
list and the packet is dropped.

Fourth packet will match the first rule in NoJodan, because this IP 
address was registered in the BlockdSSH set (black list set) in recent 
and will be forwarded to LOG-BlockdSSH chain. In this chain the packet 
is being logged and dropped (first and second rule).

They must wait 10 days or change their IP address to have a respond from 
SSH server.

I don't know why I'm doing that BlockdSSH chain will target to DROP 
instead of LOG-BlockdSSH. Surely these rules may be improved.

You may adapt this idea to your ports and times. Also this may be used 
to block port scanning (modifying LOG chain to not log every packet), 
ping flood, etc.

Regards.


On 16.11.2012 09:52, Dimitri Yioulos wrote:
> Hi, folks.
>
>   A few days ago, a major brute-force attack was launched
> against our (sendmail) mail server. It looks like a bot is
> aiming lots of zombies at us. Here's how OSSEC hids reports
> an attempt from one of the zombies:
>
> OSSEC HIDS Notification.
> 2012 Nov 13 09:08:16
>
> Received From: (plymouth) 192.168.1.2->/var/log/messages
> Rule: 40111 fired (level 10) ->  "Multiple authentication
> failures."
> Portion of the log(s):
>
> Nov 13 09:07:44 plymouth ipop3d[29926]: Login failed
> user=hod auth=hod host=201-93-132-240.dsl.telesp.net.br
> [201.93.132.240]
> Nov 13 09:07:44 plymouth ipop3d[29925]: Login failed
> user=lee auth=lee host=201-93-132-240.dsl.telesp.net.br
> [201.93.132.240]
> ~
> ~
>
> To remediate, I've put fail2ban in place on the mail server,
> and it's working. However, the attacks are still beating at
> the door, and it's significantly increased the load on the
> mail server . I'm now thinking of adding rules to our
> iptables/Netfilter firewall to rate-limit the brute-force
> connections. The rules I'd add are these:
>
> iptables -A INPUT -p tcp --dport 110 -m state --state NEW -m
> recent --set
>
> iptables -A INPUT -p tcp --dport 110 -m state --state NEW -m
> recent --update --seconds 15 --hitcount 3 -j DROP
>
> As the mail server sits in a DMZ, and packets are forwarded
> to it, is the INPUT chain the best place to put these
> rules, or should they go in the FORWARD chain (with
> appropriate modifications)?
>
> Of course, I don't want to stop legitimate mail. Is this the
> best course of action?
>
> Thanks.
>
> Dimitri
>