SSH Brute Force not working (any longer)

SSH Brute Force not working (any longer)

[curby .]

I know enough to realize that it didn't stop working just because it
hates me: most likely, i changed something in the configuration or
something to make it stop working.  However, I can't think of anyting
I could have done to two different machines to make them stop working
in exactly that same way.  Basically, every SSH attempt skips the
return to the input chain and jumps directly to log/drop or
log/tarpit.  I've tried increasing the --hitcount, tried rebooting or
otherwise restarting the network service, tried using a different
--name in recet, etc.

I have tried all the following scripts (I can paste their contents in
if requested but it would be rather annoying because of their combined
length) with no success.  Any ideas on what might have happened? 
Kernel version and iptables version didn't change, the recent module
loads with no errors, and all the iptables commands to insert rules
complete with no errors.

Links to variations that I tried with no success:

http://curby.net/doc/testwall1
http://curby.net/doc/testwall2
http://curby.net/doc/testwall3
http://curby.net/doc/testwall4

Hopefully it's a really simple typo or something, though with all the
variations I've tried, I'm starting to doubt that simple explanation. 
What might the problem be?  Thanks!

Re: SSH Brute Force not working (any longer)

[Nagy Zoltan]

hi

> Hopefully it's a really simple typo or something, though with all the
> variations I've tried, I'm starting to doubt that simple explanation.
> What might the problem be?  Thanks!

i've read 1-3 the files, i think your problem is a skip of the "update" (my english is really bad ;)
because your --update lines have other condition like:seconds and rate
the attacking machine when it begins to send those packets, recent knows nothing about him, it will be enlist with a --set match
and then you don't update it's record with a single --update, so it can't reach the hitcount :)

i've wrote another one of these recent based flood protectors ;)
i use this to prevent syn scans/icmp floods to go out from our lan or come in - this is a sort of "make the days of the conntrack table a bit easier, by droping the shit" :)
a line like: iptables -t raw -A PREROUTING -i $lan_if -j OUT_CHECK should prevent scans from your lan to go out.


message-u       "OUT_CHECK..."
        ipt="$iptables -t raw -A OUT_CHECK"
        $ipt    -p icmp                         -j OUT_F_PRO
        $ipt    -p tcp  --tcp-flags ALL SYN     -j OUT_F_PRO
        $ipt    -j ACCEPT

message-u       "OUT_F_PRO..."
        ipt="$iptables -t raw -A OUT_F_PRO"
        $ipt    -m recent --name out --rcheck   -j OUT_F_CHK
        $ipt    -m recent --name out --set      -j ACCEPT

message-u       "OUT_F_CHK..."
        ipt="$iptables -t raw -A OUT_F_CHK"
        $ipt -m recent --name out --rcheck --seconds 3 --hitcount 15    -j OUT_XDROP		
	# this is the currently used version, it forces that a client can't open more than 5 new tcp connections/sec and icmp
#       $ipt -m recent --name out --update --seconds 3 --hitcount 15    -j OUT_XDROP		
	# this version is very aggressive, it eats http traffic too ;) because of the many images/etc
	# it can be combined with small ipset, with ports like 22,135,139 that will prevent ssh bf too ;)
        $ipt -m recent --name out --update --seconds 30                 -j ACCEPT
        $ipt -m recent --name out --remove                              -j ACCEPT

message-u       "OUT_XDROP..."
        ipt="$iptables -t raw -A OUT_XDROP"
        $ipt    -s $e_n -m account --aname xdrop_e --aaddr $e_n -j DROP
        $ipt    -s $b_n -m account --aname xdrop_b --aaddr $b_n -j DROP
        #       miracle situation...in this case: flood klogd ;)
        $ipt    -m limit --limit 1/sec  -j LOG --log-prefix raw/OUT_XDROP
        $ipt                            -j DROP


kirk

Re: SSH Brute Force not working (any longer)

[Marius Mertens]

Hi everybody,

On Wednesday, July 27, 2005 6:36 PM,
curby . wrote:

> I know enough to realize that it didn't stop working just because it
> hates me: most likely, i changed something in the configuration or
> something to make it stop working.  However, I can't think of anyting
> I could have done to two different machines to make them stop working
> in exactly that same way.  Basically, every SSH attempt skips the
> return to the input chain and jumps directly to log/drop or
> log/tarpit.  I've tried increasing the --hitcount, tried rebooting or
> otherwise restarting the network service, tried using a different
> --name in recet, etc.

Well, from what your wrote I guess it suddenly ceased working. Don't worry, 
you are definitely alone. Since you mentioned that it looks like it was 
skipping the whole rule that should RETURN "good" packets, it sounds like 
you have encountered the same thing that happened to me.
I only had a closer look at your first and second ruleset example, and if I 
understand it right, the first example should always return the packets, 
since RETURN is the first rule (inverted rcheck) in SSH_Brute_Force, so the 
recent list should never receive an entry.
And I guess that in your second example the DROP rule (positive update 
check) always matches and therefore always drops, though the way I 
understand the rules, they should never match, because again the list entry 
has never been set.
If my above two guesses about your ruleset behaviour are true, then you have 
the same problem I had. Suddenly they ceased working, and the inverted match 
was always false (where it should have matched) and an added non-inverted 
match for testing with the same criteria did match, though it should have 
been the other way round.
I also tried roughly the same debugging you did, with no results. It 
happened with both a 2.6.12.1 kernel.org and a 2.6.11 debian kernel, reboot 
did not change anything, and I was unable to find out where the problem was.
What I found out was, that low hitcounts (20 or smaller) always matched a 
positive match, while high hitcounts (21 and above) never did, even after 
pushing a sufficiently large number of packets through that ruleset to have 
more than 21 entries within the last minute. So

iptables -A SSH_Brute_Force -m recent --name SSH ! --rcheck --seconds 
0  --hitcount 20  -j RETURN

never returned. While

iptables -A SSH_Brute_Force -m recent --name SSH ! --rcheck --seconds 
0  --hitcount 21  -j RETURN

always returned, regardless how many packets have been processed by that 
rule within the last minute. Maybe there is a similar "critical" number of 
hits for you.
So sorry, no idea how to get rid of it, but I thought you might be 
interested in knowing that you are not the only one encountering that 
problem. The only positive about it: I finally gave up debugging it, and as 
silently as it came, the problem disappeared. Everything is working fine 
again. With a still unchanged ruleset of course ;-)
So I hope (and assume) it will silently disappear for you, too, though I 
would give quite a lot to actually find out what the hell went wrong in the 
3 days it was not working.

Good luck,

Marius 

Re: SSH Brute Force not working (any longer)

[Marius Mertens]

On Wednesday, July 27, 2005 9:14 PM,
Marius Mertens wrote:

> [...]
> Well, from what your wrote I guess it suddenly ceased working. Don't
> worry, you are definitely alone. Since you mentioned that it looks
> [...]

Ahem... NOT alone I mean ;-) Too many negations and inverted booleans today. 
Btw. it is still working for me and did not reappear so far.

Marius 

Re: SSH Brute Force not working (any longer)

[curby .]

On 7/27/05, Nagy Zoltan <kirk@elte.hu> wrote:
> i've read 1-3 the files, i think your problem is a skip of the "update" (my english is really bad ;)
> because your --update lines have other condition like:seconds and rate
> the attacking machine when it begins to send those packets, recent knows nothing about him, it will be enlist with a --set match
> and then you don't update it's record with a single --update, so it can't reach the hitcount :)

2 and 3 have --set during the rule in the INPUT chain that jumps to
the custom chain.

On 7/27/05, Marius Mertens <marius.mertens@gmx.de> wrote:
> problem. The only positive about it: I finally gave up debugging it, and as
> silently as it came, the problem disappeared. Everything is working fine
> again. With a still unchanged ruleset of course ;-)

If this is not the result of something dumb we're doing, there's
probably a bug that should be squirreled out.  Intermittent problems
in security software are bad. Still, iptables has been so stable that
I'm thinking it's likely my fault. =P

I'll keep poking at it when I have a moment.

Re: SSH Brute Force not working (any longer)

[Marius Mertens]

On Thursday, July 28, 2005 12:50 AM,
curby . wrote:

> 2 and 3 have --set during the rule in the INPUT chain that jumps to
> the custom chain.

Ooops, blind me, I completely overlooked that. But it should work anyway.

> If this is not the result of something dumb we're doing, there's
> probably a bug that should be squirreled out.  Intermittent problems
> in security software are bad. Still, iptables has been so stable that
> I'm thinking it's likely my fault. =P

Nope, guess it's not your fault ;-) I found something with google, and it 
looks relevant to our problem. It also explains why my reboot didn't seem to 
help first, but then it magically started working again:
http://patchwork.netfilter.org/netfilter-devel/patch.pl?id=2587

Marius 

Re: SSH Brute Force not working (any longer)

[Taylor, Grant]

After having messed with the SSH_Brute_Force script(s) extensively I think I can help you with this.

<testwall1>
iptables -N SSH_Brute_Force
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -j SSH_Brute_Force
#following, adapted from grant's post, lets everything through
iptables -A SSH_Brute_Force -m recent --name SSH ! --rcheck --seconds 60  --hitcount 4  -j RETURN
#following, from grant's original post, gives iptables v1.2.9: Unknown arg `4'
#iptables -A SSH_Brute_Force -m recent --name SSH ! --rcheck --seconds 60 -m recent --hitcount 4 --set --name SSH -j RETURN
iptables -A SSH_Brute_Force -m recent --name SSH --update
iptables -A SSH_Brute_Force -j LOG --log-prefix "IPT_TEST SSH DROP:  "
iptables -A SSH_Brute_Force -p tcp -j TARPIT
</testwall1>

After a reboot I'm betting that your testwall1 script will never NOT RETURN packets as you are never really doing a SET of the information in the recent list and thus never having a packet counter get to 4 where where the inverse match would fail and thus not return.  In my experience "--update" will only increment a value that already is set to something.  Conversely "--rcheck" has to have a value to check against.

<testwall2>
/sbin/iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --set --name SSHBF
/sbin/iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --name SSHBF -j LOG --log-prefix "IPT_TEST SSH DROP: "
/sbin/iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --name SSHBF -j DROP
</testwall2>

Something tells me that you *might* get one packet through this filter but it would depend on what else you have done in the last 60 seconds.  When the packet is indeed in state NEW it will come in to the first rule and have it's value set in the SSHBF recent list.  Subsequently the packet will have it's entry in the SSHBF recent list updated (incremented by 1) while evaluating the 2nd rule.  Seeing as how this packet should now have it's recent list entry counter at 2 it will not match and thus not log as the recent match should fail but still update the recent list.  Finally the packet will have it's entry in the SSHBF recent list updated while evaluating the 3rd rule.  Seeing as how the packet should now have it's recent list entry counter at 3 it will not match and thus not be dropped.  However this will all depend on the fact that there are no entries in the recent list for any give
 n packet before it starts because if it even has a value of 1 it will be dropped by the ti
me that it reaches the 3rd rule.  I suspect that you might be able to get one packet and thus connection through this filter on the first pass but have a lot of trouble with subsequent connections.

<testwall3>
iptables -N SSH_Brute_Force
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --name SSH --set --rsource -j SSH_Brute_Force
iptables -A SSH_Brute_Force -m recent ! --rcheck --seconds 60 --hitcount 3 --name SSH --rsource -j RETURN 
iptables -A SSH_Brute_Force -j LOG --log-prefix "IPT_TEST SSH drop:  "
iptables -A SSH_Brute_Force -p tcp -j TARPIT
</testwall3>

I would expect that this script did work for you.  From the fact that it is the last testwall I'm going to assume that it did for a while.  The only thing that comes to mind as to why this would not work would be if for some reason the packet(s) started matching a state of NEW again.  The only reason that I can think of that your packet would erroneously be showing up as a state of NEW again would be if for some reason your conntrack table was full.  Your size conntrack table depends on the amount of memory that you have in the router and thus could possibly be over run as the conntrack table is a FIFO as far as I know.  If this is the case you should have seen some logs in dmesg regarding the conntrack table being full.  Sorry I don't remember the exact wording of the error as it's been over a year sense I dealt with that on a different router.

I don't know what to think about the patch (http://patchwork.netfilter.org/netfilter-devel/patch.pl?id=2587) that Marius referenced as I have not experienced this issue and thus have not had to address it, though it does sound plausible.

One thing that I can recommend is that you check the /proc/net/ipt_recent/<recent list name> file(s) for the existence of a line matching the IP (and possibly the TTL) that you are having problem with.  If there is a line in this file(s) then your IP is in that recent list with a hit count of the number of packets that are listed on the line.  Or at least this is how I understand it.

Something else that I did that you might find useful was to have a rule something like "-m recent --name SSH --remove" to make sure that the IP that you are testing from is NOT in the recent list so you can fiddle with things and then get back to a clean slate with out having to reboot.

If you would care to look at the version of the script that I am running take a look at this previous post https://lists.netfilter.org/pipermail/netfilter/2005-June/061362.html



Grant. . . .

curby . wrote:
> I know enough to realize that it didn't stop working just because it
> hates me: most likely, i changed something in the configuration or
> something to make it stop working.  However, I can't think of anyting
> I could have done to two different machines to make them stop working
> in exactly that same way.  Basically, every SSH attempt skips the
> return to the input chain and jumps directly to log/drop or
> log/tarpit.  I've tried increasing the --hitcount, tried rebooting or
> otherwise restarting the network service, tried using a different
> --name in recet, etc.
> 
> I have tried all the following scripts (I can paste their contents in
> if requested but it would be rather annoying because of their combined
> length) with no success.  Any ideas on what might have happened? 
> Kernel version and iptables version didn't change, the recent module
> loads with no errors, and all the iptables commands to insert rules
> complete with no errors.
> 
> Links to variations that I tried with no success:
> 
> http://curby.net/doc/testwall1
> http://curby.net/doc/testwall2
> http://curby.net/doc/testwall3
> http://curby.net/doc/testwall4
> 
> Hopefully it's a really simple typo or something, though with all the
> variations I've tried, I'm starting to doubt that simple explanation. 
> What might the problem be?  Thanks!