SynFloods and CPU usage with and without iptables. Confused!

SynFloods and CPU usage with and without iptables. Confused!

[Alex Flex]

Hello Netfilter,

Ive been receiving lately two types of syn floods on an Intel Xeon 
2.4ghz + 4GB machine exclusively dedicated for this and the findings 
have me very confused:
  I have syn cookies enabled and checked to be working as per syslog.
This machine has a 10gigabit uplink so I know that networking isnt a 
bottleneck here (bandwith or router hardware based).

SCENARIO 1: the first attack was: 105mbits @ 330,000 pps and it brought 
the machine to 100% CPU usage and over 50% packetloss Load average 12. 
At that time it had a simple iptables script that that had less then 5 
blacklists of port 80 ips and then a ACCEPT On port 80, nothing fancy. I 
disabled iptables and load average went down immediately to 8 but there 
was still high packet loss so basically we where DoSed efficiently.

SCENARIO 2: After that the attacker sent only a 30mbit synflood @ 70,000 
pps .. Now i had less packet loss, and interestingly with iptables 
enabled it would create almost immediate packetloss. At this time I 
tried to explore installing conntrack-tools information about the state 
table. conntrack said that with iptables enabled and syncookies the 
maximum entries where 1300 ONLY... and a CPU usage reported by HTOP of 
40% on SI. After that I decided to drop iptables all together and 
immediately port 80 started flowing with normal traffic (we have less 
than 1mbit clean traffic) . No packetloss was present, because iptables 
was disabled conntrack did not report any entries and netstat-na |wc -l 
reported less than 300.

Questions:

a.) Can anybody suggest why there is so much CPU overhead when iptables 
is turned on and dealing with such PPS? Is this normal? Usually what CPU 
usage does a syn flood cookie enabled take?

b.) Is there a chance that the attacker exausted something else iam not 
seeing?


Thanks for the help guys

Alex

Re: SynFloods and CPU usage with and without iptables. Confused!

[hdemir]

Hi Alex,

I have a that problem also.I solved with CPU Affinity and disabled all the
power properties of the hardware (s1, s2, s3). But with the syn attack the
problem is that conntrack number increasing too fast. I have a max value
of 10 million and it fills up about less than 10 min.

ý also wonder about how to stop ddos of conntrack max value. Perhaps
somebody shoul implement conntrack without syn, syn-ack. I think Junioer
has this feature. Like having syncoockies in conntrack.


Regards.

hdemir.

> Hello Netfilter,
>
> Ive been receiving lately two types of syn floods on an Intel Xeon
> 2.4ghz + 4GB machine exclusively dedicated for this and the findings
> have me very confused:
>   I have syn cookies enabled and checked to be working as per syslog.
> This machine has a 10gigabit uplink so I know that networking isnt a
> bottleneck here (bandwith or router hardware based).
>
> SCENARIO 1: the first attack was: 105mbits @ 330,000 pps and it brought
> the machine to 100% CPU usage and over 50% packetloss Load average 12.
> At that time it had a simple iptables script that that had less then 5
> blacklists of port 80 ips and then a ACCEPT On port 80, nothing fancy. I
> disabled iptables and load average went down immediately to 8 but there
> was still high packet loss so basically we where DoSed efficiently.
>
> SCENARIO 2: After that the attacker sent only a 30mbit synflood @ 70,000
> pps .. Now i had less packet loss, and interestingly with iptables
> enabled it would create almost immediate packetloss. At this time I
> tried to explore installing conntrack-tools information about the state
> table. conntrack said that with iptables enabled and syncookies the
> maximum entries where 1300 ONLY... and a CPU usage reported by HTOP of
> 40% on SI. After that I decided to drop iptables all together and
> immediately port 80 started flowing with normal traffic (we have less
> than 1mbit clean traffic) . No packetloss was present, because iptables
> was disabled conntrack did not report any entries and netstat-na |wc -l
> reported less than 300.
>
> Questions:
>
> a.) Can anybody suggest why there is so much CPU overhead when iptables
> is turned on and dealing with such PPS? Is this normal? Usually what CPU
> usage does a syn flood cookie enabled take?
>
> b.) Is there a chance that the attacker exausted something else iam not
> seeing?
>
>
> Thanks for the help guys
>
> Alex
>
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: SynFloods and CPU usage with and without iptables. Confused!

[Steve Kann]

hdemir,

	If you're tracking incoming connections using conntrack, you are really defeating the whole purpose of syncookies (i.e. handling incoming SYN in a stateless way).   You can set up a iptables firewall which is either totally stateless, or at least does not use state for the open TCP ports which can be SYN flooded.   

	However, even if you do that, you're probably not out of the woods.   Even taking ip_conntrack state out of the equation, I still haven't been able to get linux SYN Cookies to scale very well;  it still fails around 100-150,000 pps.

-SteveK


On May 4, 2013, at 5:39 PM, hdemir@metu.edu.tr wrote:

> Hi Alex,
> 
> I have a that problem also.I solved with CPU Affinity and disabled all the
> power properties of the hardware (s1, s2, s3). But with the syn attack the
> problem is that conntrack number increasing too fast. I have a max value
> of 10 million and it fills up about less than 10 min.
> 
> ý also wonder about how to stop ddos of conntrack max value. Perhaps
> somebody shoul implement conntrack without syn, syn-ack. I think Junioer
> has this feature. Like having syncoockies in conntrack.
> 
> 
> Regards.
> 
> hdemir.
> 
>> Hello Netfilter,
>> 
>> Ive been receiving lately two types of syn floods on an Intel Xeon
>> 2.4ghz + 4GB machine exclusively dedicated for this and the findings
>> have me very confused:
>>  I have syn cookies enabled and checked to be working as per syslog.
>> This machine has a 10gigabit uplink so I know that networking isnt a
>> bottleneck here (bandwith or router hardware based).
>> 
>> SCENARIO 1: the first attack was: 105mbits @ 330,000 pps and it brought
>> the machine to 100% CPU usage and over 50% packetloss Load average 12.
>> At that time it had a simple iptables script that that had less then 5
>> blacklists of port 80 ips and then a ACCEPT On port 80, nothing fancy. I
>> disabled iptables and load average went down immediately to 8 but there
>> was still high packet loss so basically we where DoSed efficiently.
>> 
>> SCENARIO 2: After that the attacker sent only a 30mbit synflood @ 70,000
>> pps .. Now i had less packet loss, and interestingly with iptables
>> enabled it would create almost immediate packetloss. At this time I
>> tried to explore installing conntrack-tools information about the state
>> table. conntrack said that with iptables enabled and syncookies the
>> maximum entries where 1300 ONLY... and a CPU usage reported by HTOP of
>> 40% on SI. After that I decided to drop iptables all together and
>> immediately port 80 started flowing with normal traffic (we have less
>> than 1mbit clean traffic) . No packetloss was present, because iptables
>> was disabled conntrack did not report any entries and netstat-na |wc -l
>> reported less than 300.
>> 
>> Questions:
>> 
>> a.) Can anybody suggest why there is so much CPU overhead when iptables
>> is turned on and dealing with such PPS? Is this normal? Usually what CPU
>> usage does a syn flood cookie enabled take?
>> 
>> b.) Is there a chance that the attacker exausted something else iam not
>> seeing?
>> 
>> 
>> Thanks for the help guys
>> 
>> Alex
>> 
>> --
>> To unsubscribe from this list: send the line "unsubscribe netfilter" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SynFloods and CPU usage with and without iptables. Confused!

[Alex Flex]

Steven,

In other words you are saying that having the following lines in my 
iptables script defeats the purpose of syn cookies?

/sbin/iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j 
ACCEPT

Just confirming.

Thanks
Alex

Re: SynFloods and CPU usage with and without iptables. Confused!

[Steve Kann]

Yes.

If you're keeping track of incoming connections, then the packets generated by the attacker in a SYN Flood connection will end up using ip_conntrack table entries.   The whole point of SYNCOOKIES is to have zero stats.    If you're getting million+ SYN/sec, you just cannot afford to keep track of them. even with quite efficient data structures, there's only so many you can possibly track.

-SteveK


On May 4, 2013, at 9:27 PM, Alex Flex <aflexzor@gmail.com> wrote:

> Steven,
> 
> In other words you are saying that having the following lines in my iptables script defeats the purpose of syn cookies?
> 
> /sbin/iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
> /sbin/iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
> 
> Just confirming.
> 
> Thanks
> Alex
> 
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SynFloods and CPU usage with and without iptables. Confused!

[Alex Flex]

Thanks Steven,

Out of curiosity is there a hash table, or a way to view such table and 
its limits?

Alex

On 05/04/2013 07:34 PM, Steve Kann wrote:
> Yes.
>
> If you're keeping track of incoming connections, then the packets generated by the attacker in a SYN Flood connection will end up using ip_conntrack table entries.   The whole point of SYNCOOKIES is to have zero stats.    If you're getting million+ SYN/sec, you just cannot afford to keep track of them. even with quite efficient data structures, there's only so many you can possibly track.
>
> -SteveK
>
>
> On May 4, 2013, at 9:27 PM, Alex Flex<aflexzor@gmail.com>  wrote:
>
>> Steven,
>>
>> In other words you are saying that having the following lines in my iptables script defeats the purpose of syn cookies?
>>
>> /sbin/iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
>> /sbin/iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
>>
>> Just confirming.
>>
>> Thanks
>> Alex
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe netfilter" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SynFloods and CPU usage with and without iptables. Confused!

[Alex Flex]

Hdemir,

I think that after thr attacks are over, iam going "back to the drawing 
board" ...

Can you please confirm if you are able to load test your server with 
hping3 for a syn flood? For some reason i have not been able to get more 
then 8k pps so far with the flood option.

Iam currently looking for ways to load test different scenarios as per 
Stevens advice.

Thanks
Alex

Re: SynFloods and CPU usage with and without iptables. Confused!

[Husnu Demir]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Alex,

http://www.binarytides.com/python-syn-flood-program-raw-sockets-linux/

this simple script allows me to make +200K pps.

hdemir.


On 05-05-2013 04:29, Alex Flex wrote:
> Hdemir,
> 
> I think that after thr attacks are over, iam going "back to the
> drawing board" ...
> 
> Can you please confirm if you are able to load test your server
> with hping3 for a syn flood? For some reason i have not been able
> to get more then 8k pps so far with the flood option.
> 
> Iam currently looking for ways to load test different scenarios as
> per Stevens advice.
> 
> Thanks Alex
> 
> -- To unsubscribe from this list: send the line "unsubscribe
> netfilter" in the body of a message to majordomo@vger.kernel.org 
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQEcBAEBAgAGBQJRh5OJAAoJEISpBAM51qlENQ0H/A1Gq31Dhj0Pq4DOLSvozgd8
8z+47/bbBPX4Xj3D15/pJj9a6Fj64cRGq2KetPV4zlmargk0MMnanZEERqRkwkJI
EI6bgNuOe2Mw3PA14wvplF6zgtJYnLRN4JkyObqofpyfKEorAlrmrxcMe5seKYEy
ljoYdyOjTG5p6+D94miCJjHRMadGHjd5wGCi2knCE3+4kg4E5FptU/DVLSJhXm3X
RCQ5yPq2mIyJQ5y6M1phstUE39HJaO7QiyX3BLXGz2JamFIIadBV7FGBaSDj4mLL
rB0ybyL3S9jL8sIUyVyUDvmX6pGsGrSrujJzqdnScyeaupJg7fDFyOP3uWY1FCM=
=wYOr
-----END PGP SIGNATURE-----

SynFloods and CPU usage with and without iptables. Confused!

[Alex Flex]

Hello Netfilter,

Ive been receiving lately two types of syn floods on an Intel Xeon 
2.4ghz + 4GB machine exclusively dedicated for this and the findings 
have me very confused:
  I have syn cookies enabled and checked to be working as per syslog.
This machine has a 10gigabit uplink so I know that networking isnt a 
bottleneck here (bandwith or router hardware based).

SCENARIO 1: the first attack was: 105mbits @ 330,000 pps and it brought 
the machine to 100% CPU usage and over 50% packetloss Load average 12. 
At that time it had a simple iptables script that that had less then 5 
blacklists of port 80 ips and then a ACCEPT On port 80, nothing fancy. I 
disabled iptables and load average went down immediately to 8 but there 
was still high packet loss so basically we where DoSed efficiently.

SCENARIO 2: After that the attacker sent only a 30mbit synflood @ 70,000 
pps .. Now i had less packet loss, and interestingly with iptables 
enabled it would create almost immediate packetloss. At this time I 
tried to explore installing conntrack-tools information about the state 
table. conntrack said that with iptables enabled and syncookies the 
maximum entries where 1300 ONLY... and a CPU usage reported by HTOP of 
40% on SI. After that I decided to drop iptables all together and 
immediately port 80 started flowing with normal traffic (we have less 
than 1mbit clean traffic) . No packetloss was present, because iptables 
was disabled conntrack did not report any entries and netstat-na |wc -l 
reported less than 300.

Questions:

a.) Can anybody suggest why there is so much CPU overhead when iptables 
is turned on and dealing with such PPS? Is this normal? Usually what CPU 
usage does a syn flood cookie enabled take?

b.) Is there a chance that the attacker exausted something else iam not 
seeing?


Thanks for the help guys

Alex

Re: SynFloods and CPU usage with and without iptables. Confused!

[Alex Flex]

Payam,

Thanks for your reply!

Unfortunately the attack subsided and I did not get a packet log with 
tcpdump , I can tell you also that during such attack once I disabled 
iptables I was no longer logging anything. Dmesg all i see is:

possible SYN flooding on port 80. Sending cookies.
possible SYN flooding on port 80. Sending cookies.
possible SYN flooding on port 80. Sending cookies.
possible SYN flooding on port 80. Sending cookies.
possible SYN flooding on port 80. Sending cookies.
possible SYN flooding on port 80. Sending cookies.
[....]

I just realized that another machine of ours got attacked, about 20mbits 
and it brought the public interface down immediately . The graphs showed 
a CPU consumption of 14-20%.. then attack subsided 30 min later. This 
other machine also has syn cookieshowever dmesg didnt show any warning 
about it using syn cookies.. but it is enabled:

[root@w1 ~]# sysctl -a | grep syn
fs.quota.syncs = 0
net.ipv4.tcp_syn_retries = 5
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv6.conf.all.max_desync_factor = 600
net.ipv6.conf.default.max_desync_factor = 600
net.ipv6.conf.lo.max_desync_factor = 600
net.ipv6.conf.eth0.max_desync_factor = 600
net.ipv6.conf.eth1.max_desync_factor = 600



To me this confirms they are exausting something else ... Do you have 
any other suggestion?

Thanks

JP







On 05/04/2013 12:00 PM, Payam Chychi wrote:
> Wait are you logging the attack? That will cause high cpu utilization, 
> specially via syslog
>
> Anything on dmsg?
>
> -- 
> Payam Chychi
> Network Engineer / Security Specialist
>
> On Saturday, 4 May, 2013 at 10:29 AM, Payam Chychi wrote:
>
>> Do you have a capture of the attack? Does not make sense that cpu 
>> would go high for a normal synflood attack
>>
>> -- 
>> Payam Chychi
>> Network Engineer / Security Specialist
>>
>> On Saturday, 4 May, 2013 at 10:15 AM, Alex Flex wrote:
>>
>>> Hello Netfilter,
>>>
>>> Ive been receiving lately two types of syn floods on an Intel Xeon
>>> 2.4ghz + 4GB machine exclusively dedicated for this and the findings
>>> have me very confused:
>>> I have syn cookies enabled and checked to be working as per syslog.
>>> This machine has a 10gigabit uplink so I know that networking isnt a
>>> bottleneck here (bandwith or router hardware based).
>>>
>>> SCENARIO 1: the first attack was: 105mbits @ 330,000 pps and it brought
>>> the machine to 100% CPU usage and over 50% packetloss Load average 12.
>>> At that time it had a simple iptables script that that had less then 5
>>> blacklists of port 80 ips and then a ACCEPT On port 80, nothing 
>>> fancy. I
>>> disabled iptables and load average went down immediately to 8 but there
>>> was still high packet loss so basically we where DoSed efficiently.
>>>
>>> SCENARIO 2: After that the attacker sent only a 30mbit synflood @ 
>>> 70,000
>>> pps .. Now i had less packet loss, and interestingly with iptables
>>> enabled it would create almost immediate packetloss. At this time I
>>> tried to explore installing conntrack-tools information about the state
>>> table. conntrack said that with iptables enabled and syncookies the
>>> maximum entries where 1300 ONLY... and a CPU usage reported by HTOP of
>>> 40% on SI. After that I decided to drop iptables all together and
>>> immediately port 80 started flowing with normal traffic (we have less
>>> than 1mbit clean traffic) . No packetloss was present, because iptables
>>> was disabled conntrack did not report any entries and netstat-na |wc -l
>>> reported less than 300.
>>>
>>> Questions:
>>>
>>> a.) Can anybody suggest why there is so much CPU overhead when iptables
>>> is turned on and dealing with such PPS? Is this normal? Usually what 
>>> CPU
>>> usage does a syn flood cookie enabled take?
>>>
>>> b.) Is there a chance that the attacker exausted something else iam not
>>> seeing?
>>>
>>>
>>> Thanks for the help guys
>>>
>>> Alex
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe netfilter" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>

Re: SynFloods and CPU usage with and without iptables. Confused!

[Alex Flex]

Payam,

This is the copy of sysctl.conf on the second machine that got DoSed 
with 20mbits if you think it helps..

#DEFAULTS
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
# sysctl.conf(5) for more details.

# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

# Disable netfilter on bridges.
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65536

# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 65536

# Controls the maximum shared segment size, in bytes
kernel.shmmax = 4294967295

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 268435456

###########################################################################################################

# MODIFIED VERSION -- REPLACES DEFAULTS
# March 12 2012 JP - Based on a Red Hat based distro.

# Disables packet forwarding
net.ipv4.ip_forward=0

# Disables IP source routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.lo.accept_source_route = 0
net.ipv4.conf.eth0.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0

# Enable IP spoofing protection, turn on source route verification
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Disable ICMP Redirect Acceptance
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.lo.accept_redirects = 0
net.ipv4.conf.eth0.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0

# Enable Log Spoofed Packets, Source Routed Packets, Redirect Packets
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.lo.log_martians = 0
net.ipv4.conf.eth0.log_martians = 0

# Disables IP source routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.lo.accept_source_route = 0
net.ipv4.conf.eth0.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0

# Enable IP spoofing protection, turn on source route verification
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Disable ICMP Redirect Acceptance
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.lo.accept_redirects = 0
net.ipv4.conf.eth0.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0

# Disables the magic-sysrq key
kernel.sysrq = 0

# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15

# Decrease the time default value for tcp_keepalive_time connection
net.ipv4.tcp_keepalive_time = 1800

# Turn off the tcp_window_scaling
#net.ipv4.tcp_window_scaling = 0

# Turn off the tcp_sack
#net.ipv4.tcp_sack = 0

# Turn off the tcp_timestamps
net.ipv4.tcp_timestamps = 0

# Enable TCP SYN Cookie Protection
net.ipv4.tcp_syncookies = 1

# Enable ignoring broadcasts request
net.ipv4.icmp_echo_ignore_broadcasts = 0

#Disable ALL ICMP
net.ipv4.icmp_echo_ignore_all = 0

# Enable bad error message Protection
net.ipv4.icmp_ignore_bogus_error_responses = 0

# Log Spoofed Packets, Source Routed Packets, Redirect Packets
net.ipv4.conf.all.log_martians = 1

# Increases the size of the socket queue (effectively, q0).
net.ipv4.tcp_max_syn_backlog = 1024

# Increase the tcp-time-wait buckets pool size
net.ipv4.tcp_max_tw_buckets = 1440000

# Allowed local port range
net.ipv4.ip_local_port_range = 2000 65535

########## IPv6 networking start ##############
#DISABLE IPV6
net.ipv6.conf.all.disable_ipv6 = 1
########## IPv6 networking ends ##############

#Enable ExecShield protection
#kernel.exec-shield = 1
#kernel.randomize_va_space = 1

# TCP and memory optimization
# increase TCP max buffer size setable using setsockopt()
#net.ipv4.tcp_rmem = 4096 87380 8388608
#net.ipv4.tcp_wmem = 4096 87380 8388608

# increase Linux auto tuning TCP buffer limits
#net.core.rmem_max = 8388608
#net.core.wmem_max = 8388608
#net.core.netdev_max_backlog = 5000
#net.ipv4.tcp_window_scaling = 1

# increase system file descriptor limit
#fs.file-max = 65535

#Allow for more PIDs
#kernel.pid_max = 65536


Thanks
Alex

Re: SynFloods and CPU usage with and without iptables. Confused!

[Alex Flex]

Payam,

I was only running port 80... and what I have hosted there is a reverse 
proxy with nginx. Could there be a chance that xtables compiled into the 
kernel be causing this? Even if iptables is disabled?

I have xtables even with geoip support... although it isnt being used at 
the moment during those attacks

Alex

Re: SynFloods and CPU usage with and without iptables. Confused!

[Jozsef Kadlecsik]

On Sat, 4 May 2013, Alex Flex wrote:

> Ive been receiving lately two types of syn floods on an Intel Xeon 
> 2.4ghz + 4GB machine exclusively dedicated for this and the findings 
> have me very confused:
>  I have syn cookies enabled and checked to be working as per syslog.
> This machine has a 10gigabit uplink so I know that networking isnt a
> bottleneck here (bandwith or router hardware based).
> 
> SCENARIO 1: the first attack was: 105mbits @ 330,000 pps and it brought the
> machine to 100% CPU usage and over 50% packetloss Load average 12. At that
> time it had a simple iptables script that that had less then 5 blacklists of
> port 80 ips and then a ACCEPT On port 80, nothing fancy. I disabled iptables
> and load average went down immediately to 8 but there was still high packet
> loss so basically we where DoSed efficiently.

What about connection tracking? Did you rmmod the corresponding kernel 
module?
 
> SCENARIO 2: After that the attacker sent only a 30mbit synflood @ 70,000 pps
> .. Now i had less packet loss, and interestingly with iptables enabled it
> would create almost immediate packetloss. At this time I tried to explore
> installing conntrack-tools information about the state table. conntrack said
> that with iptables enabled and syncookies the maximum entries where 1300
> ONLY... and a CPU usage reported by HTOP of 40% on SI. After that I decided to
> drop iptables all together and immediately port 80 started flowing with normal
> traffic (we have less than 1mbit clean traffic) . No packetloss was present,
> because iptables was disabled conntrack did not report any entries and
> netstat-na |wc -l reported less than 300.

What do you mean exactly by "dropping iptables all together"?

Syncookies are not taken into account by conntrack, so it works as usual 
unless the module is unloaded. If you did not tune (increased) the hash 
size, it might be possible that conntrack or it's hashing was successfully 
attacked.

Best regards,
Jozsef 
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary