Connection tracking counters increasing

Connection tracking counters increasing

[Daniel L. Miller]

Sorry if this is answered elsewhere - my initial googling didn't show me.

I have an Ubuntu Precise server that is using iptables.  I've noticed 
that my connection tracking counters keep increasing - and will do so 
until failure.  I've tried a few different timeout settings to see if 
that fixes it - but so far nothing helps except increasing the maximums 
and then rebooting prior to failure.

I'm assuming I'm doing something wrong - but I don't even know where to 
look.  The only things that helps is NOT using iptables - which isn't a 
good answer!

-- 
Daniel

Re: Connection tracking counters increasing

[Jan Engelhardt]

On Thursday 2013-03-07 17:13, Daniel L. Miller wrote:

> I've noticed that my connection tracking counters keep increasing -

Which counters?


> and will do so until failure.
> I've tried a few different timeout settings to see if that fixes it - but so
> far nothing helps except increasing the maximums and then rebooting prior to
> failure.
>
> I'm assuming I'm doing something wrong - but I don't even know where to look.
> The only things that helps is NOT using iptables - which isn't a good answer!

Re: Connection tracking counters increasing

[Daniel L. Miller]

On 3/7/2013 8:29 AM, Jan Engelhardt wrote:
> On Thursday 2013-03-07 17:13, Daniel L. Miller wrote:
>
>> I've noticed that my connection tracking counters keep increasing -
> Which counters?
>
/proc/sys/net/netfilter/nf_conntrack_count

and when it reaches /proc/sys/net/netfilter/nf_conntrack_max things break.

-- 
Daniel

Re: Connection tracking counters increasing

[Jan Engelhardt]

On Thursday 2013-03-07 18:07, Daniel L. Miller wrote:

> On 3/7/2013 8:29 AM, Jan Engelhardt wrote:
>> On Thursday 2013-03-07 17:13, Daniel L. Miller wrote:
>>
>>> I've noticed that my connection tracking counters keep increasing -
>> Which counters?
>>
> /proc/sys/net/netfilter/nf_conntrack_count
>
> and when it reaches /proc/sys/net/netfilter/nf_conntrack_max things break.

Could it be you are being DDoSed?

Re: Connection tracking counters increasing

[Daniel L. Miller]

On 3/7/2013 9:16 AM, Jan Engelhardt wrote:
> On Thursday 2013-03-07 18:07, Daniel L. Miller wrote:
>
>> On 3/7/2013 8:29 AM, Jan Engelhardt wrote:
>>> On Thursday 2013-03-07 17:13, Daniel L. Miller wrote:
>>>
>>>> I've noticed that my connection tracking counters keep increasing -
>>> Which counters?
>>>
>> /proc/sys/net/netfilter/nf_conntrack_count
>>
>> and when it reaches /proc/sys/net/netfilter/nf_conntrack_max things break.
> Could it be you are being DDoSed?

Anything's possible - but I don't have a high-visibility site.  The 
connection counter increments gradually - a new connection every couple 
of seconds or so.

Trying to understand what's going on, I installed the "conntrack" tool.  
conntrack -S yields:
entries                 28752
searched                27822
found                   11950927
new                     457431
invalid                 333
ignore                  347677
delete                  465404
delete_list             428962
insert                  414575
insert_failed           0
drop                    0
early_drop              0
icmp_error              0
expect_new              0
expect_create           0
expect_delete           0
search_restart          0

But more confusing is conntrack -L only shows 52 entries.

-- 
Daniel

Re: Connection tracking counters increasing

[Jan Engelhardt]

On Thursday 2013-03-07 18:21, Daniel L. Miller wrote:
>>>
>>> I've noticed that my connection tracking counters keep increasing -
>>> /proc/sys/net/netfilter/nf_conntrack_count
>>> and when it reaches /proc/sys/net/netfilter/nf_conntrack_max things break.
>>
>> Could it be you are being DDoSed?
>
> Anything's possible - but I don't have a high-visibility site.  The connection
> counter increments gradually - a new connection every couple of seconds or so.
>
> Trying to understand what's going on, I installed the "conntrack" tool.
> conntrack -S yields:
> entries                 28752
> searched                27822
> found                   11950927
> new                     457431
> invalid                 333
> ignore                  347677
> delete                  465404
> delete_list             428962
> insert                  414575
> insert_failed           0
> drop                    0
> early_drop              0
> icmp_error              0
> expect_new              0
> expect_create           0
> expect_delete           0
> search_restart          0
>
> But more confusing is conntrack -L only shows 52 entries.

You need to count in entries from different conntrack tables
too. I see something similar, however:

18:42 ares07:/home/jengelh # conntrack -L >/dev/null; conntrack -L expect
>/dev/null; cat /proc/sys/net/netfilter/nf_conntrack_count
conntrack v1.4.0 (conntrack-tools): 32 flow entries have been shown.
conntrack v1.4.0 (conntrack-tools): 0 expectations have been shown.
43
18:42 ares07:/home/jengelh # conntrack -L >/dev/null; conntrack -L expect
>/dev/null; cat /proc/sys/net/netfilter/nf_conntrack_count
conntrack v1.4.0 (conntrack-tools): 31 flow entries have been shown.
conntrack v1.4.0 (conntrack-tools): 0 expectations have been shown.
42

That might be connections in state DYING, but perhaps Pablo has an idea
what could be going on w.r.t. different numbers.

Re: Connection tracking counters increasing

[Daniel L. Miller]

On 3/7/2013 10:10 AM, Alex Bligh wrote:
> On 7 Mar 2013, at 17:21, Daniel L. Miller wrote:
>
>> Anything's possible - but I don't have a high-visibility site.  The connection counter increments gradually - a new connection every couple of seconds or so.
> Why would you not expect this to happen? It's an absolute count of the number of
> connections since reboot. Are you quite sure nothing is opening any inbound or
> outbound connections or trying to? Do you have connection tracking on loopback?
>
If it's an informative counter - then I don't care.  If it's active 
state counter - and blocks new connections from happening when it 
reaches the defined maximum - then it's very much a concern.

I'm assuming something has changed - whether it's due to an upgrade, new 
software, or new device on the network I don't know.  I don't know how 
to look at the list of connections to determine what's getting 
opened/accessed/left so I can narrow it down.

At the moment, I have a very small firehol config plus fail2ban. That's 
all I know actively using iptables.  I'm running OpenVPN - at times I 
will setup some iptables dnat/snat for connections to remotes.  So I do 
need to have NAT available otherwise I'd just unload the modules and 
forget about it.  My primary firewall is a Mikrotik router that doesn't 
have these issues - but I want at least a little protection on my main 
server (which this is).

I have a web server running on a VirtualBox virtual machine via a 
bridged interface.  I do not have connection tracking on the VM.  I 
would think the web server would be the primary target of inbound 
connections - for whatever that's worth.  I also run mail and DNS.

I've never had this problem before - but I've been dealing with it for 
about the past month.  I'm sure something's changed - whether it's my 
website's google visibility or a software change I don't know.

-- 
Daniel

Re: Connection tracking counters increasing

[Xiong Wu]

Hi Daniel,

Do you try to turn off nf_conntrack_tcp_loose as below:

echo 0 > /proc/sys/net/netfilter/nf_conntrack_tcp_loose


Thanks,
Xiong


On Thu, 2013-03-07 at 09:07 -0800, Daniel L. Miller wrote:
> On 3/7/2013 8:29 AM, Jan Engelhardt wrote:
> > On Thursday 2013-03-07 17:13, Daniel L. Miller wrote:
> >
> >> I've noticed that my connection tracking counters keep increasing -
> > Which counters?
> >
> /proc/sys/net/netfilter/nf_conntrack_count
> 
> and when it reaches /proc/sys/net/netfilter/nf_conntrack_max things break.
> 

Re: Connection tracking counters increasing

[Daniel L. Miller]

On 3/8/2013 1:08 AM, Xiong Wu wrote:
> Hi Daniel,
>
> Do you try to turn off nf_conntrack_tcp_loose as below:
>
> echo 0 > /proc/sys/net/netfilter/nf_conntrack_tcp_loose
>
I just tried that.  No apparent difference, counters continue to 
increase.  Now at 32322.

--
Daniel

Re: Connection tracking counters increasing

[Daniel L. Miller]

On 3/8/2013 4:31 AM, Alex Bligh wrote:
> On 8 Mar 2013, at 12:19, Daniel L. Miller wrote:
>
>> I just tried that.  No apparent difference, counters continue to increase.  Now at 32322.
> Does conntrack -L list anything?
>

Yes - the number fluctuates around 100 entries.

-- 
Daniel