netfilter-devel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Bernhard Schmidt <berni@birkenwald.de>
To: Pablo Neira Ayuso <pablo@netfilter.org>
Cc: netfilter-devel@vger.kernel.org
Subject: Re: null-pointer deref in ulogd2
Date: Tue, 23 Jun 2009 18:54:46 +0200	[thread overview]
Message-ID: <4A4108D6.901@birkenwald.de> (raw)
In-Reply-To: <4A40F777.7010505@netfilter.org>

Hi Pablo,

>>> now it seems to work okay. In the database about 90% of the flows have
>>> flow_end_sec NULL.
> Please, rise "netlink_socket_buffer_size" and
> "netlink_socket_buffer_maxsize". If you use the default buffer, it's
> likely to overrun and, thus, to lose events.

We had increased that in the meantime, to

netlink_socket_buffer_size=10854400
netlink_socket_buffer_maxsize=20971520

That pretty much stopped the warning messages in /var/log/ulogd.log

We had also figured that the hash was the problem, so we tried the 
hash_enable=0 and used the INSERT_OR_REPLACE_CT function. However, that 
was also pretty unsuccessful, right now we have 750k flows in ulog2_ct 
where ct_event < 4 (so, as far as I understand it, the DESTROY event has 
not yet been received). Which is a bit too much for a box that only has 
40k-50k connections at the same time according to conntrack -L. 1.67M 
flows in total, I suspect that's a bit low as well. When I did 100 HTTP 
connections through the box I could only find ~20 flows in the database, 
none of them in DESTROYed state.

>> What is happening here?
> I think that you're using the default "hash_max_entries" which is too
> small. I suggest you to rise this value. I'm going to push a patch that
> includes information on these parameter tweaking to the example config file.

I've now set

hash_buckets=81920
hash_max_entries=327680

and went back to hash_enable=1.

However, it still doesn't look too great. About five minutes after 100 
TCP connects the number of flows in the ulog2_ct table for this IP 
address has stabilized at 116, consisting of
- 9 flows with both flow_start_sec and flow_end_sec
- 83 flows with only flow_start_sec
- 24 flows with only flow_end_sec

SELECT COUNT(DISTINCT orig_l4_sport) tells me that 92 real connections 
are listed in the table somehow, so 8 connections are totally lost and 
24 connections are listed twice.

[ half an hour later ]

ARGH! I found my problem. Apparently Postgres was too slow on INSERT. 
Although the CPU load looked fine (and even IOWait wasn't out of the 
ordinary, 20% on one CPU) it seems to have blocked. Sacrificing 
consistency for speed by setting fsync=no in postgres the IOwait went 
down to 0.5% and I now have 100 flows, all of them with start and end!

> BTW, could you give a quick test to this patch, yours seems to leak
> memory since NFCT_CB_STOLEN means not to release the ct object (no
> problem, I guess that you're not familiar with libnetfilter_conntrack).

Thanks. Yes, I'm even not that familiar with C :-)

Your patch compiles and runs fine. Can't tell much about memory leaks, 
but the system has not exploded yet.

Bernhard

  reply	other threads:[~2009-06-23 17:06 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2009-06-23  7:27 null-pointer deref in ulogd2 Bernhard Schmidt
2009-06-23  8:31 ` Bernhard Schmidt
2009-06-23 15:40   ` Pablo Neira Ayuso
2009-06-23 16:54     ` Bernhard Schmidt [this message]
2009-06-23 22:39       ` Bernhard Schmidt
2009-06-24 10:59         ` conntrack segfault (was: Re: null-pointer deref in ulogd2) Bernhard Schmidt
2009-06-24 11:17           ` Krzysztof Oledzki
2009-06-24 11:57             ` Jan Engelhardt
2009-06-24 12:56               ` conntrack segfault Bernhard Schmidt
2009-06-24 17:58                 ` Pablo Neira Ayuso
2009-06-24 20:05                   ` Bernhard Schmidt
2009-06-24 22:18                   ` Bernhard Schmidt
2009-07-02 16:30                     ` Pablo Neira Ayuso
2009-07-06 10:29                     ` Krzysztof Oledzki

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=4A4108D6.901@birkenwald.de \
    --to=berni@birkenwald.de \
    --cc=netfilter-devel@vger.kernel.org \
    --cc=pablo@netfilter.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).