From: Bernhard Schmidt <berni@birkenwald.de>
To: Pablo Neira Ayuso <pablo@netfilter.org>
Cc: netfilter-devel@vger.kernel.org
Subject: Re: null-pointer deref in ulogd2
Date: Tue, 23 Jun 2009 18:54:46 +0200 [thread overview]
Message-ID: <4A4108D6.901@birkenwald.de> (raw)
In-Reply-To: <4A40F777.7010505@netfilter.org>
Hi Pablo,
>>> now it seems to work okay. In the database about 90% of the flows have
>>> flow_end_sec NULL.
> Please, rise "netlink_socket_buffer_size" and
> "netlink_socket_buffer_maxsize". If you use the default buffer, it's
> likely to overrun and, thus, to lose events.
We had increased that in the meantime, to
netlink_socket_buffer_size=10854400
netlink_socket_buffer_maxsize=20971520
That pretty much stopped the warning messages in /var/log/ulogd.log
We had also figured that the hash was the problem, so we tried the
hash_enable=0 and used the INSERT_OR_REPLACE_CT function. However, that
was also pretty unsuccessful, right now we have 750k flows in ulog2_ct
where ct_event < 4 (so, as far as I understand it, the DESTROY event has
not yet been received). Which is a bit too much for a box that only has
40k-50k connections at the same time according to conntrack -L. 1.67M
flows in total, I suspect that's a bit low as well. When I did 100 HTTP
connections through the box I could only find ~20 flows in the database,
none of them in DESTROYed state.
>> What is happening here?
> I think that you're using the default "hash_max_entries" which is too
> small. I suggest you to rise this value. I'm going to push a patch that
> includes information on these parameter tweaking to the example config file.
I've now set
hash_buckets=81920
hash_max_entries=327680
and went back to hash_enable=1.
However, it still doesn't look too great. About five minutes after 100
TCP connects the number of flows in the ulog2_ct table for this IP
address has stabilized at 116, consisting of
- 9 flows with both flow_start_sec and flow_end_sec
- 83 flows with only flow_start_sec
- 24 flows with only flow_end_sec
SELECT COUNT(DISTINCT orig_l4_sport) tells me that 92 real connections
are listed in the table somehow, so 8 connections are totally lost and
24 connections are listed twice.
[ half an hour later ]
ARGH! I found my problem. Apparently Postgres was too slow on INSERT.
Although the CPU load looked fine (and even IOWait wasn't out of the
ordinary, 20% on one CPU) it seems to have blocked. Sacrificing
consistency for speed by setting fsync=no in postgres the IOwait went
down to 0.5% and I now have 100 flows, all of them with start and end!
> BTW, could you give a quick test to this patch, yours seems to leak
> memory since NFCT_CB_STOLEN means not to release the ct object (no
> problem, I guess that you're not familiar with libnetfilter_conntrack).
Thanks. Yes, I'm even not that familiar with C :-)
Your patch compiles and runs fine. Can't tell much about memory leaks,
but the system has not exploded yet.
Bernhard
next prev parent reply other threads:[~2009-06-23 17:06 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-06-23 7:27 null-pointer deref in ulogd2 Bernhard Schmidt
2009-06-23 8:31 ` Bernhard Schmidt
2009-06-23 15:40 ` Pablo Neira Ayuso
2009-06-23 16:54 ` Bernhard Schmidt [this message]
2009-06-23 22:39 ` Bernhard Schmidt
2009-06-24 10:59 ` conntrack segfault (was: Re: null-pointer deref in ulogd2) Bernhard Schmidt
2009-06-24 11:17 ` Krzysztof Oledzki
2009-06-24 11:57 ` Jan Engelhardt
2009-06-24 12:56 ` conntrack segfault Bernhard Schmidt
2009-06-24 17:58 ` Pablo Neira Ayuso
2009-06-24 20:05 ` Bernhard Schmidt
2009-06-24 22:18 ` Bernhard Schmidt
2009-07-02 16:30 ` Pablo Neira Ayuso
2009-07-06 10:29 ` Krzysztof Oledzki
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4A4108D6.901@birkenwald.de \
--to=berni@birkenwald.de \
--cc=netfilter-devel@vger.kernel.org \
--cc=pablo@netfilter.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).