NFLOG - missing packets?

NFLOG - missing packets?

[Anton]

Hello!

Just did several test on the ULOGD2 - via NFLOG, and 
confused about the amount of data logged to the database.

In the tests - I'm transferring 700Mb of data file, but sum 
of the field in DB (ulog2.ip_totlen) for transferred amount 
is ~280M. At the same time iptables -nL -v for the 
correspondinf iptables NFLOG enties - shows correct amount 
of matches - ~700Mb

Tryed to set --nflog-range 100 or 200 (100 should be is more 
than enough though) - with no luck. 

Kernel - 2.6.24 
iptables v 1.4.0
ulogd2 SVN - rev 7519 (checked today)
DB - Postgresql 8.3.0

ulogd - logs no errors
dmesg logs no errors
db logs no errors

Am I missing something or ULOGD2 do not insert all of the 
packets into the DB (so this should be bug?) ?

Regards,
Anton.

Re: NFLOG - missing packets?

[Anton]

Something wrong quite definitelly,
in 700Mb of the data it logs only 43Mb 

(btw in prev email it was 700 kilobytes, not megabytes)

On Saturday 10 May 2008 16:57, Anton wrote:
> Hello!
>
> Just did several test on the ULOGD2 - via NFLOG, and
> confused about the amount of data logged to the database.
>
> In the tests - I'm transferring 700Mb of data file, but
> sum of the field in DB (ulog2.ip_totlen) for transferred
> amount is ~280M. At the same time iptables -nL -v for the
> correspondinf iptables NFLOG enties - shows correct
> amount of matches - ~700Mb
>
> Tryed to set --nflog-range 100 or 200 (100 should be is
> more than enough though) - with no luck.
>
> Kernel - 2.6.24
> iptables v 1.4.0
> ulogd2 SVN - rev 7519 (checked today)
> DB - Postgresql 8.3.0
>
> ulogd - logs no errors
> dmesg logs no errors
> db logs no errors
>
> Am I missing something or ULOGD2 do not insert all of the
> packets into the DB (so this should be bug?) ?
>
> Regards,
> Anton.
>
> --
> To unsubscribe from this list: send the line "unsubscribe
> netfilter-devel" in the body of a message to
> majordomo@vger.kernel.org More majordomo info at 
> http://vger.kernel.org/majordomo-info.html

Re: NFLOG - missing packets?

[Eric Leblond]

[-- Attachment #1: Type: text/plain, Size: 852 bytes --]

Hello,

On Saturday, 2008 May 10 at 17:50:37 +0500, Anton wrote:
> Something wrong quite definitelly,
> in 700Mb of the data it logs only 43Mb 

I've done some test on my system and I've got similar results. Sole
difference is the ratio between logged packets and unlogged packets. 

I've downloaded a 48Mo at a rate of 348.28K/s and I have the following
result :
Iptables : 33625 packets, 50383786 bits
Ulogd2 : 33272 packets 49856076 bits

As usual, size does not matter. The problem is due to some packets not
being loggued. It recall me results of Pablo's conntrackd benchmark.
After a given rate, the netlink subsystem is not able to transmit
messages.

IMHO, the main issue is that the lost (or the overrun) does not seem to be logged
somewhere.

BR,
-- 
Eric Leblond
INL: http://www.inl.fr/
NuFW: http://www.nufw.org/

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: NFLOG - missing packets?

[Anton]

Just some extra results,

seems DB drivers does such a difference, since while logging 
to LOGEMU (only) target - I've go result which looks true. 
But - If I enable DB logger - results in DB and LOGEMU - 
are the same. Looks like DB transfers makes ULOG to not 
accept packets from kernelspace

On Sunday 11 May 2008 21:18, Eric Leblond wrote:
> Hello,
>
> On Saturday, 2008 May 10 at 17:50:37 +0500, Anton wrote:
> > Something wrong quite definitelly,
> > in 700Mb of the data it logs only 43Mb
>
> I've done some test on my system and I've got similar
> results. Sole difference is the ratio between logged
> packets and unlogged packets.
>
> I've downloaded a 48Mo at a rate of 348.28K/s and I have
> the following result :
> Iptables : 33625 packets, 50383786 bits
> Ulogd2 : 33272 packets 49856076 bits
>
> As usual, size does not matter. The problem is due to
> some packets not being loggued. It recall me results of
> Pablo's conntrackd benchmark. After a given rate, the
> netlink subsystem is not able to transmit messages.
>
> IMHO, the main issue is that the lost (or the overrun)
> does not seem to be logged somewhere.
>
> BR,

Re: NFLOG - missing packets?

[Pablo Neira Ayuso]

Anton wrote:
> Just some extra results,
> 
> seems DB drivers does such a difference, since while logging 
> to LOGEMU (only) target - I've go result which looks true. 
> But - If I enable DB logger - results in DB and LOGEMU - 
> are the same. Looks like DB transfers makes ULOG to not 
> accept packets from kernelspace

The problem is netlink that cannot back off. Netlink is the underlying
communication subsystem that we use to communicate kernel with usepace
space. Since Netlink is unreliable, some log messages can vanish under
heavy load. I guess that database insertions consumes lots of CPU
resouces. Thus, doing online database logging in a scalable manner turns
really hard. Instead, if you need scalability, I'd suggest to use logemu
or whatever plain text logging facility and then convert it to a
database *offline* if you really need advanced queries.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: NFLOG - missing packets?

[Patrick McHardy]

Pablo Neira Ayuso wrote:
> Anton wrote:
>> Just some extra results,
>>
>> seems DB drivers does such a difference, since while logging 
>> to LOGEMU (only) target - I've go result which looks true. 
>> But - If I enable DB logger - results in DB and LOGEMU - 
>> are the same. Looks like DB transfers makes ULOG to not 
>> accept packets from kernelspace
> 
> The problem is netlink that cannot back off. Netlink is the underlying
> communication subsystem that we use to communicate kernel with usepace
> space. Since Netlink is unreliable, some log messages can vanish under
> heavy load. I guess that database insertions consumes lots of CPU
> resouces. Thus, doing online database logging in a scalable manner turns
> really hard. Instead, if you need scalability, I'd suggest to use logemu
> or whatever plain text logging facility and then convert it to a
> database *offline* if you really need advanced queries.


Yeah, but what we can do is check whether the message
was successfully transmitted in the kernel and drop
the packet in case it wasn't. That should catch 99.9%
of all error cases since a slow databse effectively
only causes the process to read less often from the
netlink socket.

I still have a very old and unfinished patch for this
somewhere ...

Re: NFLOG - missing packets?

[Anton]

On Wednesday 14 May 2008 22:53, Patrick McHardy wrote:
> > plain text logging facility and then convert it to a
> > database *offline* if you really need advanced queries.
>
> Yeah, but what we can do is check whether the message
> was successfully transmitted in the kernel and drop
> the packet in case it wasn't. That should catch 99.9%
> of all error cases since a slow databse effectively
> only causes the process to read less often from the
> netlink socket.

What we did, is trying to transport a message over network 
to another host, for further analysis - OPRINT and LOGEMU 
targets were used. Even with no further DB interaction 
(just plain dumping to a file) - causes loosing of 50% of 
packets with hacked OPRINT - regardles if it's done while 
connecting to localhost (same pc) or another host. 
PC's are P4-3Ghz.

host 1(NFLOG -> ULOG -> OPRINT_OVER_NETWORK) -> host 2
(LOG_RECEIVER)

and just plain native LOGEMU looses arround 1% of packets on 
100mbit transfers.

If there a way for checking is the message were transmitted, 
how to do that?

>
> I still have a very old and unfinished patch for this
> somewhere ...

Re: NFLOG - missing packets?

[Eric Leblond]

[-- Attachment #1: Type: text/plain, Size: 1337 bytes --]

Hello,

On Wednesday, 2008 May 14 at 19:49:26 +0200, Pablo Neira Ayuso wrote:
> Anton wrote:
> > Just some extra results,
> > 
> > seems DB drivers does such a difference, since while logging 
> > to LOGEMU (only) target - I've go result which looks true. 
> > But - If I enable DB logger - results in DB and LOGEMU - 
> > are the same. Looks like DB transfers makes ULOG to not 
> > accept packets from kernelspace
> 
> The problem is netlink that cannot back off. Netlink is the underlying
> communication subsystem that we use to communicate kernel with usepace
> space. Since Netlink is unreliable, some log messages can vanish under
> heavy load. I guess that database insertions consumes lots of CPU
> resouces. Thus, doing online database logging in a scalable manner turns
> really hard.

IMHO, this problem is more linked with a treatment delay due to the
database connection. I will try to switch the database plugin to 
non-blocking query and I will send you results and a patch (if results
are good).

> Instead, if you need scalability, I'd suggest to use logemu
> or whatever plain text logging facility and then convert it to a
> database *offline* if you really need advanced queries.

It will be sad to do so ;)

BR,
-- 
Eric Leblond
INL: http://www.inl.fr/
NuFW: http://www.nufw.org/

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: NFLOG - missing packets?

[Anton]

I did some extra test, which show diffrenece on P4 2.8G with 
the transfer rate 100Mbit/s between iptables chains couters 
and ulogd logemu records - 2% - seems it's ulogd misses, 
not netlink subsystem, and it's directly depends on the 
ulogd processing load - db inserts are slow...

Do you think it's fixable in viewable future atleast on the 
ulog level, that we know what amount of data we have 
missed?

On Sunday 11 May 2008 21:18, Eric Leblond wrote:
> Hello,
>
> On Saturday, 2008 May 10 at 17:50:37 +0500, Anton wrote:
> > Something wrong quite definitelly,
> > in 700Mb of the data it logs only 43Mb
>
> I've done some test on my system and I've got similar
> results. Sole difference is the ratio between logged
> packets and unlogged packets.
>
> I've downloaded a 48Mo at a rate of 348.28K/s and I have
> the following result :
> Iptables : 33625 packets, 50383786 bits
> Ulogd2 : 33272 packets 49856076 bits
>
> As usual, size does not matter. The problem is due to
> some packets not being loggued. It recall me results of
> Pablo's conntrackd benchmark. After a given rate, the
> netlink subsystem is not able to transmit messages.
>
> IMHO, the main issue is that the lost (or the overrun)
> does not seem to be logged somewhere.
>
> BR,

Re: NFLOG - missing packets?

[Pablo Neira Ayuso]

Eric Leblond wrote:
> Hello,
> 
> On Saturday, 2008 May 10 at 17:50:37 +0500, Anton wrote:
>> Something wrong quite definitelly,
>> in 700Mb of the data it logs only 43Mb 
> 
> I've done some test on my system and I've got similar results. Sole
> difference is the ratio between logged packets and unlogged packets. 
> 
> I've downloaded a 48Mo at a rate of 348.28K/s and I have the following
> result :
> Iptables : 33625 packets, 50383786 bits
> Ulogd2 : 33272 packets 49856076 bits
> 
> As usual, size does not matter. The problem is due to some packets not
> being loggued. It recall me results of Pablo's conntrackd benchmark.
> After a given rate, the netlink subsystem is not able to transmit
> messages.
> 
> IMHO, the main issue is that the lost (or the overrun) does not seem to be logged
> somewhere.

I'm finishing a patchset to improve this and other aspects of ulogd. 
I'll post them to the ML soon.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers