ulog: losing packets

ulog: losing packets

[Alexander Kolesnik]

Hello,

We're accounting traffic for users by ULOG-ipcad chain. Some time ago
I've found that some of users traffic does not appear in ipcad. To
check that this is not ipcad's problem I've installed ulogd and found
following in its log:
Fri Feb 27 15:25:56 2009 <3> ulogd.c:487 ulogd Version 1.22 starting
Fri Feb 27 15:25:56 2009 <5> ulogd.c:766 initialization finished, entering main loop
Fri Feb 27 15:27:00 2009 <7> ulogd.c:777 ipulog_read == -1! ipulog_errno == 6, errno = 105
Fri Feb 27 15:27:02 2009 <7> ulogd.c:777 ipulog_read == -1! ipulog_errno == 6, errno = 105
...

I have following settings for ulog and other stuff:
/etc/modprobe.conf:
options nf_conntrack hashsize=2097152
options ipt_ULOG nlbufsiz=65535 flushtimeout=100

# sysctl -a|grep rmem
net.ipv4.tcp_rmem = 4096        87380   174760
net.core.rmem_default = 221184
net.core.rmem_max = 4194304

/etc/ulogd.conf:
rmem=442368
bufsize=1000000

/etc/ipcad.conf:
buffers = 4194304;

# uname -a
Linux host.mydomain 2.6.20.20-2 #2 SMP Mon Oct 15 15:26:52 MSD 2007 i686 i686 i386 GNU/Linux

# free
             total       used       free     shared    buffers     cached
Mem:       1033124     980912      52212          0      83132     507860
-/+ buffers/cache:     389920     643204

Please, let me know what should be tuned to fix this problem!

Thanks in advance!

-- 
Alexander

Re: ulog: losing packets

[Pablo Neira Ayuso]

Alexander Kolesnik wrote:
> Hello,
> 
> We're accounting traffic for users by ULOG-ipcad chain. Some time ago
> I've found that some of users traffic does not appear in ipcad. To
> check that this is not ipcad's problem I've installed ulogd and found
> following in its log:
> Fri Feb 27 15:25:56 2009 <3> ulogd.c:487 ulogd Version 1.22 starting
> Fri Feb 27 15:25:56 2009 <5> ulogd.c:766 initialization finished, entering main loop
> Fri Feb 27 15:27:00 2009 <7> ulogd.c:777 ipulog_read == -1! ipulog_errno == 6, errno = 105
> Fri Feb 27 15:27:02 2009 <7> ulogd.c:777 ipulog_read == -1! ipulog_errno == 6, errno = 105

That means that netlink cannot back off as it is hitting ENOBUFS, thus,
you are losing log messages. Hm, ulog <= 1.24 does a primitive netlink
error handling.

> I have following settings for ulog and other stuff:
> /etc/modprobe.conf:
> options nf_conntrack hashsize=2097152
> options ipt_ULOG nlbufsiz=65535 flushtimeout=100
> 
> # sysctl -a|grep rmem
> net.ipv4.tcp_rmem = 4096        87380   174760
> net.core.rmem_default = 221184
> net.core.rmem_max = 4194304
> 
> /etc/ulogd.conf:
> rmem=442368
       ^^^^^^
Rising this value will delay hitting ENOBUFS. This is the size of the
receiver buffer.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re[2]: ulog: losing packets

[Alexander Kolesnik]

Hello Pablo,

Thanks for the answer!

>> /etc/ulogd.conf:
>> rmem=442368
PNA>        ^^^^^^
PNA> Rising this value will delay hitting ENOBUFS. This is the size of the
PNA> receiver buffer.

1. "delay" means I will get ENOBUFS in any case (early or later)?

2. What ENOBUFS does depend on? Packets per second? Bytes per second?
Amount of iptables/shaping rules? CPU performance?

3. Is there any way to calculate or predict the high limit of
traffic rate/number of rules/etc when the system will still manage to
process ULOG without alerting with ENOBUFS?

4. ipcad buffers (I suppose this is the same as rmem for ulogd) is set
to 4M:
/etc/ipcad.conf:
buffers = 4194304;
But I'm still losing ULOG messages. Does that mean I have to rise this
value more?

-- 
Best regards,
 Alexander

Re: ulog: losing packets

[Pablo Neira Ayuso]

Alexander Kolesnik wrote:
> Hello Pablo,
> 
> Thanks for the answer!
> 
>>> /etc/ulogd.conf:
>>> rmem=442368
> PNA>        ^^^^^^
> PNA> Rising this value will delay hitting ENOBUFS. This is the size of the
> PNA> receiver buffer.
> 
> 1. "delay" means I will get ENOBUFS in any case (early or later)?

Yes, but as said, you can tune different parameters to make it harder to
happen, like rising qthreshold, reducing cprange, setting a lower nice
value for ulogd.

> 2. What ENOBUFS does depend on? Packets per second? Bytes per second?
> Amount of iptables/shaping rules? CPU performance?

On the queue size, bytes/s sent to ulogd and on how slow ulogd is
reading messages.

> 3. Is there any way to calculate or predict the high limit of
> traffic rate/number of rules/etc when the system will still manage to
> process ULOG without alerting with ENOBUFS?

I don't know any, at least yet.

> 4. ipcad buffers (I suppose this is the same as rmem for ulogd) is set
> to 4M:
> /etc/ipcad.conf:
> buffers = 4194304;
> But I'm still losing ULOG messages. Does that mean I have to rise this
> value more?

Rising the value to the infinite is not either a solution, you'll hit
ENOBUFS sooner or later.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: ulog: losing packets

[Nick]

Pablo Neira Ayuso пишет:
> Alexander Kolesnik wrote:
>   
>> Hello Pablo,
>>
>> Thanks for the answer!
>>
>>     
>>>> /etc/ulogd.conf:
>>>> rmem=442368
>>>>         
>> PNA>        ^^^^^^
>> PNA> Rising this value will delay hitting ENOBUFS. This is the size of the
>> PNA> receiver buffer.
>>
>> 1. "delay" means I will get ENOBUFS in any case (early or later)?
>>     
>
> Yes, but as said, you can tune different parameters to make it harder to
> happen, like rising qthreshold, reducing cprange, setting a lower nice
> value for ulogd.
>
>   
>> 2. What ENOBUFS does depend on? Packets per second? Bytes per second?
>> Amount of iptables/shaping rules? CPU performance?
>>     
>
> On the queue size, bytes/s sent to ulogd and on how slow ulogd is
> reading messages.
>
>   
>> 3. Is there any way to calculate or predict the high limit of
>> traffic rate/number of rules/etc when the system will still manage to
>> process ULOG without alerting with ENOBUFS?
>>     
>
> I don't know any, at least yet.
>
>   
>> 4. ipcad buffers (I suppose this is the same as rmem for ulogd) is set
>> to 4M:
>> /etc/ipcad.conf:
>> buffers = 4194304;
>> But I'm still losing ULOG messages. Does that mean I have to rise this
>> value more?
>>     
>
> Rising the value to the infinite is not either a solution, you'll hit
> ENOBUFS sooner or later.
>
>   
I experimented with the configuration, but never succeeded. Packages are 
lost after 2MBit/s. For the solution of the problem I used other package 
- ulog-acctd. It's works perfect.

-- 
With best regards, Nikolay Ilkevich.