libnetfilter_conntrack, nfct_catch returns ENOBUFS

libnetfilter_conntrack, nfct_catch returns ENOBUFS

[fabien.marotte]

Hi,

I develop an application using libnetfilter_conntrack to catch events 
happening in the conntrack system.
I am currently testing the robustness of my application using ab tool 
(connection stress).

libnetfilter_conntrack version is 0.50. I use the new API.
My callback catching the events always returns NFCT_CB_CONTINUE meaning 
that ncft_catch should never returns.

But with 20 (and more) TCP  connections opening simultaneously, nfct_catch 
returns. The error code in errno is 105. Seems to be ENOBUFS.

Looking at the code, I see the comment "ENOBUFS is returned in case that 
nfnetlink is exhausted".

I am very surprised to have memory issues with so few connections.

Could you explain me why this error occurs ? Is this a kernel error 
propagated in user space or is this a user space error ?

Thanks,

Re: libnetfilter_conntrack, nfct_catch returns ENOBUFS

[Pablo Neira Ayuso]

fabien.marotte@mindspeed.com wrote:
> I develop an application using libnetfilter_conntrack to catch events 
> happening in the conntrack system.
> I am currently testing the robustness of my application using ab tool 
> (connection stress).
> 
> libnetfilter_conntrack version is 0.50. I use the new API.
> My callback catching the events always returns NFCT_CB_CONTINUE meaning 
> that ncft_catch should never returns.
> 
> But with 20 (and more) TCP  connections opening simultaneously, nfct_catch 
> returns. The error code in errno is 105. Seems to be ENOBUFS.
> 
> Looking at the code, I see the comment "ENOBUFS is returned in case that 
> nfnetlink is exhausted".
> 
> I am very surprised to have memory issues with so few connections.

So am I. Could you post the code?

> Could you explain me why this error occurs ? Is this a kernel error 
> propagated in user space or is this a user space error ?

Default buffer size in your system? cat /proc/sys/net/core/rmem_default

-- 
The dawn of the fourth age of Linux firewalling is coming; a time of 
great struggle and heroic deeds -- J.Kadlecsik got inspired by J.Morris

Re: libnetfilter_conntrack, nfct_catch returns ENOBUFS

[fabien.marotte]

> So am I. Could you post the code?

I reduced my callback to the following :
CtCatch( ...)
{
   return NFCT_CB_CONTINUE;
}
and I still see the bug with 20 simultaneous connections.


> Default buffer size in your system? cat /proc/sys/net/core/rmem_default

/proc/sys/net/core/rmem_default is set to 108544 by default.
I tried to increase the value up to 262144 and I still get the same issue.
Is it possible conntrack module to send more than one event in the same 
netlink messages ?








Pablo Neira Ayuso <pablo@netfilter.org> 
Sent by: netfilter-devel-bounces@lists.netfilter.org
06/18/2007 06:10 PM

To
fabien.marotte@mindspeed.com
cc
netfilter-devel@lists.netfilter.org
Subject
Re: libnetfilter_conntrack, nfct_catch returns ENOBUFS






fabien.marotte@mindspeed.com wrote:
> I develop an application using libnetfilter_conntrack to catch events 
> happening in the conntrack system.
> I am currently testing the robustness of my application using ab tool 
> (connection stress).
> 
> libnetfilter_conntrack version is 0.50. I use the new API.
> My callback catching the events always returns NFCT_CB_CONTINUE meaning 
> that ncft_catch should never returns.
> 
> But with 20 (and more) TCP  connections opening simultaneously, 
nfct_catch 
> returns. The error code in errno is 105. Seems to be ENOBUFS.
> 
> Looking at the code, I see the comment "ENOBUFS is returned in case that 

> nfnetlink is exhausted".
> 
> I am very surprised to have memory issues with so few connections.

So am I. Could you post the code?

> Could you explain me why this error occurs ? Is this a kernel error 
> propagated in user space or is this a user space error ?

Default buffer size in your system? cat /proc/sys/net/core/rmem_default

-- 
The dawn of the fourth age of Linux firewalling is coming; a time of 
great struggle and heroic deeds -- J.Kadlecsik got inspired by J.Morris

Re: libnetfilter_conntrack, nfct_catch returns ENOBUFS

[Pablo Neira Ayuso]

fabien.marotte@mindspeed.com wrote:
>> So am I. Could you post the code?
> 
> I reduced my callback to the following :
> CtCatch( ...)
> {
>    return NFCT_CB_CONTINUE;
> }

Well, this does not help too much. What is your callback actually doing? 
If the operation that your callback does is expensive and slow, such as 
printing the event, then may overrun sooner. Here I'm able to generate 
up a bulk of 10000 HTTP GET requests in my testbed with no overrun. Of 
course, using the callback that you suppose to be using. However, if I 
print the event, the results are different.

> and I still see the bug with 20 simultaneous connections.

This is not a bug.

>> Default buffer size in your system? cat /proc/sys/net/core/rmem_default
> 
> /proc/sys/net/core/rmem_default is set to 108544 by default.
> I tried to increase the value up to 262144 and I still get the same issue.
> Is it possible conntrack module to send more than one event in the same 
> netlink messages ?

ENOBUFS tells you that your buffer has overrun. Each process has a 
receiver buffer where netlink puts the messages into. If the sender puts 
more messages into the queue than the receiver can consume, then the 
buffer overruns. This happens if your userspace process is too slow to 
consume messages, this usually happens if your CPU reaches 100% 
consumption. Solution: You can delay the overrun by increasing the 
buffer size.

-- 
The dawn of the fourth age of Linux firewalling is coming; a time of 
great struggle and heroic deeds -- J.Kadlecsik got inspired by J.Morris