Why "No buffer space available"?

Why "No buffer space available"?

[Medialy]

Hi,
I have written a program to log the nat behavior. the program works
well when traffic is low. But when the traffic reaches 1Gb, program
always error.
According to the previous discussions about this problem, I even set
the recv buffer size to 50MB and the error still exists.
For every callback, format the data and then put it into the queue
directly.  The formating of data causes less then 1 second for 0.65
million records.
Errors always occurs  when there are less than 10 log records.
Is anyone who can help?
Thanks.

Setting:
    Redhat Enterprise Linux 5
    libnetfilter_conntrack-0.0.100
    libnfnetlink-1.0.0
    recv buffer size: 50MB
    nfct_open(CONNTRACK, NF_NETLINK_CONNTRACK_NEW|NF_NETLINK_CONNTRACK_DESTROY)
    1Gb nat traffic, 0.65 million records per minute
    circular queue size: 1 million

Error:
    nfct_catch error: No buffer space available

Program Structure:
    Callback:
        lock;
        if log number > MAX_LOG_NUM:
            discard
        else:
            put log in circular queue
            log number += 1
        unlock

    Thread 2:
        lock;
        if log number >0 :
            get lock number
        unlock
        process log data in circular queue
        lock
        log number = log number - log number processed
        unlock

Re: Why "No buffer space available"?

[Pablo Neira Ayuso]

Medialy wrote:
> Hi,
> I have written a program to log the nat behavior. the program works
> well when traffic is low. But when the traffic reaches 1Gb, program
> always error.
> According to the previous discussions about this problem, I even set
> the recv buffer size to 50MB and the error still exists.

Increasing the buffer size would not solve the problem, that will only 
delay the ENOBUFS error. There are several reasons why you may hit ENOBUFS:

a) your program is too slow to handle the Netlink messages that you 
receive from the kernel at a given rate. This is easier to trigger if 
the handling that you perform on every message takes too long.
b) the queue size is too small, but this does not seem to be your case.

ENOBUFS basically means that the kernel has to drop Netlink messages 
because your user-space program cannot back-off.

> For every callback, format the data and then put it into the queue
> directly.  The formating of data causes less then 1 second for 0.65
> million records.
> Errors always occurs  when there are less than 10 log records.

I don't understand what you mean here above.

BTW, if you use a recent Linux kernel (>=2.6.30) you can set these two 
socket options not to get ENOBUFS error and to try to improve ctnetlink 
reliability.

int on = 1;

setsockopt(nfct_fd(h), SOL_NETLINK,
            NETLINK_BROADCAST_SEND_ERROR, &on, sizeof(int));

setsockopt(nfct_fd(h), SOL_NETLINK,
            NETLINK_NO_ENOBUFS, &on, sizeof(int));

Re: Why "No buffer space available"?

[Pablo Neira Ayuso]

Medialy wrote:
> Problem solved. Thanks.
> BTW, sometimes the program stops at function nfct_close() and never return!

I don't have an explanation for that, but it should not happen.

> On Wed, Dec 30, 2009 at 8:10 PM, Pablo Neira Ayuso <pablo@netfilter.org 
> <mailto:pablo@netfilter.org>> wrote:
> 
>     Medialy wrote:
> 
>         Hi,
>         I have written a program to log the nat behavior. the program works
>         well when traffic is low. But when the traffic reaches 1Gb, program
>         always error.
>         According to the previous discussions about this problem, I even set
>         the recv buffer size to 50MB and the error still exists.
> 
> 
>     Increasing the buffer size would not solve the problem, that will
>     only delay the ENOBUFS error. There are several reasons why you may
>     hit ENOBUFS:
> 
>     a) your program is too slow to handle the Netlink messages that you
>     receive from the kernel at a given rate. This is easier to trigger
>     if the handling that you perform on every message takes too long.
>     b) the queue size is too small, but this does not seem to be your case.
> 
>     ENOBUFS basically means that the kernel has to drop Netlink messages
>     because your user-space program cannot back-off.
> 
>  
> Medialy:
> Reason: system was overloaded due to the storage capability.  The 
> program (2 threads) was set to use last CPU.  When the traffic was 
> heavy,  most of the computing power of last CPU was occupied by the 
> thread which wrote Netlink messages to the stroage.

Good analysis. It is a good idea to put the thread that digest the 
Netlink message in a spare CPU. That reduces the chances to hit ENOBUFS.

I forgot to say but reducing the nice() value also help to avoid ENOBUFS.