ENOBUFS in nfq under heavy load

ENOBUFS in nfq under heavy load

[Stanisław Pitucha]

Hi
After running nf_queue app under heavy load I get ENOBUFS error. Ok -
maybe not that heavy - around 100 concurrent voip calls - lots of
small udp packets. Nfq socket suddenly hangs. recv() from nfnl_fd()
socket returns -1, errno=ENOBUFS and when I try to destroy the queue
it stops:
stacktrace with gdb breaking...

#0  0xb7f38410 in __kernel_vsyscall ()
#1  0x4d23d0e1 in recvmsg () from /lib/libc.so.6
#2  0xb7f3185b in nfnl_talk (nfnlh=0x8061080, n=0xbfc9b5ec, peer=0,
groups=0, answer=0x0, junk=0, jarg=0x0)
    at ../../libnfnetlink-0.0.25/src/libnfnetlink.c:659
#3  0xb7f35055 in __build_send_cfg_msg (h=0x80611c0, command=<value
optimized out>, queuenum=<value optimized out>, pf=0)
    at ../../libnetfilter_queue-0.0.13/src/libnetfilter_queue.c:112
#4  0xb7f3511f in nfq_destroy_queue (qh=0x80611f8) at
../../libnetfilter_queue-0.0.13/src/libnetfilter_queue.c:258
#5  0x08049293 in main () at main.cc:287

Any idea how to avoid / fix it? Is the problem on my side?
My bufsize is 65535 for reading from nfq socket.

Thanks
Stan

Re: ENOBUFS in nfq under heavy load

[Eric Leblond]

[-- Attachment #1: Type: text/plain, Size: 788 bytes --]

Hi,

Le mardi 17 juillet 2007 à 16:46 +0100, Stanisław Pitucha a écrit :
> Hi
> After running nf_queue app under heavy load I get ENOBUFS error. Ok -
> maybe not that heavy - around 100 concurrent voip calls - lots of
> small udp packets. Nfq socket suddenly hangs. recv() from nfnl_fd()
> socket returns -1, errno=ENOBUFS and when I try to destroy the queue
> it stops:
> stacktrace with gdb breaking...
> 
> Any idea how to avoid / fix it? 

You could simply use the limit match to avoid "heavy" load.

> Is the problem on my side?

No, it's an implementation problem.

> My bufsize is 65535 for reading from nfq socket.
> 
> Thanks
> Stan

BR,
-- 
Éric Leblond, eleblond@inl.fr
Téléphone : 01 44 89 46 39, Fax : 01 44 89 45 01
INL, http://www.inl.fr

[-- Attachment #2: Ceci est une partie de message numériquement signée --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: ENOBUFS in nfq under heavy load

[Patrick McHardy]

Stanis³aw Pitucha wrote:
> After running nf_queue app under heavy load I get ENOBUFS error. Ok -
> maybe not that heavy - around 100 concurrent voip calls - lots of
> small udp packets. Nfq socket suddenly hangs. recv() from nfnl_fd()
> socket returns -1, errno=ENOBUFS and when I try to destroy the queue
> it stops:

Which kernel version? What exactly do you mean with "it stops"?

Re: ENOBUFS in nfq under heavy load

[Stanisław Pitucha]

Sorry for private mails - my bad.

On 7/17/07, Eric Leblond <e.leblond@inl.fr> wrote:
> You could simply use the limit match to avoid "heavy" load.

Not an option - I need to balance some traffic - so it will get much
more connections than that by default :/

-----

On 7/17/07, Patrick McHardy <kaber@trash.net> wrote:
> Which kernel version? What exactly do you mean with "it stops"?

2.6.21.5 generic, stock - no patches

By stops I mean that it hangs on nfq_destroy_queue() if there was
ENOBUFS before.
When I connected gdb to it I got the stacktrace from the first post.
When I break it with ctrl+c, I get:

NFNETLINK answers: No such device

on console and program quits.

Re: ENOBUFS in nfq under heavy load

[Patrick McHardy]

Stanis³aw Pitucha wrote:
>
> On 7/17/07, Patrick McHardy <kaber@trash.net> wrote:
>> Which kernel version? What exactly do you mean with "it stops"?
>
> 2.6.21.5 generic, stock - no patches
>
> By stops I mean that it hangs on nfq_destroy_queue() if there was
> ENOBUFS before.

Please try to find out what the kernel is doing using sysrq.

> When I connected gdb to it I got the stacktrace from the first post.
> When I break it with ctrl+c, I get:
>
> NFNETLINK answers: No such device
>
> on console and program quits.
>

ENODEV can happen when sending a verdict for or unbinding a non-existant
queue.

Re: ENOBUFS in nfq under heavy load

[Stanisław Pitucha]

On 7/17/07, Patrick McHardy <kaber@trash.net> wrote:
> Stanis³aw Pitucha wrote:
> > By stops I mean that it hangs on nfq_destroy_queue() if there was
> > ENOBUFS before.
>
> Please try to find out what the kernel is doing using sysrq.

Right... not the kernel really :) - libnfnetlink stops at
libnfnetlink-0.0.25/src/libnfnetlink.c:659 (in nfnl_talk()). It's just
waiting for a datagram from netfilter - recvmsg(nfnlh->fd, &msg, 0).
So kernel as far as I can tell... forgot about that socket completely
- so nothing happens. I can't see any calls to NF happening after
that. If there's any possibility to get NF in debug mode, I can
recompile and post it - is there?
But I don't know if that's recoverable at all - should I be trying to
close queue after ENOBUFS? Maybe the socket is closed by design.

> ENODEV can happen when sending a verdict for or unbinding a non-existant
> queue.

http://pastebin.ca/624472
here I get ENOBUFS on read on line 4, so it breaks out of loop and
tries to clean up.

Additionally - I tried to force opening new queue after receiving
ENOBUFS. It does one or two successful reads and throws another
ENOBUFS.

Anyone knows how to solve this? Or should I just write own module?...