libnetfilter_queue & multithreading & 1 queue freezing

libnetfilter_queue & multithreading & 1 queue freezing

[Oleg]

  Hi, all.

My program process multiple NFQUEUEs by creating a separate thread
for every NFQUEUE. An each thread do recv() and nfq_set_verdict2():

But i catch a strange behaviour when more than 1 thread(queue) are
used. In those cases one queue "freeze" while others works right.
For example, for 5 queues(here a queue with id 1 "freeze"):

# cat /proc/net/netfilter/nfnetlink_queue 
    0  -4216     4 2 65531     0     0   533513  1
    1  19395  1024 2 65531  7319     0     1024  1
    2  -4217     0 2 65531     0     0    62736  1
    3  -4218     0 2 65531     0     0   150717  1
    4  -4215    19 2 65531     0     0  1990176  1

# grep 'start thread' /var/log/messages
Aug 26 17:52:52 kvm trfl[19395]: start thread 2[19398] for nfqueue 2
Aug 26 17:52:52 kvm trfl[19395]: start thread 0[19396] for nfqueue 0
Aug 26 17:52:52 kvm trfl[19395]: start thread 1[19397] for nfqueue 1
Aug 26 17:52:52 kvm trfl[19395]: start thread 4[19400] for nfqueue 4
Aug 26 17:52:52 kvm trfl[19395]: start thread 3[19399] for nfqueue 3

# strace -p 19397
strace: Process 19397 attached
recvfrom(6, ^Cstrace: Process 19397 detached
 <detached ...>

This behaviour is not constantly reproducible.
If i use SO_RCVTIMEO:

  tv.tv_sec = 2;
  tv.tv_usec = 0;
  ret = setsockopt(nfq_fd(h), SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));

  ....

  fd = nfq_fd(h);
  do {
    while ((n = recv(fd, pkt_buf, 80000, 0)) > 0) {
      nfq_handle_packet(h, pkt_buf, n);
    }
    fprintf(stderr, "%u: RECV ERR: %s", thread_idx, strerror(errno));
  } while (errno == EWOULDBLOCK);

then i get every 2 seconds the next message:

0: RECV ERR: Resource temporarily unavailable

for case when thread with id = 0 freeze on recvfrom().

How can i resolve this problem?

Thanks!

-- 
Олег Неманов (Oleg Nemanov)

Re: libnetfilter_queue & multithreading & 1 queue freezing

[Florian Westphal]

Oleg <lego12239@yandex.ru> wrote:
>   Hi, all.
> 
> My program process multiple NFQUEUEs by creating a separate thread
> for every NFQUEUE. An each thread do recv() and nfq_set_verdict2():
> 
> But i catch a strange behaviour when more than 1 thread(queue) are
> used. In those cases one queue "freeze" while others works right.
> For example, for 5 queues(here a queue with id 1 "freeze"):

If you use libnetfilter_queue you might need

commit 4ca06bc967d94b7b7b5a6efc76e870f0efc77e24
src: make nfq_open_nfnl thread-safe

(or protect nfq_open_nfnl calls with a mutex).

Other than that I have no idea.

Re: libnetfilter_queue & multithreading & 1 queue freezing

[Oleg]

On Sat, Aug 26, 2017 at 09:26:25PM +0200, Florian Westphal wrote:
> Oleg <lego12239@yandex.ru> wrote:
> >   Hi, all.
> > 
> > My program process multiple NFQUEUEs by creating a separate thread
> > for every NFQUEUE. An each thread do recv() and nfq_set_verdict2():
> > 
> > But i catch a strange behaviour when more than 1 thread(queue) are
> > used. In those cases one queue "freeze" while others works right.
> > For example, for 5 queues(here a queue with id 1 "freeze"):
> 
> If you use libnetfilter_queue you might need
> 
> commit 4ca06bc967d94b7b7b5a6efc76e870f0efc77e24
> src: make nfq_open_nfnl thread-safe
> 
> (or protect nfq_open_nfnl calls with a mutex).

A mutex around nfq_open() seems to work. At least, 30 restarts
didn't show a frozen on recvfrom() queue.

Thanks!

-- 
Олег Неманов (Oleg Nemanov)