libnetfilter_queue issues

libnetfilter_queue issues

[dorian]

Hi all,
Sorry if the queries has been answered yet but I can't find the reliable
informations about issues concerned with libnetfilter_queue.

I need to run a program which would firstly fork let's say 10 times (or
creates 10 threads) and next each child-process (or thread) will include

   while ((rv = recv(fd, buf, sizeof(buf), 0)) && rv >= 0) {
      nfq_handle_packet(h, buf, rv);
   }

sequence.

The fork/multiple threads reason usage is that for some packets the
processing time will be relatively long.
So I would like to handle next packets in "the meantime" using another
process (or thread)

But I have 2 doubts which I would like to clarify.

1. - - - - - -
Is it possible to handle concurrently several packets with several
processes ?
i.e. if the process1 is handling a packet will the process2 receive the
next existing packet?

The essence of my doubt is:
if process1 receives the packet1 but the verdict is not issued  will
next recv() (in process2) be successful and not hold process2 until
verdict for packet1 is made?

Please confirm.


2. - - - - - -
Let's assume that we have process1 which handles packet 1, process 2
which handles packet 2, etc
Let's say verdict for packet2 is issued before verdict for packet1

Will packet2 leave the queue or has it to "wait" for verdict for packet1 ?

The essence of my doubt is:
can packet2 leave the queue before packet1 ?

In other words can we change the packets order leaving system ?

Or, maybe, although verdict for packet2 is done it has to wait for
verdict for packet1 and next both leave the queue in original order?

I would be obliged if you clarify above two matter

Best Regards,
Dorian

Re: libnetfilter_queue issues

[Felix]

excellent question !
I too am interested in this.

Another similar, related question is do packets that are not filtered
wait on a verdict for a filtered packet?
Again, if not, it means we can change packet order?
If so,it means we can degrade system performance unrelated to our
filtered packets.

Thanks,
Credzba

On Sat, Jan 12, 2013 at 6:43 AM, dorian <dorian33@o2.pl> wrote:
> Hi all,
> Sorry if the queries has been answered yet but I can't find the reliable
> informations about issues concerned with libnetfilter_queue.
>
> I need to run a program which would firstly fork let's say 10 times (or
> creates 10 threads) and next each child-process (or thread) will include
>
>    while ((rv = recv(fd, buf, sizeof(buf), 0)) && rv >= 0) {
>       nfq_handle_packet(h, buf, rv);
>    }
>
> sequence.
>
> The fork/multiple threads reason usage is that for some packets the
> processing time will be relatively long.
> So I would like to handle next packets in "the meantime" using another
> process (or thread)
>
> But I have 2 doubts which I would like to clarify.
>
> 1. - - - - - -
> Is it possible to handle concurrently several packets with several
> processes ?
> i.e. if the process1 is handling a packet will the process2 receive the
> next existing packet?
>
> The essence of my doubt is:
> if process1 receives the packet1 but the verdict is not issued  will
> next recv() (in process2) be successful and not hold process2 until
> verdict for packet1 is made?
>
> Please confirm.
>
>
> 2. - - - - - -
> Let's assume that we have process1 which handles packet 1, process 2
> which handles packet 2, etc
> Let's say verdict for packet2 is issued before verdict for packet1
>
> Will packet2 leave the queue or has it to "wait" for verdict for packet1 ?
>
> The essence of my doubt is:
> can packet2 leave the queue before packet1 ?
>
> In other words can we change the packets order leaving system ?
>
> Or, maybe, although verdict for packet2 is done it has to wait for
> verdict for packet1 and next both leave the queue in original order?
>
> I would be obliged if you clarify above two matter
>
> Best Regards,
> Dorian
>
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: libnetfilter_queue issues

[Felix]

Good job.
I and everyone who needs this information will appreciate your work.

I posted some questions on the site, but they are ipfilter/nfqueue
performance questions.
You answered the original question well.

Thank you.

On Sat, Jan 12, 2013 at 1:04 PM, Eric Leblond <eric@regit.org> wrote:
> Hello,
>
> On Sat, 2013-01-12 at 07:23 -0600, Felix wrote:
>> excellent question !
>> I too am interested in this.
>>
>> Another similar, related question is do packets that are not filtered
>> wait on a verdict for a filtered packet?
>> Again, if not, it means we can change packet order?
>> If so,it means we can degrade system performance unrelated to our
>> filtered packets.
>
> I've just wrote an article describing libnetfilter_queue and NFQUEUE. It
> has been written to answer your questions:
> https://home.regit.org/netfilter-en/using-nfqueue-and-libnetfilter_queue/
>
> Let me know if it is the case and don't hesitate if you have any
> suggestions.
>
> BR,
> --
> Eric
>
>> Thanks,
>> Credzba
>>
>> On Sat, Jan 12, 2013 at 6:43 AM, dorian <dorian33@o2.pl> wrote:
>> > Hi all,
>> > Sorry if the queries has been answered yet but I can't find the reliable
>> > informations about issues concerned with libnetfilter_queue.
>> >
>> > I need to run a program which would firstly fork let's say 10 times (or
>> > creates 10 threads) and next each child-process (or thread) will include
>> >
>> >    while ((rv = recv(fd, buf, sizeof(buf), 0)) && rv >= 0) {
>> >       nfq_handle_packet(h, buf, rv);
>> >    }
>> >
>> > sequence.
>> >
>> > The fork/multiple threads reason usage is that for some packets the
>> > processing time will be relatively long.
>> > So I would like to handle next packets in "the meantime" using another
>> > process (or thread)
>> >
>> > But I have 2 doubts which I would like to clarify.
>> >
>> > 1. - - - - - -
>> > Is it possible to handle concurrently several packets with several
>> > processes ?
>> > i.e. if the process1 is handling a packet will the process2 receive the
>> > next existing packet?
>> >
>> > The essence of my doubt is:
>> > if process1 receives the packet1 but the verdict is not issued  will
>> > next recv() (in process2) be successful and not hold process2 until
>> > verdict for packet1 is made?
>> >
>> > Please confirm.
>> >
>> >
>> > 2. - - - - - -
>> > Let's assume that we have process1 which handles packet 1, process 2
>> > which handles packet 2, etc
>> > Let's say verdict for packet2 is issued before verdict for packet1
>> >
>> > Will packet2 leave the queue or has it to "wait" for verdict for packet1 ?
>> >
>> > The essence of my doubt is:
>> > can packet2 leave the queue before packet1 ?
>> >
>> > In other words can we change the packets order leaving system ?
>> >
>> > Or, maybe, although verdict for packet2 is done it has to wait for
>> > verdict for packet1 and next both leave the queue in original order?
>> >
>> > I would be obliged if you clarify above two matter
>> >
>> > Best Regards,
>> > Dorian
>> >
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe netfilter" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe netfilter" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> --
> Eric Leblond <eric@regit.org>
> Blog: https://home.regit.org/
>

Re: libnetfilter_queue issues

[Eric Leblond]

Hello,

On Sat, 2013-01-12 at 07:23 -0600, Felix wrote:
> excellent question !
> I too am interested in this.
> 
> Another similar, related question is do packets that are not filtered
> wait on a verdict for a filtered packet?
> Again, if not, it means we can change packet order?
> If so,it means we can degrade system performance unrelated to our
> filtered packets.

I've just wrote an article describing libnetfilter_queue and NFQUEUE. It
has been written to answer your questions:
https://home.regit.org/netfilter-en/using-nfqueue-and-libnetfilter_queue/

Let me know if it is the case and don't hesitate if you have any
suggestions.

BR,
--
Eric

> Thanks,
> Credzba
> 
> On Sat, Jan 12, 2013 at 6:43 AM, dorian <dorian33@o2.pl> wrote:
> > Hi all,
> > Sorry if the queries has been answered yet but I can't find the reliable
> > informations about issues concerned with libnetfilter_queue.
> >
> > I need to run a program which would firstly fork let's say 10 times (or
> > creates 10 threads) and next each child-process (or thread) will include
> >
> >    while ((rv = recv(fd, buf, sizeof(buf), 0)) && rv >= 0) {
> >       nfq_handle_packet(h, buf, rv);
> >    }
> >
> > sequence.
> >
> > The fork/multiple threads reason usage is that for some packets the
> > processing time will be relatively long.
> > So I would like to handle next packets in "the meantime" using another
> > process (or thread)
> >
> > But I have 2 doubts which I would like to clarify.
> >
> > 1. - - - - - -
> > Is it possible to handle concurrently several packets with several
> > processes ?
> > i.e. if the process1 is handling a packet will the process2 receive the
> > next existing packet?
> >
> > The essence of my doubt is:
> > if process1 receives the packet1 but the verdict is not issued  will
> > next recv() (in process2) be successful and not hold process2 until
> > verdict for packet1 is made?
> >
> > Please confirm.
> >
> >
> > 2. - - - - - -
> > Let's assume that we have process1 which handles packet 1, process 2
> > which handles packet 2, etc
> > Let's say verdict for packet2 is issued before verdict for packet1
> >
> > Will packet2 leave the queue or has it to "wait" for verdict for packet1 ?
> >
> > The essence of my doubt is:
> > can packet2 leave the queue before packet1 ?
> >
> > In other words can we change the packets order leaving system ?
> >
> > Or, maybe, although verdict for packet2 is done it has to wait for
> > verdict for packet1 and next both leave the queue in original order?
> >
> > I would be obliged if you clarify above two matter
> >
> > Best Regards,
> > Dorian
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe netfilter" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Eric Leblond <eric@regit.org>
Blog: https://home.regit.org/

Re: libnetfilter_queue issues

[dorian]

Hi Eric,
Thanks a lot for reply.

I read your article but apart from clarifying my doubts it created the
new ones :)

Below are the issues....

1.- - - - -
You wrote: "packet can be verdict without order".
I am very sorry but my English is not perfect so I understand above as
"having packets in the order 1,2,3,4 the verdict can be set in any order
(f.ex. in the order 4,1,3,2)"
Am I right?

If my above understanding is correct it also contains implied
information that I can call successfully several times recv() function
and none of the calls block despite the fact that the verdicts have not
been set for the recv'ed packets.

Am I right till now?

2. - - - - - -
Next you wrote
a) "The packet queue is a real queue"
and next
b) "A packet is released when userspace issue a verdict"

Well, I have some doubts here.

According to my primitive understanding (but which is the same as at
http://en.wikipedia.org/wiki/Queue_%28abstract_data_type%29)  the queue
is First-In-First-Out (FIFO) data structure.
So statement (b) is not true or not complete.

Because having several packets in queue (let's say 1,2,3,4) and setting
verdict for packet 2 doesn't mean that it will be released (=sent) since
it is  _in the middle_ of the queue.

Please explain above.

Anyway at the bottom you also mentioned about 'packet reordering' which
suggests that queue you are writing about in the article is not a queue
or my understanding of therm 'queue' is too primitive.

3. - - - - - -
And "new doubt" appeared after reading your article.
You wrote: "The send/recv operation need to be protected by lock to
avoid concurrent writing"

Well, I wrote a number of  multi-processed programs (run also on
multi-core CPUs) which used the same file descriptor by each
child-process and I never found the situation that any locking has been
required.
Each read or written messages (=sequence of bytes)  were consistent.
Of course without locking I do not have control on the order of the
messages read/written from/to file descriptor but none of the
read/written data were corrupted (i.e. messages were not mixed)

So is the lock really required? Why?
Is it a matter of the fact that 'fd' is the netlink socket?


4. - - - - - -
Another "new doubt" requires longer explanation.

Using the file 'source-nfq.c' as reference we can found there

   while ((rv = recv(fd, buf, sizeof(buf), 0)) && rv >= 0) {
      nfq_handle_packet(h, buf, rv);
   }


At the article you wrote: "the kernel sends a message containing packet
data and related information to a socket and userpace reads this message"
Does it means that 'buf' contains packet data?

I am asking about since 'fd' is the only socket I can find in the
source-nfq.c.
On the other hand the packet data are accessible via struct nfq_data
*nfa in callback function and 'nfa' has nothing in common (at least
documentation does not state it) with the 'buf'.

And if the 'buf' really contains packet data what for the callback
function is required?
Wasn't be simpler to access packet directly from 'buf'?

Till now I assumed that 'buf' contained only some kind of 
"libnetfilter_queue internal metadata" which was not interesting for me.

Any comment will be appreciated.


- - - - - - - -
And that's all.
I would be be grateful to you for helping me to understand how to use
the libnetfilter_queue and what can be done using it.

With best regards,
Dorian



> Hello,
>
> On Sat, 2013-01-12 at 07:23 -0600, Felix wrote:
>> excellent question !
>> I too am interested in this.
>>
>> Another similar, related question is do packets that are not filtered
>> wait on a verdict for a filtered packet?
>> Again, if not, it means we can change packet order?
>> If so,it means we can degrade system performance unrelated to our
>> filtered packets.
> I've just wrote an article describing libnetfilter_queue and NFQUEUE. It
> has been written to answer your questions:
> https://home.regit.org/netfilter-en/using-nfqueue-and-libnetfilter_queue/
>
> Let me know if it is the case and don't hesitate if you have any
> suggestions.
>
> BR,
> --
> Eric
>
>> Thanks,
>> Credzba
>>
>> On Sat, Jan 12, 2013 at 6:43 AM, dorian <dorian33@o2.pl> wrote:
>>> Hi all,
>>> Sorry if the queries has been answered yet but I can't find the reliable
>>> informations about issues concerned with libnetfilter_queue.
>>>
>>> I need to run a program which would firstly fork let's say 10 times (or
>>> creates 10 threads) and next each child-process (or thread) will include
>>>
>>>    while ((rv = recv(fd, buf, sizeof(buf), 0)) && rv >= 0) {
>>>       nfq_handle_packet(h, buf, rv);
>>>    }
>>>
>>> sequence.
>>>
>>> The fork/multiple threads reason usage is that for some packets the
>>> processing time will be relatively long.
>>> So I would like to handle next packets in "the meantime" using another
>>> process (or thread)
>>>
>>> But I have 2 doubts which I would like to clarify.
>>>
>>> 1. - - - - - -
>>> Is it possible to handle concurrently several packets with several
>>> processes ?
>>> i.e. if the process1 is handling a packet will the process2 receive the
>>> next existing packet?
>>>
>>> The essence of my doubt is:
>>> if process1 receives the packet1 but the verdict is not issued  will
>>> next recv() (in process2) be successful and not hold process2 until
>>> verdict for packet1 is made?
>>>
>>> Please confirm.
>>>
>>>
>>> 2. - - - - - -
>>> Let's assume that we have process1 which handles packet 1, process 2
>>> which handles packet 2, etc
>>> Let's say verdict for packet2 is issued before verdict for packet1
>>>
>>> Will packet2 leave the queue or has it to "wait" for verdict for packet1 ?
>>>
>>> The essence of my doubt is:
>>> can packet2 leave the queue before packet1 ?
>>>
>>> In other words can we change the packets order leaving system ?
>>>
>>> Or, maybe, although verdict for packet2 is done it has to wait for
>>> verdict for packet1 and next both leave the queue in original order?
>>>
>>> I would be obliged if you clarify above two matter
>>>
>>> Best Regards,
>>> Dorian
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe netfilter" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe netfilter" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: libnetfilter_queue issues

[Eric Leblond]

Hello,

Thanks for all these questions, I will improve/update the article thanks
to that.

On Sat, 2013-01-12 at 22:16 +0100, dorian wrote:
> Hi Eric,
> Thanks a lot for reply.
> 
> I read your article but apart from clarifying my doubts it created the
> new ones :)
> 
> Below are the issues....
> 
> 1.- - - - -
> You wrote: "packet can be verdict without order".
> I am very sorry but my English is not perfect so I understand above as
> "having packets in the order 1,2,3,4 the verdict can be set in any order
> (f.ex. in the order 4,1,3,2)"
> Am I right?

Exactly.

> If my above understanding is correct it also contains implied
> information that I can call successfully several times recv() function
> and none of the calls block despite the fact that the verdicts have not
> been set for the recv'ed packets.
> Am I right till now?

Yes. You can read packets, take a coffee and came back later to verdict
them all at once.

> 
> 2. - - - - - -
> Next you wrote
> a) "The packet queue is a real queue"
> and next
> b) "A packet is released when userspace issue a verdict"
> 
> Well, I have some doubts here.
> 
> According to my primitive understanding (but which is the same as at
> http://en.wikipedia.org/wiki/Queue_%28abstract_data_type%29)  the queue
> is First-In-First-Out (FIFO) data structure.
> So statement (b) is not true or not complete.
> 
> Because having several packets in queue (let's say 1,2,3,4) and setting
> verdict for packet 2 doesn't mean that it will be released (=sent) since
> it is  _in the middle_ of the queue.

OK real queue was not a correct term. See explanation below.

> Please explain above.
> 
> Anyway at the bottom you also mentioned about 'packet reordering' which
> suggests that queue you are writing about in the article is not a queue
> or my understanding of therm 'queue' is too primitive.

It is a chained list with the following operation:
      * Enqueue: element is added at the end
      * Dequeue (verdict): element is removed from the chained list.

> 3. - - - - - -
> And "new doubt" appeared after reading your article.
> You wrote: "The send/recv operation need to be protected by lock to
> avoid concurrent writing"
> 
> Well, I wrote a number of  multi-processed programs (run also on
> multi-core CPUs) which used the same file descriptor by each
> child-process and I never found the situation that any locking has been
> required.
> Each read or written messages (=sequence of bytes)  were consistent.
> Of course without locking I do not have control on the order of the
> messages read/written from/to file descriptor but none of the
> read/written data were corrupted (i.e. messages were not mixed)
> 
> So is the lock really required? Why?
> Is it a matter of the fact that 'fd' is the netlink socket?

Humm, you may have a point here. I'm doing this by habit and a check
could be needed.

> 4. - - - - - -
> Another "new doubt" requires longer explanation.
> 
> Using the file 'source-nfq.c' as reference we can found there
> 
>    while ((rv = recv(fd, buf, sizeof(buf), 0)) && rv >= 0) {
>       nfq_handle_packet(h, buf, rv);
>    }

> At the article you wrote: "the kernel sends a message containing packet
> data and related information to a socket and userpace reads this message"
> Does it means that 'buf' contains packet data?
> 
> I am asking about since 'fd' is the only socket I can find in the
> source-nfq.c.
> On the other hand the packet data are accessible via struct nfq_data
> *nfa in callback function and 'nfa' has nothing in common (at least
> documentation does not state it) with the 'buf'.
> 
> And if the 'buf' really contains packet data what for the callback
> function is required?
> Wasn't be simpler to access packet directly from 'buf'?
> 
> Till now I assumed that 'buf' contained only some kind of 
> "libnetfilter_queue internal metadata" which was not interesting for me.

buf is containing a nfnetlink message not the packet by itself. The
protocol messages are nfnetlink formatted.

> Any comment will be appreciated.
> 
> 
> - - - - - - - -
> And that's all.
> I would be be grateful to you for helping me to understand how to use
> the libnetfilter_queue and what can be done using it.
> 
> With best regards,
> Dorian

BR,
--
Eric


> 
> 
> 
> > Hello,
> >
> > On Sat, 2013-01-12 at 07:23 -0600, Felix wrote:
> >> excellent question !
> >> I too am interested in this.
> >>
> >> Another similar, related question is do packets that are not filtered
> >> wait on a verdict for a filtered packet?
> >> Again, if not, it means we can change packet order?
> >> If so,it means we can degrade system performance unrelated to our
> >> filtered packets.
> > I've just wrote an article describing libnetfilter_queue and NFQUEUE. It
> > has been written to answer your questions:
> > https://home.regit.org/netfilter-en/using-nfqueue-and-libnetfilter_queue/
> >
> > Let me know if it is the case and don't hesitate if you have any
> > suggestions.
> >
> > BR,
> > --
> > Eric
> >
> >> Thanks,
> >> Credzba
> >>
> >> On Sat, Jan 12, 2013 at 6:43 AM, dorian <dorian33@o2.pl> wrote:
> >>> Hi all,
> >>> Sorry if the queries has been answered yet but I can't find the reliable
> >>> informations about issues concerned with libnetfilter_queue.
> >>>
> >>> I need to run a program which would firstly fork let's say 10 times (or
> >>> creates 10 threads) and next each child-process (or thread) will include
> >>>
> >>>    while ((rv = recv(fd, buf, sizeof(buf), 0)) && rv >= 0) {
> >>>       nfq_handle_packet(h, buf, rv);
> >>>    }
> >>>
> >>> sequence.
> >>>
> >>> The fork/multiple threads reason usage is that for some packets the
> >>> processing time will be relatively long.
> >>> So I would like to handle next packets in "the meantime" using another
> >>> process (or thread)
> >>>
> >>> But I have 2 doubts which I would like to clarify.
> >>>
> >>> 1. - - - - - -
> >>> Is it possible to handle concurrently several packets with several
> >>> processes ?
> >>> i.e. if the process1 is handling a packet will the process2 receive the
> >>> next existing packet?
> >>>
> >>> The essence of my doubt is:
> >>> if process1 receives the packet1 but the verdict is not issued  will
> >>> next recv() (in process2) be successful and not hold process2 until
> >>> verdict for packet1 is made?
> >>>
> >>> Please confirm.
> >>>
> >>>
> >>> 2. - - - - - -
> >>> Let's assume that we have process1 which handles packet 1, process 2
> >>> which handles packet 2, etc
> >>> Let's say verdict for packet2 is issued before verdict for packet1
> >>>
> >>> Will packet2 leave the queue or has it to "wait" for verdict for packet1 ?
> >>>
> >>> The essence of my doubt is:
> >>> can packet2 leave the queue before packet1 ?
> >>>
> >>> In other words can we change the packets order leaving system ?
> >>>
> >>> Or, maybe, although verdict for packet2 is done it has to wait for
> >>> verdict for packet1 and next both leave the queue in original order?
> >>>
> >>> I would be obliged if you clarify above two matter
> >>>
> >>> Best Regards,
> >>> Dorian
> >>>
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> >>> the body of a message to majordomo@vger.kernel.org
> >>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Eric Leblond <eric@regit.org>
Blog: https://home.regit.org/

Re: libnetfilter_queue issues

[dorian]

Many thanks for explanations & updates.

Seems now I have a proper (although probably not full) image how
libnetfilter_queue works

Thanks once again.

Dorian

> Hello,
>
> Thanks for all these questions, I will improve/update the article thanks
> to that.
>
> On Sat, 2013-01-12 at 22:16 +0100, dorian wrote:
>> Hi Eric,
>> Thanks a lot for reply.
>>
>> I read your article but apart from clarifying my doubts it created the
>> new ones :)
>>
>> Below are the issues....
>>
>> 1.- - - - -
>> You wrote: "packet can be verdict without order".
>> I am very sorry but my English is not perfect so I understand above as
>> "having packets in the order 1,2,3,4 the verdict can be set in any order
>> (f.ex. in the order 4,1,3,2)"
>> Am I right?
> Exactly.
>
>> If my above understanding is correct it also contains implied
>> information that I can call successfully several times recv() function
>> and none of the calls block despite the fact that the verdicts have not
>> been set for the recv'ed packets.
>> Am I right till now?
> Yes. You can read packets, take a coffee and came back later to verdict
> them all at once.
>
>> 2. - - - - - -
>> Next you wrote
>> a) "The packet queue is a real queue"
>> and next
>> b) "A packet is released when userspace issue a verdict"
>>
>> Well, I have some doubts here.
>>
>> According to my primitive understanding (but which is the same as at
>> http://en.wikipedia.org/wiki/Queue_%28abstract_data_type%29)  the queue
>> is First-In-First-Out (FIFO) data structure.
>> So statement (b) is not true or not complete.
>>
>> Because having several packets in queue (let's say 1,2,3,4) and setting
>> verdict for packet 2 doesn't mean that it will be released (=sent) since
>> it is  _in the middle_ of the queue.
> OK real queue was not a correct term. See explanation below.
>
>> Please explain above.
>>
>> Anyway at the bottom you also mentioned about 'packet reordering' which
>> suggests that queue you are writing about in the article is not a queue
>> or my understanding of therm 'queue' is too primitive.
> It is a chained list with the following operation:
>       * Enqueue: element is added at the end
>       * Dequeue (verdict): element is removed from the chained list.
>
>> 3. - - - - - -
>> And "new doubt" appeared after reading your article.
>> You wrote: "The send/recv operation need to be protected by lock to
>> avoid concurrent writing"
>>
>> Well, I wrote a number of  multi-processed programs (run also on
>> multi-core CPUs) which used the same file descriptor by each
>> child-process and I never found the situation that any locking has been
>> required.
>> Each read or written messages (=sequence of bytes)  were consistent.
>> Of course without locking I do not have control on the order of the
>> messages read/written from/to file descriptor but none of the
>> read/written data were corrupted (i.e. messages were not mixed)
>>
>> So is the lock really required? Why?
>> Is it a matter of the fact that 'fd' is the netlink socket?
> Humm, you may have a point here. I'm doing this by habit and a check
> could be needed.
>
>> 4. - - - - - -
>> Another "new doubt" requires longer explanation.
>>
>> Using the file 'source-nfq.c' as reference we can found there
>>
>>    while ((rv = recv(fd, buf, sizeof(buf), 0)) && rv >= 0) {
>>       nfq_handle_packet(h, buf, rv);
>>    }
>> At the article you wrote: "the kernel sends a message containing packet
>> data and related information to a socket and userpace reads this message"
>> Does it means that 'buf' contains packet data?
>>
>> I am asking about since 'fd' is the only socket I can find in the
>> source-nfq.c.
>> On the other hand the packet data are accessible via struct nfq_data
>> *nfa in callback function and 'nfa' has nothing in common (at least
>> documentation does not state it) with the 'buf'.
>>
>> And if the 'buf' really contains packet data what for the callback
>> function is required?
>> Wasn't be simpler to access packet directly from 'buf'?
>>
>> Till now I assumed that 'buf' contained only some kind of 
>> "libnetfilter_queue internal metadata" which was not interesting for me.
> buf is containing a nfnetlink message not the packet by itself. The
> protocol messages are nfnetlink formatted.
>
>> Any comment will be appreciated.
>>
>>
>> - - - - - - - -
>> And that's all.
>> I would be be grateful to you for helping me to understand how to use
>> the libnetfilter_queue and what can be done using it.
>>
>> With best regards,
>> Dorian
> BR,
> --
> Eric
>
>
>>
>>
>>> Hello,
>>>
>>> On Sat, 2013-01-12 at 07:23 -0600, Felix wrote:
>>>> excellent question !
>>>> I too am interested in this.
>>>>
>>>> Another similar, related question is do packets that are not filtered
>>>> wait on a verdict for a filtered packet?
>>>> Again, if not, it means we can change packet order?
>>>> If so,it means we can degrade system performance unrelated to our
>>>> filtered packets.
>>> I've just wrote an article describing libnetfilter_queue and NFQUEUE. It
>>> has been written to answer your questions:
>>> https://home.regit.org/netfilter-en/using-nfqueue-and-libnetfilter_queue/
>>>
>>> Let me know if it is the case and don't hesitate if you have any
>>> suggestions.
>>>
>>> BR,
>>> --
>>> Eric
>>>
>>>> Thanks,
>>>> Credzba
>>>>
>>>> On Sat, Jan 12, 2013 at 6:43 AM, dorian <dorian33@o2.pl> wrote:
>>>>> Hi all,
>>>>> Sorry if the queries has been answered yet but I can't find the reliable
>>>>> informations about issues concerned with libnetfilter_queue.
>>>>>
>>>>> I need to run a program which would firstly fork let's say 10 times (or
>>>>> creates 10 threads) and next each child-process (or thread) will include
>>>>>
>>>>>    while ((rv = recv(fd, buf, sizeof(buf), 0)) && rv >= 0) {
>>>>>       nfq_handle_packet(h, buf, rv);
>>>>>    }
>>>>>
>>>>> sequence.
>>>>>
>>>>> The fork/multiple threads reason usage is that for some packets the
>>>>> processing time will be relatively long.
>>>>> So I would like to handle next packets in "the meantime" using another
>>>>> process (or thread)
>>>>>
>>>>> But I have 2 doubts which I would like to clarify.
>>>>>
>>>>> 1. - - - - - -
>>>>> Is it possible to handle concurrently several packets with several
>>>>> processes ?
>>>>> i.e. if the process1 is handling a packet will the process2 receive the
>>>>> next existing packet?
>>>>>
>>>>> The essence of my doubt is:
>>>>> if process1 receives the packet1 but the verdict is not issued  will
>>>>> next recv() (in process2) be successful and not hold process2 until
>>>>> verdict for packet1 is made?
>>>>>
>>>>> Please confirm.
>>>>>
>>>>>
>>>>> 2. - - - - - -
>>>>> Let's assume that we have process1 which handles packet 1, process 2
>>>>> which handles packet 2, etc
>>>>> Let's say verdict for packet2 is issued before verdict for packet1
>>>>>
>>>>> Will packet2 leave the queue or has it to "wait" for verdict for packet1 ?
>>>>>
>>>>> The essence of my doubt is:
>>>>> can packet2 leave the queue before packet1 ?
>>>>>
>>>>> In other words can we change the packets order leaving system ?
>>>>>
>>>>> Or, maybe, although verdict for packet2 is done it has to wait for
>>>>> verdict for packet1 and next both leave the queue in original order?
>>>>>
>>>>> I would be obliged if you clarify above two matter
>>>>>
>>>>> Best Regards,
>>>>> Dorian
>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe netfilter" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe netfilter" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe netfilter" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: libnetfilter_queue issues

[Pablo Neira Ayuso]

On Sat, Jan 12, 2013 at 08:04:01PM +0100, Eric Leblond wrote:
> Hello,
> 
> On Sat, 2013-01-12 at 07:23 -0600, Felix wrote:
> > excellent question !
> > I too am interested in this.
> > 
> > Another similar, related question is do packets that are not filtered
> > wait on a verdict for a filtered packet?
> > Again, if not, it means we can change packet order?
> > If so,it means we can degrade system performance unrelated to our
> > filtered packets.
> 
> I've just wrote an article describing libnetfilter_queue and NFQUEUE. It
> has been written to answer your questions:
> https://home.regit.org/netfilter-en/using-nfqueue-and-libnetfilter_queue/

It would be good to integrate part of this information to the doxygen
library documentation:

http://www.netfilter.org/projects/libnetfilter_queue/

Or simply add reference to Eric's page in the main page of the doxygen.

Regards.

Re: libnetfilter_queue issues

[Eric Leblond]

Hello,

On Sun, 2013-01-13 at 12:29 +0100, Pablo Neira Ayuso wrote:
> On Sat, Jan 12, 2013 at 08:04:01PM +0100, Eric Leblond wrote:
> > Hello,
> > 
> > On Sat, 2013-01-12 at 07:23 -0600, Felix wrote:
> > > excellent question !
> > > I too am interested in this.
> > > 
> > > Another similar, related question is do packets that are not filtered
> > > wait on a verdict for a filtered packet?
> > > Again, if not, it means we can change packet order?
> > > If so,it means we can degrade system performance unrelated to our
> > > filtered packets.
> > 
> > I've just wrote an article describing libnetfilter_queue and NFQUEUE. It
> > has been written to answer your questions:
> > https://home.regit.org/netfilter-en/using-nfqueue-and-libnetfilter_queue/
> 
> It would be good to integrate part of this information to the doxygen
> library documentation:
> 
> http://www.netfilter.org/projects/libnetfilter_queue/
> 
> Or simply add reference to Eric's page in the main page of the doxygen.

To follow a small patchset trying to improve libnetfilter_queue doxygen
documentation.

BR,
-- 
Eric Leblond <eric@regit.org>
Blog: https://home.regit.org/

[libnetfilter_queue PATCH 1/2] doxygen: improve fail-open documentation.

[Eric Leblond]

---
 src/libnetfilter_queue.c |    8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/src/libnetfilter_queue.c b/src/libnetfilter_queue.c
index 9fe9dfa..f93ac66 100644
--- a/src/libnetfilter_queue.c
+++ b/src/libnetfilter_queue.c
@@ -87,6 +87,8 @@
  * (requires Linux kernel >= 2.6.30).
  * - see --queue-balance option in NFQUEUE target for multi-threaded apps
  * (it requires Linux kernel >= 2.6.31).
+ * - consider using fail-open option see nfq_set_queue_flags() (it requires
+ *  Linux kernel >= 3.6)
  */
 
 struct nfq_handle
@@ -620,6 +622,12 @@ int nfq_set_mode(struct nfq_q_handle *qh,
 	flags &= ~NFQA_CFG_F_FAIL_OPEN;
 	err = nfq_set_queue_flags(qh, mask, flags);
 \endverbatim
+ *
+ * If NFQA_CFG_F_FAIL_OPEN is used, the kernel will accept instead of
+ * drop packets that should have been enqueued to a full queue. This
+ * results in the system being able to handle high network load but at
+ * the depend of the control on the packets.
+ *
  * \return -1 on error with errno set appropriately; =0 otherwise.
  */
 int nfq_set_queue_flags(struct nfq_q_handle *qh,
-- 
1.7.10.4

[libnetfilter_queue PATCH 2/2] doxygen: improve documentation

[Eric Leblond]

This patch improves the doxygen documentation and adds a reference
to an external article.
---
 src/libnetfilter_queue.c |   22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

diff --git a/src/libnetfilter_queue.c b/src/libnetfilter_queue.c
index f93ac66..be25217 100644
--- a/src/libnetfilter_queue.c
+++ b/src/libnetfilter_queue.c
@@ -51,7 +51,21 @@
  *  - receiving queued packets from the kernel nfnetlink_queue subsystem
  *  - issuing verdicts and/or reinjecting altered packets to the kernel
  *  nfnetlink_queue subsystem
- * 
+ *
+ * The cinematic is the following: When an iptables rules with target NFQUEUE
+ * matches, the kernel en-queued the packet in a chained list. It then format
+ * a nfnetlink message and sends the information (packet data , packet id and
+ * metadata) via a socket to the software connected to the queue. The software
+ * can then read the message.
+ *
+ * To remove the packet from the queue, the userspace software must issue a
+ * verdict asking kernel to accept or drop the packet. Userspace can also alter
+ * the packet. Verdict can be done in asynchronous manner, as the only needed
+ * information is the packet id.
+ *
+ * When a queue is full, packets that should have been en-queued are dropped by
+ * kernel instead of being en-queued.
+ *
  * \section Git Tree
  * The current development version of libnetfilter_queue can be accessed
  * at https://git.netfilter.org/cgi-bin/gitweb.cgi?p=libnetfilter_queue.git;a=summary.
@@ -66,6 +80,10 @@
  * the doxygen documentation (start by \link LibrarySetup \endlink page) and
  * nf-queue.c source file.
  *
+ * Another source of information on libnetfilter_queue usage is the following
+ * article:
+ *  https://home.regit.org/netfilter-en/using-nfqueue-and-libnetfilter_queue/
+ *
  * \section errors ENOBUFS errors in recv()
  *
  * recv() may return -1 and errno is set to ENOBUFS in case that your
@@ -89,6 +107,8 @@
  * (it requires Linux kernel >= 2.6.31).
  * - consider using fail-open option see nfq_set_queue_flags() (it requires
  *  Linux kernel >= 3.6)
+ * - increase queue max length with nfq_set_queue_maxlen() to resist to packets
+ * burst
  */
 
 struct nfq_handle
-- 
1.7.10.4

Re: libnetfilter_queue issues

[dorian]

Hi Eric,
Thanks a lot for explanations.

I would like ask you about another matter.

Previously, without deeper knowledge about dequeuing rule based on
verdict rather then the order of the packets  I've tried to implement
another solution.

I assumed that for queued packets 1,2,3,4 the verdict should be set in
the (same) order 1,2,3,4.

But since for packet 2 I needed to make some processing (which took a
while) and not wanting to block the packets 3 & 4 I used the following
strategy in my program.

packet1 -> NF_ACCEPT
packet2 -> need to be processed -> copy packet to program buffer -> NF_DROP
packet3 -> NF_ACCEPT
packet4 -> NF_ACCEPT

And the copy of the packet was processed "off-line" and finally the
packet should  be  "accepted"

For this I needed to find out how to re-insert the copy of the packet 2
into the network and  raw udp socket seemed to be right choice.

And I found the problem - using sendto() function I observed that the
packet incoming to the client was altered (somewhere in the system).

To be more specific:  all above concerned http packets transmitted from
a server to the client and the problem was that the tcp source
(server's) port  has been altered.

Additionally tcp checksum was changed for new but correct value (but
probably this was recalculated by interface)

Finally the packet arrived to the destination host but was refused by
the host as it was not from port 80.

I detected also that above "change" concerns only packets without 'fin'
flag.

Packet2 with 'fin' flag has been delivered properly.



I understand that above does not directly concern libnetfilter_queue
matters but it is an issue which explanation maybe interesting.

Maybe you have some knowledge why above could happened?

Why system modifies the packet (and additionally in specific packet
place: always source port) sent with raw socket?

And how to send packet without this unexpected system interventions ?

Any advice will be appreciated.

Best regards,
Dorian



> Hello,
>
> On Sun, 2013-01-13 at 12:29 +0100, Pablo Neira Ayuso wrote:
>> On Sat, Jan 12, 2013 at 08:04:01PM +0100, Eric Leblond wrote:
>>> Hello,
>>>
>>> On Sat, 2013-01-12 at 07:23 -0600, Felix wrote:
>>>> excellent question !
>>>> I too am interested in this.
>>>>
>>>> Another similar, related question is do packets that are not filtered
>>>> wait on a verdict for a filtered packet?
>>>> Again, if not, it means we can change packet order?
>>>> If so,it means we can degrade system performance unrelated to our
>>>> filtered packets.
>>> I've just wrote an article describing libnetfilter_queue and NFQUEUE. It
>>> has been written to answer your questions:
>>> https://home.regit.org/netfilter-en/using-nfqueue-and-libnetfilter_queue/
>> It would be good to integrate part of this information to the doxygen
>> library documentation:
>>
>> http://www.netfilter.org/projects/libnetfilter_queue/
>>
>> Or simply add reference to Eric's page in the main page of the doxygen.
> To follow a small patchset trying to improve libnetfilter_queue doxygen
> documentation.
>
> BR,