i386 vs x86_64 struct tpacket_hdr layout

i386 vs x86_64 struct tpacket_hdr layout

[Norbert van Bolhuis]

I'm compiling my 32bit application with -m32 on a x86_64 system/kernel.

Things aren't working because for my application tp_len is at offset 4
but for the kernel it is at offset 8.

struct tpacket_hdr {
         unsigned long   tp_status;
         unsigned int    tp_len;
         unsigned int    tp_snaplen;
         unsigned short  tp_mac;
         unsigned short  tp_net;
         unsigned int    tp_sec;
         unsigned int    tp_usec;
};

How is this suppose to work ?

thanks,
Norbert van Bolhuis

Re: i386 vs x86_64 struct tpacket_hdr layout

[David Miller]

From: Norbert van Bolhuis <nvbolhuis@aimvalley.nl>
Date: Fri, 20 Dec 2013 14:30:06 +0100

> 
> I'm compiling my 32bit application with -m32 on a x86_64
> system/kernel.
> 
> Things aren't working because for my application tp_len is at offset 4
> but for the kernel it is at offset 8.
> 
> struct tpacket_hdr {
>         unsigned long   tp_status;
>         unsigned int    tp_len;
>         unsigned int    tp_snaplen;
>         unsigned short  tp_mac;
>         unsigned short  tp_net;
>         unsigned int    tp_sec;
>         unsigned int    tp_usec;
> };
> 
> How is this suppose to work ?

This is why you should use tpacket layout v2 or v3, rather than v1,
they fix these issues.

Re: i386 vs x86_64 struct tpacket_hdr layout

[Daniel Borkmann]

On 12/20/2013 07:38 PM, David Miller wrote:
> From: Norbert van Bolhuis <nvbolhuis@aimvalley.nl>
> Date: Fri, 20 Dec 2013 14:30:06 +0100
>
>>
>> I'm compiling my 32bit application with -m32 on a x86_64
>> system/kernel.
>>
>> Things aren't working because for my application tp_len is at offset 4
>> but for the kernel it is at offset 8.
>>
>> struct tpacket_hdr {
>>          unsigned long   tp_status;
>>          unsigned int    tp_len;
>>          unsigned int    tp_snaplen;
>>          unsigned short  tp_mac;
>>          unsigned short  tp_net;
>>          unsigned int    tp_sec;
>>          unsigned int    tp_usec;
>> };
>>
>> How is this suppose to work ?
>
> This is why you should use tpacket layout v2 or v3, rather than v1,
> they fix these issues.

Norbert, please also read Documentation/networking/packet_mmap.txt

Thanks !

Re: i386 vs x86_64 struct tpacket_hdr layout

[Norbert van Bolhuis]

On 12/20/13 19:45, Daniel Borkmann wrote:
> On 12/20/2013 07:38 PM, David Miller wrote:
>> From: Norbert van Bolhuis <nvbolhuis@aimvalley.nl>
>> Date: Fri, 20 Dec 2013 14:30:06 +0100
>>
>>>
>>> I'm compiling my 32bit application with -m32 on a x86_64
>>> system/kernel.
>>>
>>> Things aren't working because for my application tp_len is at offset 4
>>> but for the kernel it is at offset 8.
>>>
>>> struct tpacket_hdr {
>>> unsigned long tp_status;
>>> unsigned int tp_len;
>>> unsigned int tp_snaplen;
>>> unsigned short tp_mac;
>>> unsigned short tp_net;
>>> unsigned int tp_sec;
>>> unsigned int tp_usec;
>>> };
>>>
>>> How is this suppose to work ?
>>
>> This is why you should use tpacket layout v2 or v3, rather than v1,
>> they fix these issues.
>
> Norbert, please also read Documentation/networking/packet_mmap.txt
>
> Thanks !


Ah... thanks guys, now I see. struct tpacket2_hdr has fixed layout. I will use v2 (or v3).

I could not find this info in [linux-3.6.8/]Documentation/networking/packet_mmap.txt
though.

---
Norbert.

single process receives own frames due to PACKET_MMAP

[Norbert van Bolhuis]

Our application uses raw AF_PACKET socket to send and receive
on one particular ethernet interface.

Recently we started using PACKET_MMAP (TPACKET_V2). This makes
the Appl use a TX socket and a RX socket.
Both sockets are bound to the same (eth) interface. I noticed
the RX socket receives all frames that are sent via the
TX socket (same process, different thread). This I do not want.

I know it is supposed to happen for different processes
(otherwise wireshark won't work), but I did not expect it to
happen for one single process (with different threads).

I can filter them out in user-space (PACKET_OUTGOING)
or via kernel packet filter (SO_ATTACH_FILTER), but performance is
critical.

I wonder whether this (PACKET_MMAP) behaviour is OK.

It did not happen before (with a non-PACKET_MMAP AF_PACKET socket
which was used by both threads of the same Appl process). So
why is it happening now ?
I'd say it makes no sense to make the same process receive its
own transmitted frames on that same interface (unless its lo).

If I'm not doing something wrong, this means this behaviour
causes my CPU to be loaded much more (since all transmitted frames
have to be filtered out).

Let me know what you think.

thanks,
Norbert van Bolhuis

Re: single process receives own frames due to PACKET_MMAP

[Daniel Borkmann]

On 01/06/2014 11:58 PM, Norbert van Bolhuis wrote:
>
> Our application uses raw AF_PACKET socket to send and receive
> on one particular ethernet interface.
>
> Recently we started using PACKET_MMAP (TPACKET_V2). This makes
> the Appl use a TX socket and a RX socket.
> Both sockets are bound to the same (eth) interface. I noticed
> the RX socket receives all frames that are sent via the
> TX socket (same process, different thread). This I do not want.
>
> I know it is supposed to happen for different processes
> (otherwise wireshark won't work), but I did not expect it to
> happen for one single process (with different threads).
>
> I can filter them out in user-space (PACKET_OUTGOING)
> or via kernel packet filter (SO_ATTACH_FILTER), but performance is
> critical.
>
> I wonder whether this (PACKET_MMAP) behaviour is OK.

For your use-case, we recently introduced in d346a3fae3ff1
("packet: introduce PACKET_QDISC_BYPASS socket option") a
bypass of dev_queue_xmit() (that internally invokes
dev_queue_xmit_nit()).

> It did not happen before (with a non-PACKET_MMAP AF_PACKET socket
> which was used by both threads of the same Appl process). So
> why is it happening now ?

Can you elaborate a bit on which kernel versions that behaviour
changed?

> I'd say it makes no sense to make the same process receive its
> own transmitted frames on that same interface (unless its lo).
>
> If I'm not doing something wrong, this means this behaviour
> causes my CPU to be loaded much more (since all transmitted frames
> have to be filtered out).
>
> Let me know what you think.
>
> thanks,
> Norbert van Bolhuis
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: single process receives own frames due to PACKET_MMAP

[Jesper Dangaard Brouer]

On Tue, 07 Jan 2014 10:32:01 +0100
Daniel Borkmann <dborkman@redhat.com> wrote:

> On 01/06/2014 11:58 PM, Norbert van Bolhuis wrote:
> >
> > Our application uses raw AF_PACKET socket to send and receive
> > on one particular ethernet interface.
> >
> > Recently we started using PACKET_MMAP (TPACKET_V2). This makes
> > the Appl use a TX socket and a RX socket.
> > Both sockets are bound to the same (eth) interface. I noticed
> > the RX socket receives all frames that are sent via the
> > TX socket (same process, different thread). This I do not want.
> >
> > I know it is supposed to happen for different processes
> > (otherwise wireshark won't work), but I did not expect it to
> > happen for one single process (with different threads).
> >
> > I can filter them out in user-space (PACKET_OUTGOING)
> > or via kernel packet filter (SO_ATTACH_FILTER), but performance is
> > critical.
> >
> > I wonder whether this (PACKET_MMAP) behaviour is OK.
> 
> For your use-case, we recently introduced in d346a3fae3ff1
> ("packet: introduce PACKET_QDISC_BYPASS socket option") a
> bypass of dev_queue_xmit() (that internally invokes
> dev_queue_xmit_nit()).
> 
> > It did not happen before (with a non-PACKET_MMAP AF_PACKET socket
> > which was used by both threads of the same Appl process). So
> > why is it happening now ?
> 
> Can you elaborate a bit on which kernel versions that behaviour
> changed?
> 
> > I'd say it makes no sense to make the same process receive its
> > own transmitted frames on that same interface (unless its lo).

Have you setup:
 ring->s_ll.sll_protocol = 0

This is what I did in trafgen to avoid this problem.

See line 55 in netsniff-ng/ring.c:
 https://github.com/borkmann/netsniff-ng/blob/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1/ring.c#L55

Commit:
 https://github.com/borkmann/netsniff-ng/commit/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1

> > If I'm not doing something wrong, this means this behaviour
> > causes my CPU to be loaded much more (since all transmitted frames
> > have to be filtered out).
> >
> > Let me know what you think.
> >
> > thanks,
> > Norbert van Bolhuis

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

Re: single process receives own frames due to PACKET_MMAP

[Norbert van Bolhuis]

On 01/07/14 11:06, Jesper Dangaard Brouer wrote:
> On Tue, 07 Jan 2014 10:32:01 +0100
> Daniel Borkmann<dborkman@redhat.com>  wrote:
>
>> On 01/06/2014 11:58 PM, Norbert van Bolhuis wrote:
>>>
>>> Our application uses raw AF_PACKET socket to send and receive
>>> on one particular ethernet interface.
>>>
>>> Recently we started using PACKET_MMAP (TPACKET_V2). This makes
>>> the Appl use a TX socket and a RX socket.
>>> Both sockets are bound to the same (eth) interface. I noticed
>>> the RX socket receives all frames that are sent via the
>>> TX socket (same process, different thread). This I do not want.
>>>
>>> I know it is supposed to happen for different processes
>>> (otherwise wireshark won't work), but I did not expect it to
>>> happen for one single process (with different threads).
>>>
>>> I can filter them out in user-space (PACKET_OUTGOING)
>>> or via kernel packet filter (SO_ATTACH_FILTER), but performance is
>>> critical.
>>>
>>> I wonder whether this (PACKET_MMAP) behaviour is OK.
>>
>> For your use-case, we recently introduced in d346a3fae3ff1
>> ("packet: introduce PACKET_QDISC_BYPASS socket option") a
>> bypass of dev_queue_xmit() (that internally invokes
>> dev_queue_xmit_nit()).
>>
>>> It did not happen before (with a non-PACKET_MMAP AF_PACKET socket
>>> which was used by both threads of the same Appl process). So
>>> why is it happening now ?
>>
>> Can you elaborate a bit on which kernel versions that behaviour
>> changed?


Sorry, I wasn't very clear. This does not regard kernel (versions).
With "before" I mean our previous version of our application which
used an AF_PACKET socket, but not the PACKET_MMAP option.


>>
>>> I'd say it makes no sense to make the same process receive its
>>> own transmitted frames on that same interface (unless its lo).
>
> Have you setup:
>   ring->s_ll.sll_protocol = 0
>
> This is what I did in trafgen to avoid this problem.
>
> See line 55 in netsniff-ng/ring.c:
>   https://github.com/borkmann/netsniff-ng/blob/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1/ring.c#L55
>
> Commit:
>   https://github.com/borkmann/netsniff-ng/commit/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1
>


No I did not do that, I was checking my code against netsniff-ng-0.5.8-rc4.

But I just tried it, I believe I do the same as netsniff-ng-0.5.8-rc5, but it doesn't
work for me. Maybe because I have an old FC14 system (kernel 2.6.35.14-106.fc14.x86_64).

So I tried to see whether netsniff-ng-0.5.8-rc5/trafgen still makes the
kernel call packet_rcv() on my FC14 system. So I build and run it, but I'm not sure
how to (easily) check that. In anyway, Wireshark does capture the trafgen generated
frames, does that say anything ?

In the future, I can at least use PACKET_QDISC_BYPASS as a "workaround".

Thanks a lot for your answers.

---
Norbert

Re: single process receives own frames due to PACKET_MMAP

[Daniel Borkmann]

On 01/07/2014 02:16 PM, Norbert van Bolhuis wrote:
> On 01/07/14 11:06, Jesper Dangaard Brouer wrote:
>> On Tue, 07 Jan 2014 10:32:01 +0100
>> Daniel Borkmann<dborkman@redhat.com>  wrote:
>>> On 01/06/2014 11:58 PM, Norbert van Bolhuis wrote:
>>>>
>>>> Our application uses raw AF_PACKET socket to send and receive
>>>> on one particular ethernet interface.
>>>>
>>>> Recently we started using PACKET_MMAP (TPACKET_V2). This makes
>>>> the Appl use a TX socket and a RX socket.
>>>> Both sockets are bound to the same (eth) interface. I noticed
>>>> the RX socket receives all frames that are sent via the
>>>> TX socket (same process, different thread). This I do not want.
>>>>
>>>> I know it is supposed to happen for different processes
>>>> (otherwise wireshark won't work), but I did not expect it to
>>>> happen for one single process (with different threads).
>>>>
>>>> I can filter them out in user-space (PACKET_OUTGOING)
>>>> or via kernel packet filter (SO_ATTACH_FILTER), but performance is
>>>> critical.
>>>>
>>>> I wonder whether this (PACKET_MMAP) behaviour is OK.
>>>
>>> For your use-case, we recently introduced in d346a3fae3ff1
>>> ("packet: introduce PACKET_QDISC_BYPASS socket option") a
>>> bypass of dev_queue_xmit() (that internally invokes
>>> dev_queue_xmit_nit()).
>>>
>>>> It did not happen before (with a non-PACKET_MMAP AF_PACKET socket
>>>> which was used by both threads of the same Appl process). So
>>>> why is it happening now ?
>>>
>>> Can you elaborate a bit on which kernel versions that behaviour
>>> changed?
>
>
> Sorry, I wasn't very clear. This does not regard kernel (versions).
> With "before" I mean our previous version of our application which
> used an AF_PACKET socket, but not the PACKET_MMAP option.

Hmm, taking a look into 2.6.35.14, it doesn't seem to be so much of
a difference.

W/o seeing how you've invoked mmap vs non-mmap API, it's hard to say.
Could you try with latest upstream resp. net-next kernel?

>>>> I'd say it makes no sense to make the same process receive its
>>>> own transmitted frames on that same interface (unless its lo).
>>
>> Have you setup:
>>   ring->s_ll.sll_protocol = 0
>>
>> This is what I did in trafgen to avoid this problem.
>>
>> See line 55 in netsniff-ng/ring.c:
>>   https://github.com/borkmann/netsniff-ng/blob/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1/ring.c#L55
>>
>> Commit:
>>   https://github.com/borkmann/netsniff-ng/commit/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1
>>
>
>
> No I did not do that, I was checking my code against netsniff-ng-0.5.8-rc4.
>
> But I just tried it, I believe I do the same as netsniff-ng-0.5.8-rc5, but it doesn't
> work for me. Maybe because I have an old FC14 system (kernel 2.6.35.14-106.fc14.x86_64).
>
> So I tried to see whether netsniff-ng-0.5.8-rc5/trafgen still makes the
> kernel call packet_rcv() on my FC14 system. So I build and run it, but I'm not sure
> how to (easily) check that. In anyway, Wireshark does capture the trafgen generated
> frames, does that say anything ?

Yep, kernel is definitely too old for that.

> In the future, I can at least use PACKET_QDISC_BYPASS as a "workaround".
>
> Thanks a lot for your answers.
>
> ---
> Norbert
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: single process receives own frames due to PACKET_MMAP

[Jesper Dangaard Brouer]

On Tue, 07 Jan 2014 14:16:03 +0100
Norbert van Bolhuis <nvbolhuis@aimvalley.nl> wrote:
> On 01/07/14 11:06, Jesper Dangaard Brouer wrote:
> > On Tue, 07 Jan 2014 10:32:01 +0100
> > Daniel Borkmann<dborkman@redhat.com>  wrote:
> >
> >> On 01/06/2014 11:58 PM, Norbert van Bolhuis wrote:
> >>>
[...]
> >>
> >>> I'd say it makes no sense to make the same process receive its
> >>> own transmitted frames on that same interface (unless its lo).
> >
> > Have you setup:
> >   ring->s_ll.sll_protocol = 0
> >
> > This is what I did in trafgen to avoid this problem.
> >
> > See line 55 in netsniff-ng/ring.c:
> >   https://github.com/borkmann/netsniff-ng/blob/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1/ring.c#L55
> >
> > Commit:
> >   https://github.com/borkmann/netsniff-ng/commit/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1
> >
> 
> 
> No I did not do that, I was checking my code against netsniff-ng-0.5.8-rc4.
> 
> But I just tried it, I believe I do the same as netsniff-ng-0.5.8-rc5, but it doesn't
> work for me. Maybe because I have an old FC14 system (kernel 2.6.35.14-106.fc14.x86_64).
> 
> So I tried to see whether netsniff-ng-0.5.8-rc5/trafgen still makes the
> kernel call packet_rcv() on my FC14 system. So I build and run it, but I'm not sure
> how to (easily) check that.

The easiest way is to:
  cat /proc/net/ptype
And look if someone registered a proto handler/function: packet_rcv (or tpacket_rcv).

The more exact method is, to run "perf record -a -g" and then look (at
the result with "perf report") for a lock contention, and "expand" the
spin_lock and see if packet_rcv() is calling this spin lock.


> In anyway, Wireshark does capture the trafgen generated
> frames, does that say anything ?

Be careful not to start a wireshark/tcpdump, at the sametime, as this
will slow you down.
 
> In the future, I can at least use PACKET_QDISC_BYPASS as a "workaround".

And in the future with PACKET_QDISC_BYPASS, your wireshark will not
catch these packets, remember that.

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

Re: single process receives own frames due to PACKET_MMAP

[Norbert van Bolhuis]

On 01/07/14 15:09, Jesper Dangaard Brouer wrote:
>
> On Tue, 07 Jan 2014 14:16:03 +0100
> Norbert van Bolhuis<nvbolhuis@aimvalley.nl>  wrote:
>> On 01/07/14 11:06, Jesper Dangaard Brouer wrote:
>>> On Tue, 07 Jan 2014 10:32:01 +0100
>>> Daniel Borkmann<dborkman@redhat.com>   wrote:
>>>
>>>> On 01/06/2014 11:58 PM, Norbert van Bolhuis wrote:
>>>>>
> [...]
>>>>
>>>>> I'd say it makes no sense to make the same process receive its
>>>>> own transmitted frames on that same interface (unless its lo).
>>>
>>> Have you setup:
>>>    ring->s_ll.sll_protocol = 0
>>>
>>> This is what I did in trafgen to avoid this problem.
>>>
>>> See line 55 in netsniff-ng/ring.c:
>>>    https://github.com/borkmann/netsniff-ng/blob/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1/ring.c#L55
>>>
>>> Commit:
>>>    https://github.com/borkmann/netsniff-ng/commit/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1
>>>
>>
>>
>> No I did not do that, I was checking my code against netsniff-ng-0.5.8-rc4.
>>
>> But I just tried it, I believe I do the same as netsniff-ng-0.5.8-rc5, but it doesn't
>> work for me. Maybe because I have an old FC14 system (kernel 2.6.35.14-106.fc14.x86_64).
>>
>> So I tried to see whether netsniff-ng-0.5.8-rc5/trafgen still makes the
>> kernel call packet_rcv() on my FC14 system. So I build and run it, but I'm not sure
>> how to (easily) check that.
>
> The easiest way is to:
>    cat /proc/net/ptype
> And look if someone registered a proto handler/function: packet_rcv (or tpacket_rcv).
>
> The more exact method is, to run "perf record -a -g" and then look (at
> the result with "perf report") for a lock contention, and "expand" the
> spin_lock and see if packet_rcv() is calling this spin lock.
>


I checked the easy way.
Even on my old FC14 system the "protocol=0 patch" seems to make a difference
for trafgen.
Without the patch I see for each CPU in use by trafgen a "packet_rcv entry" in
/proc/net/ptype.
With the patch I see no additional "packet_rcv entry".

It could be my Appl is wrong or maybe the "protocol=0 patch" does not help.
I think the latter, afterall my Appl has, unlike trafgen, another RX
(AF_PACKET) socket.


>
>> In anyway, Wireshark does capture the trafgen generated
>> frames, does that say anything ?
>
> Be careful not to start a wireshark/tcpdump, at the sametime, as this
> will slow you down.
>
>> In the future, I can at least use PACKET_QDISC_BYPASS as a "workaround".
>
> And in the future with PACKET_QDISC_BYPASS, your wireshark will not
> catch these packets, remember that.
>


Yes, this is why I would love to see the "protocol=0 patch" work for my Appl.

So I will try my Appl with the latest net-next kernel to see if that makes
it work. Hopefully I can find some time in the next coming days, I will keep
you informed.

---
Norbert

Re: single process receives own frames due to PACKET_MMAP

[Daniel Borkmann]

On 01/07/2014 04:16 PM, Norbert van Bolhuis wrote:
> On 01/07/14 15:09, Jesper Dangaard Brouer wrote:
>> On Tue, 07 Jan 2014 14:16:03 +0100
>> Norbert van Bolhuis<nvbolhuis@aimvalley.nl>  wrote:
>>> On 01/07/14 11:06, Jesper Dangaard Brouer wrote:
>>>> On Tue, 07 Jan 2014 10:32:01 +0100
>>>> Daniel Borkmann<dborkman@redhat.com>   wrote:
>>>>
>>>>> On 01/06/2014 11:58 PM, Norbert van Bolhuis wrote:
>>>>>>
>> [...]
>>>>>
>>>>>> I'd say it makes no sense to make the same process receive its
>>>>>> own transmitted frames on that same interface (unless its lo).
>>>>
>>>> Have you setup:
>>>>    ring->s_ll.sll_protocol = 0
>>>>
>>>> This is what I did in trafgen to avoid this problem.
>>>>
>>>> See line 55 in netsniff-ng/ring.c:
>>>>    https://github.com/borkmann/netsniff-ng/blob/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1/ring.c#L55
>>>>
>>>> Commit:
>>>>    https://github.com/borkmann/netsniff-ng/commit/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1
>>>>
>>>
>>>
>>> No I did not do that, I was checking my code against netsniff-ng-0.5.8-rc4.
>>>
>>> But I just tried it, I believe I do the same as netsniff-ng-0.5.8-rc5, but it doesn't
>>> work for me. Maybe because I have an old FC14 system (kernel 2.6.35.14-106.fc14.x86_64).
>>>
>>> So I tried to see whether netsniff-ng-0.5.8-rc5/trafgen still makes the
>>> kernel call packet_rcv() on my FC14 system. So I build and run it, but I'm not sure
>>> how to (easily) check that.
>>
>> The easiest way is to:
>>    cat /proc/net/ptype
>> And look if someone registered a proto handler/function: packet_rcv (or tpacket_rcv).
>>
>> The more exact method is, to run "perf record -a -g" and then look (at
>> the result with "perf report") for a lock contention, and "expand" the
>> spin_lock and see if packet_rcv() is calling this spin lock.
>>
>
>
> I checked the easy way.
> Even on my old FC14 system the "protocol=0 patch" seems to make a difference
> for trafgen.
> Without the patch I see for each CPU in use by trafgen a "packet_rcv entry" in
> /proc/net/ptype.
> With the patch I see no additional "packet_rcv entry".

Yes, that is expected behaviour. ;-) See more below.

> It could be my Appl is wrong or maybe the "protocol=0 patch" does not help.
> I think the latter, afterall my Appl has, unlike trafgen, another RX
> (AF_PACKET) socket.
>
>
>>
>>> In anyway, Wireshark does capture the trafgen generated
>>> frames, does that say anything ?
>>
>> Be careful not to start a wireshark/tcpdump, at the sametime, as this
>> will slow you down.
>>
>>> In the future, I can at least use PACKET_QDISC_BYPASS as a "workaround".
>>
>> And in the future with PACKET_QDISC_BYPASS, your wireshark will not
>> catch these packets, remember that.
>>
>
>
> Yes, this is why I would love to see the "protocol=0 patch" work for my Appl.
>
> So I will try my Appl with the latest net-next kernel to see if that makes
> it work. Hopefully I can find some time in the next coming days, I will keep
> you informed.

As long as there's at least one single PF_PACKET receive socket open and you
do not make use of PACKET_QDISC_BYPASS on your tx socket, then those packets go
back the dev_queue_xmit_nit() path, even if your tx socket uses protocol=0.

If you make use of PACKET_QDISC_BYPASS [1] for your particular tx socket, then
packets generated by that socket will not hit the dev_queue_xmit_nit() path
back to other possible rx listeners that are present on your system (w/ the
side-effects for tx as described in [1]).

   [1] Documentation/networking/packet_mmap.txt +960

> ---
> Norbert

Re: single process receives own frames due to PACKET_MMAP

[Norbert van Bolhuis]

On 01/07/14 16:26, Daniel Borkmann wrote:
> On 01/07/2014 04:16 PM, Norbert van Bolhuis wrote:
>> On 01/07/14 15:09, Jesper Dangaard Brouer wrote:
>>> On Tue, 07 Jan 2014 14:16:03 +0100
>>> Norbert van Bolhuis<nvbolhuis@aimvalley.nl> wrote:
>>>> On 01/07/14 11:06, Jesper Dangaard Brouer wrote:
>>>>> On Tue, 07 Jan 2014 10:32:01 +0100
>>>>> Daniel Borkmann<dborkman@redhat.com> wrote:
>>>>>
>>>>>> On 01/06/2014 11:58 PM, Norbert van Bolhuis wrote:
>>>>>>>
>>> [...]
>>>>>>
>>>>>>> I'd say it makes no sense to make the same process receive its
>>>>>>> own transmitted frames on that same interface (unless its lo).
>>>>>
>>>>> Have you setup:
>>>>> ring->s_ll.sll_protocol = 0
>>>>>
>>>>> This is what I did in trafgen to avoid this problem.
>>>>>
>>>>> See line 55 in netsniff-ng/ring.c:
>>>>> https://github.com/borkmann/netsniff-ng/blob/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1/ring.c#L55
>>>>>
>>>>> Commit:
>>>>> https://github.com/borkmann/netsniff-ng/commit/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1
>>>>>
>>>>
>>>>
>>>> No I did not do that, I was checking my code against netsniff-ng-0.5.8-rc4.
>>>>
>>>> But I just tried it, I believe I do the same as netsniff-ng-0.5.8-rc5, but it doesn't
>>>> work for me. Maybe because I have an old FC14 system (kernel 2.6.35.14-106.fc14.x86_64).
>>>>
>>>> So I tried to see whether netsniff-ng-0.5.8-rc5/trafgen still makes the
>>>> kernel call packet_rcv() on my FC14 system. So I build and run it, but I'm not sure
>>>> how to (easily) check that.
>>>
>>> The easiest way is to:
>>> cat /proc/net/ptype
>>> And look if someone registered a proto handler/function: packet_rcv (or tpacket_rcv).
>>>
>>> The more exact method is, to run "perf record -a -g" and then look (at
>>> the result with "perf report") for a lock contention, and "expand" the
>>> spin_lock and see if packet_rcv() is calling this spin lock.
>>>
>>
>>
>> I checked the easy way.
>> Even on my old FC14 system the "protocol=0 patch" seems to make a difference
>> for trafgen.
>> Without the patch I see for each CPU in use by trafgen a "packet_rcv entry" in
>> /proc/net/ptype.
>> With the patch I see no additional "packet_rcv entry".
>
> Yes, that is expected behaviour. ;-) See more below.
>
>> It could be my Appl is wrong or maybe the "protocol=0 patch" does not help.
>> I think the latter, afterall my Appl has, unlike trafgen, another RX
>> (AF_PACKET) socket.
>>
>>
>>>
>>>> In anyway, Wireshark does capture the trafgen generated
>>>> frames, does that say anything ?
>>>
>>> Be careful not to start a wireshark/tcpdump, at the sametime, as this
>>> will slow you down.
>>>
>>>> In the future, I can at least use PACKET_QDISC_BYPASS as a "workaround".
>>>
>>> And in the future with PACKET_QDISC_BYPASS, your wireshark will not
>>> catch these packets, remember that.
>>>
>>
>>
>> Yes, this is why I would love to see the "protocol=0 patch" work for my Appl.
>>
>> So I will try my Appl with the latest net-next kernel to see if that makes
>> it work. Hopefully I can find some time in the next coming days, I will keep
>> you informed.
>
> As long as there's at least one single PF_PACKET receive socket open and you
> do not make use of PACKET_QDISC_BYPASS on your tx socket, then those packets go
> back the dev_queue_xmit_nit() path, even if your tx socket uses protocol=0.
>
> If you make use of PACKET_QDISC_BYPASS [1] for your particular tx socket, then
> packets generated by that socket will not hit the dev_queue_xmit_nit() path
> back to other possible rx listeners that are present on your system (w/ the
> side-effects for tx as described in [1]).
>
> [1] Documentation/networking/packet_mmap.txt +960
>


Ok, that's clear.

But this means my PF_PACKET socket application performs worse because of
using PACKET_MMAP. I expected the opposit.

Afterall my old PF_PACKET socket application (which does not use PACKET_MMAP)
uses only one PF_PACKET socket (for TX and RX). Because packets are never sent
back to the socket they originated from, my old PF_PACKET socket application
performs better.

Is there a way to use one PF_PACKET socket for both TX and RX and use PACKET_MMAP ?

---
Norbert

Re: single process receives own frames due to PACKET_MMAP

[Daniel Borkmann]

On 01/07/2014 04:46 PM, Norbert van Bolhuis wrote:
> On 01/07/14 16:26, Daniel Borkmann wrote:
>> On 01/07/2014 04:16 PM, Norbert van Bolhuis wrote:
>>> On 01/07/14 15:09, Jesper Dangaard Brouer wrote:
>>>> On Tue, 07 Jan 2014 14:16:03 +0100
>>>> Norbert van Bolhuis<nvbolhuis@aimvalley.nl> wrote:
>>>>> On 01/07/14 11:06, Jesper Dangaard Brouer wrote:
>>>>>> On Tue, 07 Jan 2014 10:32:01 +0100
>>>>>> Daniel Borkmann<dborkman@redhat.com> wrote:
>>>>>>
>>>>>>> On 01/06/2014 11:58 PM, Norbert van Bolhuis wrote:
>>>>>>>>
>>>> [...]
>>>>>>>
>>>>>>>> I'd say it makes no sense to make the same process receive its
>>>>>>>> own transmitted frames on that same interface (unless its lo).
>>>>>>
>>>>>> Have you setup:
>>>>>> ring->s_ll.sll_protocol = 0
>>>>>>
>>>>>> This is what I did in trafgen to avoid this problem.
>>>>>>
>>>>>> See line 55 in netsniff-ng/ring.c:
>>>>>> https://github.com/borkmann/netsniff-ng/blob/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1/ring.c#L55
>>>>>>
>>>>>> Commit:
>>>>>> https://github.com/borkmann/netsniff-ng/commit/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1
>>>>>>
>>>>>
>>>>>
>>>>> No I did not do that, I was checking my code against netsniff-ng-0.5.8-rc4.
>>>>>
>>>>> But I just tried it, I believe I do the same as netsniff-ng-0.5.8-rc5, but it doesn't
>>>>> work for me. Maybe because I have an old FC14 system (kernel 2.6.35.14-106.fc14.x86_64).
>>>>>
>>>>> So I tried to see whether netsniff-ng-0.5.8-rc5/trafgen still makes the
>>>>> kernel call packet_rcv() on my FC14 system. So I build and run it, but I'm not sure
>>>>> how to (easily) check that.
>>>>
>>>> The easiest way is to:
>>>> cat /proc/net/ptype
>>>> And look if someone registered a proto handler/function: packet_rcv (or tpacket_rcv).
>>>>
>>>> The more exact method is, to run "perf record -a -g" and then look (at
>>>> the result with "perf report") for a lock contention, and "expand" the
>>>> spin_lock and see if packet_rcv() is calling this spin lock.
>>>>
>>>
>>>
>>> I checked the easy way.
>>> Even on my old FC14 system the "protocol=0 patch" seems to make a difference
>>> for trafgen.
>>> Without the patch I see for each CPU in use by trafgen a "packet_rcv entry" in
>>> /proc/net/ptype.
>>> With the patch I see no additional "packet_rcv entry".
>>
>> Yes, that is expected behaviour. ;-) See more below.
>>
>>> It could be my Appl is wrong or maybe the "protocol=0 patch" does not help.
>>> I think the latter, afterall my Appl has, unlike trafgen, another RX
>>> (AF_PACKET) socket.
>>>
>>>
>>>>
>>>>> In anyway, Wireshark does capture the trafgen generated
>>>>> frames, does that say anything ?
>>>>
>>>> Be careful not to start a wireshark/tcpdump, at the sametime, as this
>>>> will slow you down.
>>>>
>>>>> In the future, I can at least use PACKET_QDISC_BYPASS as a "workaround".
>>>>
>>>> And in the future with PACKET_QDISC_BYPASS, your wireshark will not
>>>> catch these packets, remember that.
>>>>
>>>
>>>
>>> Yes, this is why I would love to see the "protocol=0 patch" work for my Appl.
>>>
>>> So I will try my Appl with the latest net-next kernel to see if that makes
>>> it work. Hopefully I can find some time in the next coming days, I will keep
>>> you informed.
>>
>> As long as there's at least one single PF_PACKET receive socket open and you
>> do not make use of PACKET_QDISC_BYPASS on your tx socket, then those packets go
>> back the dev_queue_xmit_nit() path, even if your tx socket uses protocol=0.
>>
>> If you make use of PACKET_QDISC_BYPASS [1] for your particular tx socket, then
>> packets generated by that socket will not hit the dev_queue_xmit_nit() path
>> back to other possible rx listeners that are present on your system (w/ the
>> side-effects for tx as described in [1]).
>>
>> [1] Documentation/networking/packet_mmap.txt +960
>>
>
>
> Ok, that's clear.
>
> But this means my PF_PACKET socket application performs worse because of
> using PACKET_MMAP. I expected the opposit.
>
> Afterall my old PF_PACKET socket application (which does not use PACKET_MMAP)
> uses only one PF_PACKET socket (for TX and RX). Because packets are never sent
> back to the socket they originated from, my old PF_PACKET socket application
> performs better.
>
> Is there a way to use one PF_PACKET socket for both TX and RX and use PACKET_MMAP ?

Yep:

http://thread.gmane.org/gmane.linux.network/269129/focus=269188

Feel free to make a patch and add this to Documentation/networking/packet_mmap.txt
I think could be useful for others as well.

Re: single process receives own frames due to PACKET_MMAP

[Norbert van Bolhuis]

On 01/07/14 16:57, Daniel Borkmann wrote:
> On 01/07/2014 04:46 PM, Norbert van Bolhuis wrote:
>> On 01/07/14 16:26, Daniel Borkmann wrote:
>>> On 01/07/2014 04:16 PM, Norbert van Bolhuis wrote:
>>>> On 01/07/14 15:09, Jesper Dangaard Brouer wrote:
>>>>> On Tue, 07 Jan 2014 14:16:03 +0100
>>>>> Norbert van Bolhuis<nvbolhuis@aimvalley.nl> wrote:
>>>>>> On 01/07/14 11:06, Jesper Dangaard Brouer wrote:
>>>>>>> On Tue, 07 Jan 2014 10:32:01 +0100
>>>>>>> Daniel Borkmann<dborkman@redhat.com> wrote:
>>>>>>>
>>>>>>>> On 01/06/2014 11:58 PM, Norbert van Bolhuis wrote:
>>>>>>>>>
>>>>> [...]
>>>>>>>>
>>>>>>>>> I'd say it makes no sense to make the same process receive its
>>>>>>>>> own transmitted frames on that same interface (unless its lo).
>>>>>>>
>>>>>>> Have you setup:
>>>>>>> ring->s_ll.sll_protocol = 0
>>>>>>>
>>>>>>> This is what I did in trafgen to avoid this problem.
>>>>>>>
>>>>>>> See line 55 in netsniff-ng/ring.c:
>>>>>>> https://github.com/borkmann/netsniff-ng/blob/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1/ring.c#L55
>>>>>>>
>>>>>>> Commit:
>>>>>>> https://github.com/borkmann/netsniff-ng/commit/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1
>>>>>>>
>>>>>>
>>>>>>
>>>>>> No I did not do that, I was checking my code against netsniff-ng-0.5.8-rc4.
>>>>>>
>>>>>> But I just tried it, I believe I do the same as netsniff-ng-0.5.8-rc5, but it doesn't
>>>>>> work for me. Maybe because I have an old FC14 system (kernel 2.6.35.14-106.fc14.x86_64).
>>>>>>
>>>>>> So I tried to see whether netsniff-ng-0.5.8-rc5/trafgen still makes the
>>>>>> kernel call packet_rcv() on my FC14 system. So I build and run it, but I'm not sure
>>>>>> how to (easily) check that.
>>>>>
>>>>> The easiest way is to:
>>>>> cat /proc/net/ptype
>>>>> And look if someone registered a proto handler/function: packet_rcv (or tpacket_rcv).
>>>>>
>>>>> The more exact method is, to run "perf record -a -g" and then look (at
>>>>> the result with "perf report") for a lock contention, and "expand" the
>>>>> spin_lock and see if packet_rcv() is calling this spin lock.
>>>>>
>>>>
>>>>
>>>> I checked the easy way.
>>>> Even on my old FC14 system the "protocol=0 patch" seems to make a difference
>>>> for trafgen.
>>>> Without the patch I see for each CPU in use by trafgen a "packet_rcv entry" in
>>>> /proc/net/ptype.
>>>> With the patch I see no additional "packet_rcv entry".
>>>
>>> Yes, that is expected behaviour. ;-) See more below.
>>>
>>>> It could be my Appl is wrong or maybe the "protocol=0 patch" does not help.
>>>> I think the latter, afterall my Appl has, unlike trafgen, another RX
>>>> (AF_PACKET) socket.
>>>>
>>>>
>>>>>
>>>>>> In anyway, Wireshark does capture the trafgen generated
>>>>>> frames, does that say anything ?
>>>>>
>>>>> Be careful not to start a wireshark/tcpdump, at the sametime, as this
>>>>> will slow you down.
>>>>>
>>>>>> In the future, I can at least use PACKET_QDISC_BYPASS as a "workaround".
>>>>>
>>>>> And in the future with PACKET_QDISC_BYPASS, your wireshark will not
>>>>> catch these packets, remember that.
>>>>>
>>>>
>>>>
>>>> Yes, this is why I would love to see the "protocol=0 patch" work for my Appl.
>>>>
>>>> So I will try my Appl with the latest net-next kernel to see if that makes
>>>> it work. Hopefully I can find some time in the next coming days, I will keep
>>>> you informed.
>>>
>>> As long as there's at least one single PF_PACKET receive socket open and you
>>> do not make use of PACKET_QDISC_BYPASS on your tx socket, then those packets go
>>> back the dev_queue_xmit_nit() path, even if your tx socket uses protocol=0.
>>>
>>> If you make use of PACKET_QDISC_BYPASS [1] for your particular tx socket, then
>>> packets generated by that socket will not hit the dev_queue_xmit_nit() path
>>> back to other possible rx listeners that are present on your system (w/ the
>>> side-effects for tx as described in [1]).
>>>
>>> [1] Documentation/networking/packet_mmap.txt +960
>>>
>>
>>
>> Ok, that's clear.
>>
>> But this means my PF_PACKET socket application performs worse because of
>> using PACKET_MMAP. I expected the opposit.
>>
>> Afterall my old PF_PACKET socket application (which does not use PACKET_MMAP)
>> uses only one PF_PACKET socket (for TX and RX). Because packets are never sent
>> back to the socket they originated from, my old PF_PACKET socket application
>> performs better.
>>
>> Is there a way to use one PF_PACKET socket for both TX and RX and use PACKET_MMAP ?
>
> Yep:
>
> http://thread.gmane.org/gmane.linux.network/269129/focus=269188
>
> Feel free to make a patch and add this to Documentation/networking/packet_mmap.txt
> I think could be useful for others as well.


Good, it all works fine now, though performance is still not as good as I'd hoped.

I will sent a doc patch soon.

thanks for all help!

---
Norbert.

Re: single process receives own frames due to PACKET_MMAP

[Daniel Borkmann]

On 01/08/2014 03:18 PM, Norbert van Bolhuis wrote:

>>> Is there a way to use one PF_PACKET socket for both TX and RX and use PACKET_MMAP ?
>>
>> Yep:
>>
>> http://thread.gmane.org/gmane.linux.network/269129/focus=269188
>>
>> Feel free to make a patch and add this to Documentation/networking/packet_mmap.txt
>> I think could be useful for others as well.
>
>
> Good, it all works fine now, though performance is still not as good as I'd hoped.

Does your use case allow for TPACKET_V3 or for using fanout ?

> I will sent a doc patch soon.

Sounds great, thanks!

> thanks for all help!

Re: single process receives own frames due to PACKET_MMAP

[Norbert van Bolhuis]

On 01/08/14 15:24, Daniel Borkmann wrote:
> On 01/08/2014 03:18 PM, Norbert van Bolhuis wrote:
>
>>>> Is there a way to use one PF_PACKET socket for both TX and RX and use PACKET_MMAP ?
>>>
>>> Yep:
>>>
>>> http://thread.gmane.org/gmane.linux.network/269129/focus=269188
>>>
>>> Feel free to make a patch and add this to Documentation/networking/packet_mmap.txt
>>> I think could be useful for others as well.
>>
>>
>> Good, it all works fine now, though performance is still not as good as I'd hoped.
>
> Does your use case allow for TPACKET_V3 or for using fanout ?
>

no, unfortunately not.

Though currently our performance is limited by application/system stuff, so
at the moment it doesn't really matter.

--
Norbert