kenel level packet capturing

kenel level packet capturing

[raviraj joshi]

We are making a kernel level packet capture module as a part of our BE project .
We are finding it difficult to understand kenel networking code.
We studied structures and functions like sk_buff, netif_rx(),dev_alloc_skb.
How ever we were not able to trace the actual path a simple packet follows.
Moreover we tried to count the no of times netif_rx function was
called it was much much lower than amount of packets actually being
received.
Can anyone please let us know the reason for this and how to approach
the problem of writing a kernel level packet capture module.

regards,
Raviraj Joshi
PICT,Pune.

Re: kenel level packet capturing

[Bruno Prémont]

Hi,

You would probably have more/better replies if you asked netdev <netdev@vger.kernel.org>.

On Tue, 18 October 2011 raviraj joshi <raviraj.j1991@gmail.com> wrote:
> We are making a kernel level packet capture module as a part of our BE project .
> We are finding it difficult to understand kenel networking code.
> We studied structures and functions like sk_buff, netif_rx(),dev_alloc_skb.
> How ever we were not able to trace the actual path a simple packet follows.
> Moreover we tried to count the no of times netif_rx function was
> called it was much much lower than amount of packets actually being
> received.

netif_rx may fetch more than a single packet from the network interface
controller in order to improve performance.

> Can anyone please let us know the reason for this and how to approach
> the problem of writing a kernel level packet capture module.

The better place to capture packets is probably by hooking somewhere into
netfilter code as all packets traverse it (assuming it is loaded/enabled).

Bruno


> regards,
> Raviraj Joshi
> PICT,Pune.

kenel level packet capturing

[raviraj joshi]

Hi,

We are making a kernel level packet capture module as a part of our BE project .
We are finding it difficult to understand kenel networking code.
We studied structures and functions like sk_buff, netif_rx(),dev_alloc_skb.

However we were not able to trace the actual path a simple packet follows.

Moreover we tried to count the no of times netif_rx function was
called it was much much lower than amount of packets actually being
received.

Can anyone please let us know the reason for this and how to approach
the problem of writing a kernel level packet capture module ,resoruces
we need to refer to (we are currently constrained to documents on
kernel-networking available online).

And even from where to get information about how forwarding(or packet
flow) occours when we have virtual bridges (in terms of networking
code).

regards,
Raviraj Joshi
PICT,Pune.

Re: kenel level packet capturing

[Eric Dumazet]

Le mercredi 19 octobre 2011 à 12:24 +0530, raviraj joshi a écrit :
> Hi,
> 
> We are making a kernel level packet capture module as a part of our BE project .
> We are finding it difficult to understand kenel networking code.
> We studied structures and functions like sk_buff, netif_rx(),dev_alloc_skb.
> 
> However we were not able to trace the actual path a simple packet follows.
> 
> Moreover we tried to count the no of times netif_rx function was
> called it was much much lower than amount of packets actually being
> received.
> 
> Can anyone please let us know the reason for this and how to approach
> the problem of writing a kernel level packet capture module ,resoruces
> we need to refer to (we are currently constrained to documents on
> kernel-networking available online).
> 

Most drivers call netif_receive_skb() from softirq handler

netif_rx() is used only on cases where we are in hard irq handler, or we
want to avoid a recursion (tunnels for example)

To install a packet capture, you should take a look at dev_add_pack()
and friends [ net/packet/af_packet.c being a good reference ]



--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kenel level packet capturing

[David Miller]

From: raviraj joshi <raviraj.j1991@gmail.com>
Date: Wed, 19 Oct 2011 12:24:01 +0530

> We are finding it difficult to understand kenel networking code.

That's surprising, because the Linux networking stack is the cleanest
real world networking implementation on the planet.

Perhaps you should think about re-evaluating your project choice?

Re: kenel level packet capturing

[Ajith Adapa]

Yeah you are right David. But ..

For a newbie who just know more about linux userspace level need to
know starting steps like where to start from and flow of packet might
be confusing because of the level of abstraction being used. I guess
it might take some time until you are really serious about making
hands dirty :)

Regards,
Ajith



On Wed, Oct 19, 2011 at 12:43 PM, David Miller <davem@davemloft.net> wrote:
>
> From: raviraj joshi <raviraj.j1991@gmail.com>
> Date: Wed, 19 Oct 2011 12:24:01 +0530
>
> > We are finding it difficult to understand kenel networking code.
>
> That's surprising, because the Linux networking stack is the cleanest
> real world networking implementation on the planet.
>
> Perhaps you should think about re-evaluating your project choice?
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kenel level packet capturing

[Ajith Adapa]

Yeah you are right David. But ..

For a newbie who just know more about linux userspace level need to
know starting steps like where to start from and flow of packet might
be confusing because of the level of abstraction being used. I guess
it might take some time until you are really serious about making
hands dirty :)

Regards,
Ajith



On Wed, Oct 19, 2011 at 12:43 PM, David Miller <davem@davemloft.net> wrote:
>
> From: raviraj joshi <raviraj.j1991@gmail.com>
> Date: Wed, 19 Oct 2011 12:24:01 +0530
>
> > We are finding it difficult to understand kenel networking code.
>
> That's surprising, because the Linux networking stack is the cleanest
> real world networking implementation on the planet.
>
> Perhaps you should think about re-evaluating your project choice?
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kenel level packet capturing

[Richard Cochran]

On Wed, Oct 19, 2011 at 02:22:35PM +0530, Ajith Adapa wrote:
> Yeah you are right David. But ..
> 
> For a newbie who just know more about linux userspace level need to
> know starting steps like where to start from and flow of packet might
> be confusing because of the level of abstraction being used. I guess
> it might take some time until you are really serious about making
> hands dirty :)

For working with or just reading kernel sources I recommend getting to
know the cscope tool. Do a "make cscope" and then use one of the front
end browsers.

If you use emacs, then you really should have this set

	; When working on the kernel
	(setq cscope-do-not-update-database t)

otherwise it tries to re-index after every search.

HTH,
Richard

Re: kenel level packet capturing

[Benjamin Poirier]

On 11/10/19 12:24, raviraj joshi wrote:
> Hi,
> 
> We are making a kernel level packet capture module as a part of our BE project .
> We are finding it difficult to understand kenel networking code.
> We studied structures and functions like sk_buff, netif_rx(),dev_alloc_skb.
> 
> However we were not able to trace the actual path a simple packet follows.

You might find
http://vger.kernel.org/netconf2011_slides/jesse_brandeburg_netconf2011.pdf
interesting. It contains call chains of UDP processing at all levels in
the stack (driver, below ip stack, above ip stack) for tx and rx.

-Ben

Re: kenel level packet capturing

[raviraj joshi]

Hi,
    We have to implement a kernel module for capturing of packets and
dumping onto disk as a part of our BE project.
We have decided to use PF_RING(a kernel module to capture packets) for
the same due to the number of advantages.

We have to now write a kernel module that will interact with pf_ring
module to get data and dump it onto disk.
To do this we figured out two approaches :

1. pf_ring has provided userland code to communicate with kernel
module which in turn used "socket" system calls and specified protocol
family as PF_RING(socket(PF_RING, SOCK_RAW, htons(ETH_P_ALL));) and
subsequently used other functions like setsockopt to get packets.
So one approach can be to use kernel counterparts of socket and
setsockopt and reuse the userland code in our kernel module(this will
save us from writing code to built some specific data structures and
handleing exceptional conditions).

2. second approach can be to identify functions which capture packets
in PF_RING kernel module and invoke them from our
kernel moduel.

Can anyone please comment on these approachs or tell us about other
options available, we have to essentially use services provided by
pf_ring module via a kernel module?

and what things should we follow to accomplish these approaches ?



PF_RING specific questions(if any one is aware of pf_ring please answer this):
We went through the userland code of pf_ring and got to know that
using pf_ring involves
1.setting up socket connection
2.binding the address
3.setting socket options
4.enabling socket ring
5.use packet_recv function to get packets
6.finally display packets and close the socket connection.
is this procedure which we figured out right?

we also found that the data we receive has two parts
1.pfring_pkthdr
2.sk_buff struct

is this correct?

we know this is not the right place to ask questions about pf_ring and
we are extremely sorry for this ,but we posted the
same questions on ntop mailing list and there was no response.

thanking,
Raviraj Joshi
PICT,Pune.

On Wed, Oct 19, 2011 at 6:07 PM, Benjamin Poirier
<benjamin.poirier@gmail.com> wrote:
> On 11/10/19 12:24, raviraj joshi wrote:
>> Hi,
>>
>> We are making a kernel level packet capture module as a part of our BE project .
>> We are finding it difficult to understand kenel networking code.
>> We studied structures and functions like sk_buff, netif_rx(),dev_alloc_skb.
>>
>> However we were not able to trace the actual path a simple packet follows.
>
> You might find
> http://vger.kernel.org/netconf2011_slides/jesse_brandeburg_netconf2011.pdf
> interesting. It contains call chains of UDP processing at all levels in
> the stack (driver, below ip stack, above ip stack) for tx and rx.
>
> -Ben

Re: kenel level packet capturing

[David Miller]

From: raviraj joshi <raviraj.j1991@gmail.com>
Date: Tue, 13 Dec 2011 17:34:22 +0530

> We have decided to use PF_RING(a kernel module to capture packets) for
> the same due to the number of advantages.

What "advantages"?  The AF_PACKET socket layer already upstream in the
kernel supports every relevant performance feature PF_RING does, and
then some.

Re: kenel level packet capturing

[Jasper Spaans]

On 13/12/11 19:36, David Miller wrote:
> We have decided to use PF_RING(a kernel module to capture packets) for
> the same due to the number of advantages.
> What "advantages"?  The AF_PACKET socket layer already upstream in the
> kernel supports every relevant performance feature PF_RING does, and
> then some.
Some documentation for fanout is one feature that would be nice. (How to
use it from userspace - after 5 minutes of googling, the first place
where I could find an example is in the suricata source code[1]).

Having not tested it in practice, so my assumptions about rxhash might
be untrue: is there an option to override the NIC rxhash
implementations? if you are doing packet capturing, you will probably
see traffic in both directions, so hashing on the address-tuples of both
directions makes sense (or else both half-streams of tcp packets might
end up being sent to different sockets.

It seems hardware vendors are focussed on only hashing the senders
address, which does make sense if you're building a
fileserver/webserver/whatever.

We ended up using atrocious bpfs like "((tcp[0:2] & 1) == (tcp[2:2] &
1)) and tcp" to distribute packets over multiple sockets, and now our
bottleneck has shifted away from the capturing side of our application.

(Note to Raviraj: we're not using PF_RING because of stability issues -
it crashed our testing system one too many times.)

Cheers,
Jasper

[1]
https://redmine.openinfosecfoundation.org/embedded/suricata/source-af-packet_8c-source.html#l00516

-- 
 /\____/\   Ir. Jasper Spaans      
 \   (_)/   Fox-IT Experts in IT Security!
  \    X    T: +31-15-2847999
   \  / \   M: +31-6-41588725   
    \/      KvK Haaglanden 27301624

Re: kenel level packet capturing

[Eric Dumazet]

Le mercredi 14 décembre 2011 à 10:34 +0100, Jasper Spaans a écrit :
> On 13/12/11 19:36, David Miller wrote:
> > We have decided to use PF_RING(a kernel module to capture packets) for
> > the same due to the number of advantages.
> > What "advantages"?  The AF_PACKET socket layer already upstream in the
> > kernel supports every relevant performance feature PF_RING does, and
> > then some.
> Some documentation for fanout is one feature that would be nice. (How to
> use it from userspace - after 5 minutes of googling, the first place
> where I could find an example is in the suricata source code[1]).
> 
> Having not tested it in practice, so my assumptions about rxhash might
> be untrue: is there an option to override the NIC rxhash
> implementations? if you are doing packet capturing, you will probably
> see traffic in both directions, so hashing on the address-tuples of both
> directions makes sense (or else both half-streams of tcp packets might
> end up being sent to different sockets.
> 

Disabling NIC rxhash is possible :

# ethtool -k eth2 | grep receive-hashing
receive-hashing: on
# ethtool -K eth2 rxhash off
# ethtool -k eth2 | grep receive-hashing
receive-hashing: off

> It seems hardware vendors are focussed on only hashing the senders
> address, which does make sense if you're building a
> fileserver/webserver/whatever.
> 
> We ended up using atrocious bpfs like "((tcp[0:2] & 1) == (tcp[2:2] &
> 1)) and tcp" to distribute packets over multiple sockets, and now our
> bottleneck has shifted away from the capturing side of our application.

If you want to distribute load on tcp sockets, you can take a look at
various tricks documented in Documentation/networking/scaling.txt

(RPS, RFS, ...)


> (Note to Raviraj: we're not using PF_RING because of stability issues -
> it crashed our testing system one too many times.)
> 
> Cheers,
> Jasper
> 
> [1]
> https://redmine.openinfosecfoundation.org/embedded/suricata/source-af-packet_8c-source.html#l00516
> 

Re: kenel level packet capturing

[raviraj joshi]

On Wed, Dec 14, 2011 at 12:06 AM, David Miller <davem@davemloft.net> wrote:
> From: raviraj joshi <raviraj.j1991@gmail.com>
> Date: Tue, 13 Dec 2011 17:34:22 +0530
>
>> We have decided to use PF_RING(a kernel module to capture packets) for
>> the same due to the number of advantages.
>
> What "advantages"?  The AF_PACKET socket layer already upstream in the
> kernel supports every relevant performance feature PF_RING does, and
> then some.
I refered to the document on "A Measurement Study of Packet Reception
using Linux"[1]  which said pf_ring maintains
a ring buffer, so we dont have to issue a receive system call for each
packet in contrast to AF_PACKET which issues a system call for each
packet(pls correct me if i am wrong).

I would like to know can we use AF_PACKET services from a kernel
module as we are not bound to pf_ring and can use
AF_PACKET if its efficient and easy to use?


thanking,

Raviraj Joshi


[1]
http://www.cs.wustl.edu/~jain/cse567-11/ftp/pkt_recp/index.html

Re: kenel level packet capturing

[Jan Engelhardt]

On Wednesday 2011-12-14 11:46, raviraj joshi wrote:
>On Wed, Dec 14, 2011 at 12:06 AM, David Miller wrote:
>>
>>>We have decided to use PF_RING(a kernel module to capture packets)
>>>for the same due to the number of advantages.
>>
>>What "advantages"?  The AF_PACKET socket layer already upstream in
>>the kernel supports every relevant performance feature PF_RING
>>does, and then some.
>
>I refered to the document on "A Measurement Study of Packet
>Reception using Linux"[1] which said pf_ring maintains a ring
>buffer, so we dont have to issue a receive system call for each
>packet in contrast to AF_PACKET which issues a system call for each
>packet(pls correct me if i am wrong).

AF_PACKET has had mmap support for a loooong time. The initial code
traces back to 2.3.15pre3 (1999-Aug-23).
So much for $academia.

>[1]
>http://www.cs.wustl.edu/~jain/cse567-11/ftp/pkt_recp/index.html
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kenel level packet capturing

[Jan Engelhardt]

On Wednesday 2011-12-14 11:46, raviraj joshi wrote:
>On Wed, Dec 14, 2011 at 12:06 AM, David Miller wrote:
>>
>>>We have decided to use PF_RING(a kernel module to capture packets)
>>>for the same due to the number of advantages.
>>
>>What "advantages"?  The AF_PACKET socket layer already upstream in
>>the kernel supports every relevant performance feature PF_RING
>>does, and then some.
>
>I refered to the document on "A Measurement Study of Packet
>Reception using Linux"[1] which said pf_ring maintains a ring
>buffer, so we dont have to issue a receive system call for each
>packet in contrast to AF_PACKET which issues a system call for each
>packet(pls correct me if i am wrong).

AF_PACKET has had mmap support for a loooong time. The initial code
traces back to 2.3.15pre3 (1999-Aug-23).
So much for $academia.

>[1]
>http://www.cs.wustl.edu/~jain/cse567-11/ftp/pkt_recp/index.html
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kenel level packet capturing

[David Miller]

From: raviraj joshi <raviraj.j1991@gmail.com>
Date: Wed, 14 Dec 2011 16:16:01 +0530

> On Wed, Dec 14, 2011 at 12:06 AM, David Miller <davem@davemloft.net> wrote:
>> From: raviraj joshi <raviraj.j1991@gmail.com>
>> Date: Tue, 13 Dec 2011 17:34:22 +0530
>>
>>> We have decided to use PF_RING(a kernel module to capture packets) for
>>> the same due to the number of advantages.
>>
>> What "advantages"?  The AF_PACKET socket layer already upstream in the
>> kernel supports every relevant performance feature PF_RING does, and
>> then some.
> I refered to the document on "A Measurement Study of Packet Reception
> using Linux"[1]  which said pf_ring maintains
> a ring buffer, so we dont have to issue a receive system call for each
> packet in contrast to AF_PACKET which issues a system call for each
> packet(pls correct me if i am wrong).

AF_PACKET supports mmap()'d packet rings, and even supports variable
packet lengths within those rings.

AF_PACKET supports all the worthwhile performance features of PR_RING
and it's upstream, stop kidding yourself.

I'm really sick and tired of people saying PF_RING is better than
what we have upstream, it really isn't.
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kenel level packet capturing

[Eduardo Panisset]

Hi Raviraj,

You could be interested in taking a look at
"Documentation/networking/packet_mmap.txt" (this talks about mmap()'d
packet ring implementation for AF_PACKET).

Rgds,
Eduardo Panisset.

On Wed, Dec 14, 2011 at 2:09 PM, David Miller <davem@davemloft.net> wrote:
> From: raviraj joshi <raviraj.j1991@gmail.com>
> Date: Wed, 14 Dec 2011 16:16:01 +0530
>
>> On Wed, Dec 14, 2011 at 12:06 AM, David Miller <davem@davemloft.net> wrote:
>>> From: raviraj joshi <raviraj.j1991@gmail.com>
>>> Date: Tue, 13 Dec 2011 17:34:22 +0530
>>>
>>>> We have decided to use PF_RING(a kernel module to capture packets) for
>>>> the same due to the number of advantages.
>>>
>>> What "advantages"?  The AF_PACKET socket layer already upstream in the
>>> kernel supports every relevant performance feature PF_RING does, and
>>> then some.
>> I refered to the document on "A Measurement Study of Packet Reception
>> using Linux"[1]  which said pf_ring maintains
>> a ring buffer, so we dont have to issue a receive system call for each
>> packet in contrast to AF_PACKET which issues a system call for each
>> packet(pls correct me if i am wrong).
>
> AF_PACKET supports mmap()'d packet rings, and even supports variable
> packet lengths within those rings.
>
> AF_PACKET supports all the worthwhile performance features of PR_RING
> and it's upstream, stop kidding yourself.
>
> I'm really sick and tired of people saying PF_RING is better than
> what we have upstream, it really isn't.
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: kenel level packet capturing

[chetan loke]

> Some documentation for fanout is one feature that would be nice. (How
> to
> use it from userspace - after 5 minutes of googling, the first place
> where I could find an example is in the suricata source code[1]).

How about netdev's email archives ;) ? You would have noticed Dave's
sample fanout code. you can take that email and easily churn it into
20-50 lines of doc.

Re: kenel level packet capturing

[Jasper Spaans]

On 14/12/11 15:55, chetan loke wrote:
>> Some documentation for fanout is one feature that would be nice. (How
>> to
>> use it from userspace - after 5 minutes of googling, the first place
>> where I could find an example is in the suricata source code[1]).
> How about netdev's email archives ;) ? You would have noticed Dave's
> sample fanout code. you can take that email and easily churn it into
> 20-50 lines of doc.
Hmm google didn't tell me about those, so they don't exist (...)

I'll make some time to write up some documentation, as I'm interested in
using fanout.

-- 
 /\____/\   Ir. Jasper Spaans      
 \   (_)/   Fox-IT Experts in IT Security!
  \    X    T: +31-15-2847999
   \  / \   M: +31-6-41588725   
    \/      KvK Haaglanden 27301624

RE: kenel level packet capturing

[Eric Dumazet]

Le mercredi 14 décembre 2011 à 09:55 -0500, chetan loke a écrit :
> > Some documentation for fanout is one feature that would be nice. (How
> > to
> > use it from userspace - after 5 minutes of googling, the first place
> > where I could find an example is in the suricata source code[1]).
> 
> How about netdev's email archives ;) ? You would have noticed Dave's
> sample fanout code. you can take that email and easily churn it into
> 20-50 lines of doc.

Contributions to Documentation/network are higly welcomed.

Thanks !