NetFlow / sFlow / IPFIX network probe proposal

NetFlow / sFlow / IPFIX network probe proposal

[Roman Tsisyk]

Hi All !

I would like to discuss about a little module idea.

I've been working with commodity hardware Linux-based software network
routers for many years.
In my experience, logging and accounting of network packets was one of
hardest problems.
Various technologies for logging and accounting network packets
exists, e.g. ulog/pcap/PF_RING + nProbe and so on. All these tools
work in user space and require a lot of context switches. Context
switches make soo heavy cpu load if you have 400kpps and more packet
throughput.

As far as I can see, the PF_RING socket is a fastest way for exporting
packets to user space, but aggregation is still doing in user space.
In my tests, it makes about 30-40% CPU load of modern Core Quad cpu,
where complex iptables + ipset rules can take not more than 10%.

I've been nurturing the idea of a network monitoring module for the
NetFilter subsystem for a long time.
This module may be useful for commodity hardware routers as well as
embedded devices and will allow to use the user space only for a
configuration.
Furthermore, its possible with iptables to capture packets in a
forward chain and fill correctly more fields in accounting records.

It would be really good to have such features:

1. Support multiple formats
NetFlow v5, v9 / IPFIX (RFC 3917, 3954, 5101, 5102, 5470) and sFlow (RFC 3176)
IPFIX is more universal and more flexible than NetFlow v5 and sFlow,
so its a more important.

2. Correctly and fully fill all fields in a packet
As I pointed above, pcap-based tools can't correctly fill all fields,
e.g. output interface, nexthop, and AS (but I don't know where get AS
information in the kernel yet).

3. Support multiple targets
Module need robust configuration, which should support more than one
target in same time. It can look like this:
---
sysctl -w net.probe.target0.format=v5
sysctl -w net.probe.target0.destination=172.16.12.12:9996

sysctl -w net.probe.target1.version=ipfix
# IPFIX template specification
sysctl -w net.probe.target1.fields=IPV4_SRC_ADDR,IPV4_DST_ADDR,PROTOCOL,IN_BYTES,IN_PKTS,OUT_BYTES,OUT_PKTS,L4_SRC_PORT,L4_DST_PORT
sysctl -w net.probe.target1.destination=172.16.12.14:9996

sysctl -w net.probe.target2.format=sflow
sysctl -w net.probe.target2.destination=172.16.12.15:6343

# Probe one network to first server
iptables -A FORWARD -s 192.168.0.0/24 -j PROBE --target 0
iptables -A FORWARD -d 192.168.0.0/24 -j PROBE --target 0

# Probe another network to second server
iptables -A FORWARD -s 192.168.5.0/24 -j PROBE --target 1
iptables -A FORWARD -d 192.168.5.0/24 -j PROBE --target 1

# Probe outgoing email flow to third server
iptables -A FORWARD -s 192.168.5.0/24 --dport 25 -j PROBE --target 2
---

4. Support for exporting statistics via /proc
Traffic and packet rate, udp socket information and everything what
can be useful.

5. Round-robin load-balancing for destinations
For high throughput networks it is impossible to save all flow data to
one server. Idea is to add support for round-robin balancing for the
collector address:
---
sysctl -w net.probe.target0.destinations="172.16.12.12:9996 172.16.12.13:9996"
---

6. Support for capturing mirrored to NIC packets (which is addressed
to another host).
Of course, Iptables simple drop these crap packets. May be make sysctl
option for this feature?

PROBE name is just an example and may be changed to something else.
Please correct me, if I forgot something.

What is wrong in ipt_NETFLOW module
(http://sourceforge.net/projects/ipt-netflow/) ?
Ipt_NETFLOW is a really useful module and it never load CPU more than
1%. But internal architecture of ipt_NETFLOW cannot support all of
described above features (because have one common hash table and
configuration with module params). I'd prefer not to reinvent the
wheel and think about redesigning ipt_NETFLOW or writing new module
using ipt_NETFLOW base.

I'm a student and want to work on this project within Summer of Code
program. The Linux Foundation site pointed to the mail list.
My main goal is to make high-quality featured module, that will be
useful for anybody and acceptable for inclusion into mainstream
Kernel.

What do you think about this? Please, advice me or give some ideas.
May be somebody wanted to be a mentor? I'll send detailed information
about my bio, contacts, skills, work experience as well as schedule
plan directly to a potential mentor and/or GSoC site.

BTW, a NetFlow exporter feature is a most rated request on the Vyatta
site, so it should be interesting not only for me.

-- 
WBR, Tsisyk Roman

Re: NetFlow / sFlow / IPFIX network probe proposal

[Changli Gao]

On Tue, Mar 30, 2010 at 10:06 PM, Roman Tsisyk <roman@tsisyk.com> wrote:
> Hi All !
>
> I would like to discuss about a little module idea.
>

I am interested in this feature. FreeBSD supports netflow v5, and
netflow v5 is more common, we can support it first, then add others.


-- 
Regards，
Changli Gao(xiaosuo@gmail.com)

Re: NetFlow / sFlow / IPFIX network probe proposal

[Jan Engelhardt]

On Tuesday 2010-03-30 16:06, Roman Tsisyk wrote:

># Probe one network to first server
>iptables -A FORWARD -s 192.168.0.0/24 -j PROBE --target 0
>iptables -A FORWARD -d 192.168.0.0/24 -j PROBE --target 0
>
># Probe another network to second server
>iptables -A FORWARD -s 192.168.5.0/24 -j PROBE --target 1
>iptables -A FORWARD -d 192.168.5.0/24 -j PROBE --target 1
>
># Probe outgoing email flow to third server
>iptables -A FORWARD -s 192.168.5.0/24 --dport 25 -j PROBE --target 2
>---
>
>4. Support for exporting statistics via /proc
>Traffic and packet rate, udp socket information and everything what
>can be useful.

Should probably be using netlink or sysfs instead.

>6. Support for capturing mirrored to NIC packets (which is addressed
>to another host).
>Of course, Iptables simple drop these crap packets. May be make sysctl
>option for this feature?

iptables does not drop these, your NIC does when it's not in promiscuous mode.

Re: NetFlow / sFlow / IPFIX network probe proposal

[Roman Tsisyk]

On Tue, Mar 30, 2010 at 9:29 PM, Changli Gao <xiaosuo@gmail.com> wrote:
> I am interested in this feature. FreeBSD supports netflow v5, and
> netflow v5 is more common, we can support it first, then add others.

We used ng_netflow before PF_RING. There was a lot of problems with
ksocket udp packets loss in the BSD kernel.  After that we have
switched to the ntop solution.
Ipt_NETFLOW provides basic v5 support, but as I already said doesn't
have flexible configuration and other features. So I intend to make
universal module and start with v5.

On Tue, Mar 30, 2010 at 9:56 PM, Jan Engelhardt <jengelh@medozas.de> wrote:
> On Tuesday 2010-03-30 16:06, Roman Tsisyk wrote:
>
> iptables does not drop these, your NIC does when it's not in promiscuous mode.
>

If I am no mistaken, for all packets whose mac doesn't match to the
nic mac pkt_type is set to PACKET_OTHERHOST.
Iptables drop packets with PACKET_OTHERHOST, I don't remember exactly
where, may be in ip_rcv routine.

-- 
WBR, Tsisyk Roman

Re: NetFlow / sFlow / IPFIX network probe proposal

[Jan Engelhardt]

On Tuesday 2010-03-30 17:57, Roman Tsisyk wrote:
>On Tue, Mar 30, 2010 at 9:56 PM, Jan Engelhardt <jengelh@medozas.de> wrote:
>> On Tuesday 2010-03-30 16:06, Roman Tsisyk wrote:
>>
>> iptables does not drop these, your NIC does when it's not in promiscuous mode.
>>
>
>If I am no mistaken, for all packets whose mac doesn't match to the
>nic mac pkt_type is set to PACKET_OTHERHOST.
>Iptables drop packets with PACKET_OTHERHOST, I don't remember exactly
>where, may be in ip_rcv routine.

Indeed. You could make it a runtime option (e.g. sysfs-moduleparam)
during your tests.

Re: NetFlow / sFlow / IPFIX network probe proposal

[Patrick McHardy]

Jan Engelhardt wrote:
> On Tuesday 2010-03-30 16:06, Roman Tsisyk wrote:
> 
>> # Probe one network to first server
>> iptables -A FORWARD -s 192.168.0.0/24 -j PROBE --target 0
>> iptables -A FORWARD -d 192.168.0.0/24 -j PROBE --target 0
>>
>> # Probe another network to second server
>> iptables -A FORWARD -s 192.168.5.0/24 -j PROBE --target 1
>> iptables -A FORWARD -d 192.168.5.0/24 -j PROBE --target 1
>>
>> # Probe outgoing email flow to third server
>> iptables -A FORWARD -s 192.168.5.0/24 --dport 25 -j PROBE --target 2
>> ---
>>
>> 4. Support for exporting statistics via /proc
>> Traffic and packet rate, udp socket information and everything what
>> can be useful.
> 
> Should probably be using netlink or sysfs instead.

We already have conntrack and ctnetlink to gather per-connection
statistics, which should decrease the overhead for doing this in
userspace a lot. There also exists a netflow plugin for ulogd2,
but I'm not sure it was already submitted and merged.

Re: NetFlow / sFlow / IPFIX network probe proposal

[Roman Tsisyk]

On Wed, Mar 31, 2010 at 2:06 AM, Stig Thormodsrud <stig@vyatta.com> wrote:
> Hi Roman,
>
> You mentioned that netflow exporter is the most requested feature on
> Vyatta.  That was true until today when VC6 was release with our 1st
> netflow/sflow exporter.  I used pmacct (http://www.pmacct.net) and ULOG
> to get the packets to uacctd.  There's certainly a lot more that can be
> done to support higher bandwidth interfaces.  I'm considering PF_RING.

I've read Vyatta site before VC6 was released :)
This is a ULOG-based solution or not?

On Tue, Mar 30, 2010 at 11:39 PM, Patrick McHardy <kaber@trash.net> wrote:
> We already have conntrack and ctnetlink to gather per-connection
> statistics, which should decrease the overhead for doing this in
> userspace a lot. There also exists a netflow plugin for ulogd2,
> but I'm not sure it was already submitted and merged.
>

Thank you for pointing it out, I didn't know about conntrack support in ulogd.

As far as I understood, IPFIX output in ulogd is in a early stage and
don't work. So, I tested ulogd + ctnetlink with null output and it
worked very well.
CPU load was about 5-10%, and it's just nothing on this router.
However, I'm not sure that output is correct and all flows was
accounted. I also don't know what is about active and inactive
timeouts in this approach.
I'll look to ulogd_inpflow_NFCT more closely.

Patrick, decision is to optimize ctnetlink and not to make accounting
in the kernel space?

-- 
WBR, Tsisyk Roman

Re: NetFlow / sFlow / IPFIX network probe proposal

[Patrick McHardy]

Roman Tsisyk wrote:
> On Wed, Mar 31, 2010 at 2:06 AM, Stig Thormodsrud <stig@vyatta.com> wrote:
> On Tue, Mar 30, 2010 at 11:39 PM, Patrick McHardy <kaber@trash.net> wrote:
>> We already have conntrack and ctnetlink to gather per-connection
>> statistics, which should decrease the overhead for doing this in
>> userspace a lot. There also exists a netflow plugin for ulogd2,
>> but I'm not sure it was already submitted and merged.
>>
> 
> Thank you for pointing it out, I didn't know about conntrack support in ulogd.
> 
> As far as I understood, IPFIX output in ulogd is in a early stage and
> don't work. So, I tested ulogd + ctnetlink with null output and it
> worked very well.

IIRC Holger Eitzenberger (CCed) has done some work to make this work
properly. Maybe he can tell use more.

> CPU load was about 5-10%, and it's just nothing on this router.
> However, I'm not sure that output is correct and all flows was
> accounted. I also don't know what is about active and inactive
> timeouts in this approach.
> I'll look to ulogd_inpflow_NFCT more closely.
> 
> Patrick, decision is to optimize ctnetlink and not to make accounting
> in the kernel space?

Accounting is done in kernel space by conntrack, but aggregation should
be done in userspace in my opinion. I don't think you need a lot of
optimization, AFAIK Holger's patches already scale to large setups
very well.

Re: NetFlow / sFlow / IPFIX network probe proposal

[Holger Eitzenberger]

(replying by private mail)

Hi,

> > Thank you for pointing it out, I didn't know about conntrack support in ulogd.
> > 
> > As far as I understood, IPFIX output in ulogd is in a early stage and
> > don't work. So, I tested ulogd + ctnetlink with null output and it
> > worked very well.

I've done an IPFIX implementation a few months ago for ulogd2, but it
currently is of prototype character only, as it only implements IPFIX
over UDP, but SCTP and possibly TCP (+ SSL) are required for a
conforming implementation.  I use the "bi-flow" approach of RFC 51303,
but it shouldn't be a problem at all to have it support the single
direction flow approach instead from RFC 5101/5102.

In general ctnetlink is not a problem here from a performance standpoint,
and I think there is no need for doing that in kernel space.

Some bits and pieces may be missing on the kernel side if you need to
classify your flows though.  E. g. some IPFIX Aggregrators require the
DSCP value for that.

> Accounting is done in kernel space by conntrack, but aggregation should
> be done in userspace in my opinion. I don't think you need a lot of
> optimization, AFAIK Holger's patches already scale to large setups
> very well.

I have setups with hundreds of thounsands of flows, and it accounts
quite well :).

Hope that helps a bit.

 /holger

Re: NetFlow / sFlow / IPFIX network probe proposal

[Pablo Neira Ayuso]

Hi,

Holger Eitzenberger wrote:
> (replying by private mail)
>>> Thank you for pointing it out, I didn't know about conntrack support in ulogd.
>>>
>>> As far as I understood, IPFIX output in ulogd is in a early stage and
>>> don't work. So, I tested ulogd + ctnetlink with null output and it
>>> worked very well.

IPFIX support in ulog2 in the git tree has been indeed
broken/unifinished since long time ago.

> I've done an IPFIX implementation a few months ago for ulogd2, but it
> currently is of prototype character only, as it only implements IPFIX
> over UDP, but SCTP and possibly TCP (+ SSL) are required for a
> conforming implementation.  I use the "bi-flow" approach of RFC 51303,
> but it shouldn't be a problem at all to have it support the single
> direction flow approach instead from RFC 5101/5102.

I've also have one of my internal tree, I started last summer but thesis
has prevent me from working on that.

> In general ctnetlink is not a problem here from a performance standpoint,
> and I think there is no need for doing that in kernel space.

Indeed.

> Some bits and pieces may be missing on the kernel side if you need to
> classify your flows though.  E. g. some IPFIX Aggregrators require the
> DSCP value for that.

We can post a patch to add the missing stuff to ctnetlink.

Re: NetFlow / sFlow / IPFIX network probe proposal

[Patrick McHardy]

Pablo Neira Ayuso wrote:
> Holger Eitzenberger wrote:
>> Some bits and pieces may be missing on the kernel side if you need to
>> classify your flows though.  E. g. some IPFIX Aggregrators require the
>> DSCP value for that.
> 
> We can post a patch to add the missing stuff to ctnetlink.

Adding DSCP doesn't work since we don't distinguish connections
by that.

Re: NetFlow / sFlow / IPFIX network probe proposal

[Holger Eitzenberger]

On Thu, Apr 01, 2010 at 04:33:10PM +0200, Patrick McHardy wrote:

> >> Some bits and pieces may be missing on the kernel side if you need to
> >> classify your flows though.  E. g. some IPFIX Aggregrators require the
> >> DSCP value for that.
> 
> Adding DSCP doesn't work since we don't distinguish connections
> by that.

That patch would be easy, but I was unsure at that time whether DSCP was a
"flow characteristic" enough, especially as you have something like the
Netfilter DSCP target to modify it, and that possibly be done not from
the very start of a flow.

 /holger

Re: NetFlow / sFlow / IPFIX network probe proposal

[Patrick McHardy]

Holger Eitzenberger wrote:
> On Thu, Apr 01, 2010 at 04:33:10PM +0200, Patrick McHardy wrote:
>
>   
>>>> Some bits and pieces may be missing on the kernel side if you need to
>>>> classify your flows though.  E. g. some IPFIX Aggregrators require the
>>>> DSCP value for that.
>>>>         
>> Adding DSCP doesn't work since we don't distinguish connections
>> by that.
>>     
>
> That patch would be easy, but I was unsure at that time whether DSCP was a
> "flow characteristic" enough, especially as you have something like the
> Netfilter DSCP target to modify it, and that possibly be done not from
> the very start of a flow.
>   
We certainly can't add it to the conntrack tuple, so I'm not sure
how that patch would work. But I'll gladly review it in case someone
takes a stab at this :)

Re: NetFlow / sFlow / IPFIX network probe proposal

[Jan Engelhardt]

On Thursday 2010-04-01 16:55, Patrick McHardy wrote:
>Holger Eitzenberger wrote:
>> On Thu, Apr 01, 2010 at 04:33:10PM +0200, Patrick McHardy wrote:
>>
>>   
>>>>> Some bits and pieces may be missing on the kernel side if you need to
>>>>> classify your flows though.  E. g. some IPFIX Aggregrators require the
>>>>> DSCP value for that.
>>>>>         
>>> Adding DSCP doesn't work since we don't distinguish connections
>>> by that.
>>>     
>>
>> That patch would be easy, but I was unsure at that time whether DSCP was a
>> "flow characteristic" enough, especially as you have something like the
>> Netfilter DSCP target to modify it, and that possibly be done not from
>> the very start of a flow.
>>   
>We certainly can't add it to the conntrack tuple, so I'm not sure
>how that patch would work. But I'll gladly review it in case someone
>takes a stab at this :)

 -j CT --save-tos

*patsch* *patsch* *autsch* ;)

Re: NetFlow / sFlow / IPFIX network probe proposal

[Florian Weimer]

* Roman Tsisyk:

> 2. Correctly and fully fill all fields in a packet
> As I pointed above, pcap-based tools can't correctly fill all fields,
> e.g. output interface, nexthop, and AS (but I don't know where get AS
> information in the kernel yet).

(Older) IOS populates those fields from route tags found in the FIB.
Quagga etc. would have to be taught to set these tags.  The advantage
is that no additional route lookups are required (at least when
rp_filter is active).

-- 
Florian Weimer                <fweimer@bfk.de>
BFK edv-consulting GmbH       http://www.bfk.de/
Kriegsstraße 100              tel: +49-721-96201-1
D-76133 Karlsruhe             fax: +49-721-96201-99