flow start_time

flow start_time

[Michael Tokarev]

Hello.

Right now I'm evaluating various ways to perform traffic accounting
in Linux.  Nowadays solutions are all based on netfilter and packet
queuing over netlink interface.  And most tools used for that sort
of stuff are based on Cisco "flows".  So far so good.

There are several tools available to perform logging of various
statistics which are being logging over netlink, including the
successor of ulogd.  But all the tools I've seen so far perform
"flow management" in userspace in addition to all the housekeeping
the kernel already does - namely, mixing-n-matching various
conntrack entries together, keeping/managing hash tables of
various (src_IP, dst_IP, src_port, dst_port etc) stuff.

For flow-based accounting (compatible with cisco netflows I mean)
this all is only to get one field of the flow structure: it's
the start_time, i.e., in terms of conntrack, it's the time when
the conntrack entry were created.

But isn't it a bit excessive to duplicate all the (quite heavy
and fast-changing) data structures in both kernel- and user-space?
I mean, at least for the "industry standard" format, the one
missing field can be kept by kernel, especially since there,
it costs only extra 4 bytes, and zero CPU, while in duplicating
the whole thing in user space is something entirely else?

I understand that this is not a standard format, not even to
think about "The" standard, and there are other possible formats
too, with their own complications/requirements, and that for
users who does not use such statistics that's 4 bytes wasted
per conntrack which also may mean something...

But still, now not talking about cisco netflows in particular,
but from just logical point of view -- the information exposed
by the kernel is incomplete and lacks this very field, -- flow
start_time -- I think.

So, are there any objections/comments/suggestions about just
adding this one thing (conditionally using CONFIG_foo or not)?
Or maybe it is already there and there's nothing to do?

Thanks!

/mjt

Re: flow start_time

[Patrick McHardy]

Michael Tokarev wrote:
> Hello.
> 
> Right now I'm evaluating various ways to perform traffic accounting
> in Linux.  Nowadays solutions are all based on netfilter and packet
> queuing over netlink interface.  And most tools used for that sort
> of stuff are based on Cisco "flows".  So far so good.
> 
> There are several tools available to perform logging of various
> statistics which are being logging over netlink, including the
> successor of ulogd.  But all the tools I've seen so far perform
> "flow management" in userspace in addition to all the housekeeping
> the kernel already does - namely, mixing-n-matching various
> conntrack entries together, keeping/managing hash tables of
> various (src_IP, dst_IP, src_port, dst_port etc) stuff.
> 
> For flow-based accounting (compatible with cisco netflows I mean)
> this all is only to get one field of the flow structure: it's
> the start_time, i.e., in terms of conntrack, it's the time when
> the conntrack entry were created.
> 
> But isn't it a bit excessive to duplicate all the (quite heavy
> and fast-changing) data structures in both kernel- and user-space?
> I mean, at least for the "industry standard" format, the one
> missing field can be kept by kernel, especially since there,
> it costs only extra 4 bytes, and zero CPU, while in duplicating
> the whole thing in user space is something entirely else?

I think its done for aggregation, but not sure. And we used
to have 32bit counters for a while, so userspace needed to
track overflows.

> I understand that this is not a standard format, not even to
> think about "The" standard, and there are other possible formats
> too, with their own complications/requirements, and that for
> users who does not use such statistics that's 4 bytes wasted
> per conntrack which also may mean something...
> 
> But still, now not talking about cisco netflows in particular,
> but from just logical point of view -- the information exposed
> by the kernel is incomplete and lacks this very field, -- flow
> start_time -- I think.
> 
> So, are there any objections/comments/suggestions about just
> adding this one thing (conditionally using CONFIG_foo or not)?
> Or maybe it is already there and there's nothing to do?

How are you going to log the data of a connection thats removed
from the conntrack tables without using netlink? If you're using
netlink, you can generate a time stamp when you receive a NEW
event.

Re: flow start_time

[Michael Tokarev]

Patrick McHardy wrote:
> Michael Tokarev wrote:
>> Hello.
>>
>> Right now I'm evaluating various ways to perform traffic accounting
>> in Linux.  Nowadays solutions are all based on netfilter and packet
>> queuing over netlink interface.  And most tools used for that sort
>> of stuff are based on Cisco "flows".  So far so good.
>>
>> There are several tools available to perform logging of various
>> statistics which are being logging over netlink, including the
>> successor of ulogd.  But all the tools I've seen so far perform
>> "flow management" in userspace in addition to all the housekeeping
>> the kernel already does - namely, mixing-n-matching various
>> conntrack entries together, keeping/managing hash tables of
>> various (src_IP, dst_IP, src_port, dst_port etc) stuff.
>>
>> For flow-based accounting (compatible with cisco netflows I mean)
>> this all is only to get one field of the flow structure: it's
>> the start_time, i.e., in terms of conntrack, it's the time when
>> the conntrack entry were created.
>>
>> But isn't it a bit excessive to duplicate all the (quite heavy
>> and fast-changing) data structures in both kernel- and user-space?
>> I mean, at least for the "industry standard" format, the one
>> missing field can be kept by kernel, especially since there,
>> it costs only extra 4 bytes, and zero CPU, while in duplicating
>> the whole thing in user space is something entirely else?
> 
> I think its done for aggregation, but not sure. And we used
> to have 32bit counters for a while, so userspace needed to
> track overflows.

Aggregation is nice when needed.  Most tools that deals with
netflows does their own aggregation later, probably after some
filtering and probably on another hardware.

As for counters, they're 64bits now, so should be fine.

[]
>> So, are there any objections/comments/suggestions about just
>> adding this one thing (conditionally using CONFIG_foo or not)?
>> Or maybe it is already there and there's nothing to do?
> 
> How are you going to log the data of a connection thats removed
> from the conntrack tables without using netlink? If you're using
> netlink, you can generate a time stamp when you receive a NEW
> event.

In case either "new" or "remove" event is missing now, that connection
can't be logged properly, because we don't see some part of the
info in either of the two cases.

For nowadays tools, in case we've seen "new" but not "remove" event,
that connection will be in our hash table sorta forever (ok, till
restart).  Which is, I think, worse than not logging it at all (it
will not be logged anyway).

Basically, I don't see such a case (lost "remove" event) as valid.

But what I do see is: if our accounting tool were started long
time after system startup and when some connections are already
established, that tool will not know the actual start time --
the tool can force all current entries to be sent over netlink
again to catch up, but all that will be with current timestamp
obviously, since kernel does not keep that info now.

And there's one more interesting thing, too.

Suppose our tool gets restarted somehow.  Currently, it's forced
to write down all its current connections as if they were all
suddenly finished, and after restart there will be a bunch of
"new" connections -- all the same as just logged as "finished".
With the suggested approach, we don't need to care about that,
since we only interested in really finished connections, kernel
keeps track of the start time for us.  Sure we still can do our
own timekeeping as before -- for example, to force "daily" stats
at midnight even for connections which are still established.

Thanks.

/mjt

Re: flow start_time

[Pablo Neira Ayuso]

Michael Tokarev wrote:
> Patrick McHardy wrote:
>> How are you going to log the data of a connection thats removed
>> from the conntrack tables without using netlink? If you're using
>> netlink, you can generate a time stamp when you receive a NEW
>> event.
> 
> In case either "new" or "remove" event is missing now, that connection
> can't be logged properly, because we don't see some part of the
> info in either of the two cases.
> 
> For nowadays tools, in case we've seen "new" but not "remove" event,
> that connection will be in our hash table sorta forever (ok, till
> restart).  Which is, I think, worse than not logging it at all (it
> will not be logged anyway).
> 
> Basically, I don't see such a case (lost "remove" event) as valid.

I have a couple of patches here that I'm still working on it here to
avoid "remove" event loss. It should be ready for review once the
nf-next-2.6 tree is open.

To know when a flow starts and avoid the problem of "new" event loss. I
think that we can still add a conntrack extension to store the start
time. I think that's enough, right?

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: flow start_time

[Michael Tokarev]

Pablo Neira Ayuso wrote:
[]
>> For nowadays tools, in case we've seen "new" but not "remove" event,
>> that connection will be in our hash table sorta forever (ok, till
>> restart).  Which is, I think, worse than not logging it at all (it
>> will not be logged anyway).
>>
>> Basically, I don't see such a case (lost "remove" event) as valid.
> 
> I have a couple of patches here that I'm still working on it here to
> avoid "remove" event loss. It should be ready for review once the
> nf-next-2.6 tree is open.

Well, I didn't know it can occur to start with :)  But it's good to
know it's almost fixed -- thanks!

> To know when a flow starts and avoid the problem of "new" event loss. I
> think that we can still add a conntrack extension to store the start
> time. I think that's enough, right?

It is _exactly_ what I'm talking/asking about.  To say - Does anyone
have an objection to that?

Which time to use is entirely another question.  Should it be 'ticks'
or 'jiffies' or like time(2)/gettimeofday(2) with all the fun stuff
about system time resets etc.....  Oh well.

Thanks.

/mjt