How to modify conntrack accounting?

How to modify conntrack accounting?

[Ed W]

Hi, I have a requirement to account for "bytes I pay for" over some 
link, and conntrack very nearly gives me the right answer...  This link 
uses accounting somewhat like ATM, where the IP data is sliced into 
fixed size cells and you have to pay for the overhead per cell, plus the 
wasted space in the extra cell.

I look at the latest kernel sources and all the packet size accounting 
seems to be performed in: nf_conntrack_core.c / __nf_ct_refresh_acct() 
and __nf_ct_kill_acct().

I see several options:

1) Modify the accounting procedure in nf_conntrack_core.c so that 
certain connections will use a different accounting formula. However, 
how would I mark from userspace that a certain interface has this 
unusual accounting property?

2) Could/Should I produce a new netfilter module which operates per 
packet, looks up the connection object for a given packet, and then adds 
a "fudge" to the connection accounting number to correct for the effect 
of the odd packetisation? Presumably from userspace you would then 
simply create an iptables rule tagging packets out of a certain 
interface with "-m my_odd_accounting".

I don't yet know how to build option 2), but it seems appealing (anyone 
got any consultancy time and want to bill me to build it?)

I would appreciate feedback from those more knowledgeable? Given the 
small niche of the solution a modification to nf_conntrack_core.c is 
appealing, but I'm unsure how to indicate which are the peculiar 
interfaces, only userspace will know this.

Thanks for your thoughts/hints

Ed W

Re: How to modify conntrack accounting?

[Eric Leblond]

[-- Attachment #1: Type: text/plain, Size: 2146 bytes --]

Hello,

Le mardi 02 avril 2013 à 20:11 +0100, Ed W a écrit :
> Hi, I have a requirement to account for "bytes I pay for" over some 
> link, and conntrack very nearly gives me the right answer...  This link 
> uses accounting somewhat like ATM, where the IP data is sliced into 
> fixed size cells and you have to pay for the overhead per cell, plus the 
> wasted space in the extra cell.

I'm not sure I really understood your ATM comparison but why not use the
new accounting system like described here:

https://home.regit.org/2012/07/flow-accounting-with-netfilter-and-ulogd2/


BR,

> 
> I look at the latest kernel sources and all the packet size accounting 
> seems to be performed in: nf_conntrack_core.c / __nf_ct_refresh_acct() 
> and __nf_ct_kill_acct().
> 
> I see several options:
> 
> 1) Modify the accounting procedure in nf_conntrack_core.c so that 
> certain connections will use a different accounting formula. However, 
> how would I mark from userspace that a certain interface has this 
> unusual accounting property?
> 
> 2) Could/Should I produce a new netfilter module which operates per 
> packet, looks up the connection object for a given packet, and then adds 
> a "fudge" to the connection accounting number to correct for the effect 
> of the odd packetisation? Presumably from userspace you would then 
> simply create an iptables rule tagging packets out of a certain 
> interface with "-m my_odd_accounting".
> 
> I don't yet know how to build option 2), but it seems appealing (anyone 
> got any consultancy time and want to bill me to build it?)
> 
> I would appreciate feedback from those more knowledgeable? Given the 
> small niche of the solution a modification to nf_conntrack_core.c is 
> appealing, but I'm unsure how to indicate which are the peculiar 
> interfaces, only userspace will know this.
> 
> Thanks for your thoughts/hints
> 
> Ed W
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: How to modify conntrack accounting?

[Ed W]

On 02/04/2013 21:46, Eric Leblond wrote:
> Hello,
>
> Le mardi 02 avril 2013 à 20:11 +0100, Ed W a écrit :
>> Hi, I have a requirement to account for "bytes I pay for" over some
>> link, and conntrack very nearly gives me the right answer...  This link
>> uses accounting somewhat like ATM, where the IP data is sliced into
>> fixed size cells and you have to pay for the overhead per cell, plus the
>> wasted space in the extra cell.
> I'm not sure I really understood your ATM comparison but why not use the
> new accounting system like described here:
>
> https://home.regit.org/2012/07/flow-accounting-with-netfilter-and-ulogd2/
>

Thanks - I really need the raw conntrack detail so that I can do my own 
aggregations (its quite a complex requirement for accounting)

Apologies for the ATM assumptions.  OK, ATM works by encapsulating the 
data into small fixed sized packets of 48 bytes with a 5 byte header.  
So for example if you send a 1000 byte IP packet then it will consume 21 
ATM cells, which means a total bandwidth you pay for of 21 x (48+5) = 
1,113 bytes.   Annoyingly even the last (part filled) packet still has 
to be 48 bytes, plus you have that 5 byte header on each packet.

The above is how ADSL connections work, and why using standard QOS fails 
to work as expected when you have a bunch of small packets (ATM 
bandwidth becomes large compared with IP bandwidth).

I believe newer kernels have some options to correctly compute the 
overheads for ATM encapsulation, effectively I need to implement 
something similar for conntrack

Thanks for further thoughts

Ed W

--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How to modify conntrack accounting?

[Eric Leblond]

Hi,

On Tue, 2013-04-02 at 23:45 +0100, Ed W wrote:
> On 02/04/2013 21:46, Eric Leblond wrote:
> > Hello,
> >
> > Le mardi 02 avril 2013 à 20:11 +0100, Ed W a écrit :
> >> Hi, I have a requirement to account for "bytes I pay for" over some
> >> link, and conntrack very nearly gives me the right answer...  This link
> >> uses accounting somewhat like ATM, where the IP data is sliced into
> >> fixed size cells and you have to pay for the overhead per cell, plus the
> >> wasted space in the extra cell.
> > I'm not sure I really understood your ATM comparison but why not use the
> > new accounting system like described here:
> >
> > https://home.regit.org/2012/07/flow-accounting-with-netfilter-and-ulogd2/
> >
> 
> Thanks - I really need the raw conntrack detail so that I can do my own 
> aggregations (its quite a complex requirement for accounting)
> 
> Apologies for the ATM assumptions.  OK, ATM works by encapsulating the 
> data into small fixed sized packets of 48 bytes with a 5 byte header.  
> So for example if you send a 1000 byte IP packet then it will consume 21 
> ATM cells, which means a total bandwidth you pay for of 21 x (48+5) = 
> 1,113 bytes.   Annoyingly even the last (part filled) packet still has 
> to be 48 bytes, plus you have that 5 byte header on each packet.
> 
> The above is how ADSL connections work, and why using standard QOS fails 
> to work as expected when you have a bunch of small packets (ATM 
> bandwidth becomes large compared with IP bandwidth).

OK that is what I was fearing. You want to do ATM accounting at the IP
level ;)

> I believe newer kernels have some options to correctly compute the 
> overheads for ATM encapsulation, effectively I need to implement 
> something similar for conntrack
> 
> Thanks for further thoughts

You said you want a per-interface accounting but conntrack is not
interface aware. So a better solution may be to update nfacct modue to
increment counter with ATM-rounded value at each packet. Given the type
of patch this will require, it should be easy to maintain other time.
This will provide a more performant and easy to use framework.

By the way, if you want consulting time, I've got some available ;)

BR,
> 
> Ed W
> 
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Eric Leblond <eric@regit.org>
Blog: https://home.regit.org/

--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How to modify conntrack accounting?

[Ed W]

On 03/04/2013 10:05, Eric Leblond wrote:

>> Thanks - I really need the raw conntrack detail so that I can do my own
>> aggregations (its quite a complex requirement for accounting)
>>
>> Apologies for the ATM assumptions.  OK, ATM works by encapsulating the
>> data into small fixed sized packets of 48 bytes with a 5 byte header.
>> So for example if you send a 1000 byte IP packet then it will consume 21
>> ATM cells, which means a total bandwidth you pay for of 21 x (48+5) =
>> 1,113 bytes.   Annoyingly even the last (part filled) packet still has
>> to be 48 bytes, plus you have that 5 byte header on each packet.
>>
>> The above is how ADSL connections work, and why using standard QOS fails
>> to work as expected when you have a bunch of small packets (ATM
>> bandwidth becomes large compared with IP bandwidth).
> OK that is what I was fearing. You want to do ATM accounting at the IP
> level ;)

Actually, although others probably want this, my requirements are very 
slightly different.  I have to account for charged bandwidth over an 
(expensive) satellite network, the specification is a bit peculiar, but 
I suspect that it's a type of MLPPP (same basic scenario though, cells 
of xx bytes with a y byte overhead)


> You said you want a per-interface accounting but conntrack is not
> interface aware.

Yes, so I need to tag my flows in some way that conntrack can see.


> So a better solution may be to update nfacct modue to
> increment counter with ATM-rounded value at each packet. Given the type
> of patch this will require, it should be easy to maintain other time.
> This will provide a more performant and easy to use framework.

OK, can I run the larger requirements past you because I like the idea 
of nfacct, but unsure that it will do what I desire

- I have a router with multiple internet connections. Could be anything, 
but lets assume we have 1) wifi to the internet, 2) 3G datacard 
(cellular) and 3) a relatively expensive satellite link.
- Customer is mobile so any of the above may work and roughly they will 
want to choose the cheapest wherever they are
- We provide a web interface which allows the customer to indicate a 
route preference and after that we use forced routing to put all the 
customers data out of the correct internet connection (interestingly 
this means two clients can use different internet connections at the 
same time, there is not a single primary internet connection at any moment)

- For now I need to log bandwidth consumed during a session (start/end 
time) using the tuple (customer, inet), IP/MAC+Time will be good enough 
to identify the customer for this scenario. This will be dumped into 
some radius accounting server.
- We will need interim updates on bandwidth consumed during a session in 
order to implement access control such that if the user exceeds xxMB or 
so much time online, etc.
- In the future we require more detailed bandwidth breakdown, eg we can 
plot some pretty graphs of usage by customer / protocol / inet choice / 
time.  Essentially we want to be able to show that "bob" was surfing 
facebook during his lunchbreak, over the expensive satellite connection.

Conntrack accounting gives me fairly close to the flows I want (seems to 
drop a few ICMP packets, but ...).  What I need is to be able to adjust 
for the services where we have some funny accounting due to being 
rounded into "cells".  It's not quite close enough to just guess.

I updated the nDPI module a month or so back to give some L7 
capabilities to the kernel. However, its a bit icky and I'm still not 
sure how to approach proper integration there though:
- I think I can see how to extend conntrack so that nDPI netfilter can 
store it's tracking info in the conntrack. This means we can tag all 
connections with L7 info. However, I haven't figured out how to extend 
this to userspace so that it's emitted as part of the connection 
accounting info?
- Also nDPI is moving in the direction of being extremely flexible and 
emitting full info about a flow, eg "this is an HTTPS connection with a 
cert for somewhere.com". So now I'm thinking that the netfilter 
integration should instead match the connection but emit the details of 
that match through nflog?  After all the design of nDPI is to categorise 
a connection and once categorised that's the end of it, no point 
dragging around that info in kernel space
- Alternatively we could leave all connection marking to userspace, ie 
the netfilter module could match packets and based on that I tag the 
flow with some connmark or assign it to pfacct category?

So, in theory how well would pfacct handle the situation of:

Separate accounting bin for:
- Source IP (either class C or possibly class B)
- Output inet interface (up to 32 of them) - this changes the byte 
counting forumula
- L7 protocol used (up to 255)


I have a captive portal which notices users arriving and so some of the 
above can be build dynamically, however, I'm trying to build everything 
such that it's setup once at boot and stays static thereafter


Could I trouble you for your thoughts please?


> By the way, if you want consulting time, I've got some available ;)

Sure - I'm interested

Cheers

Ed W