[LARTC] Re: Bandwidth Metering

[LARTC] Re: Bandwidth Metering

[Patrick Coleman]

Thanks for those links. I'm building this server from scratch, so
kernel recompiling is fine. But will these tools be able to monitor
the bandwidth of individual users on the server itself? I realise you
grab the apache logs to monitor bandwidth for web servers, but what
about other services, say ssh, scp or wget? Is there an way to log the
total amount of data passing through an interface? Is this possible?
Thanks for your help,
Patrick


On Tue, 28 Sep 2004 18:54:59 +0200, Daniel Frederiksen
<cyberdoc@cyberdoc.dk> wrote:
> On Tue, 2004-09-28 at 16:25, Tomasz Chilinski wrote:
> > This is not good solution cause of high load where you account many
> nets
> > and/or hosts. In my opinion ACCOUNT/account from netfilter.org
> > patch-o-matic-ng are the best solutions for mass accounting.
> >
> > Bests,
> > Tomasz Chilinski
>
> Perhaps your right, but I use it on a bridge in front of a complete
> subnet, so at the time the easiest sollution was to log via libpcap.
> Of course it depends on how many hosts/subnets you are logging and the
> available CPU for it. My suggestion might be an easier sollution for
> Patrick, in that he does not need to patch his kernel and iptables for
> it, if his distro does not have ACCOUNT/account build in.
>
> Daniel Frederiksen, Cyberdoc.dk
>
>
>
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
>
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Re: Bandwidth Metering

[Daniel Frederiksen]

Hej Patrick

Again there might be to ways I think. You could log the traffic via
libpcap or netfilter. 

Netfilter could be set up to log specific traffic and afterwords you
could parse the logfile and flush it. The collected data could then be
put into a RRD base and graphed. The parsing of the logfiles is quite
simple.

Libpcap can do pretty much the same. However it might use up more CPU
and memory in the process. I guess I could whip up a perlscript that
uses libpcap to do that, if you don't know how.

There are other projects out there resembling all this, but with a lot
of other features included. Although I can't remember them at the time.
*ponders*. If it comes back to me I'll post it.

Best regards
Daniel Frederiksen, Cyberdoc.dk

On Wed, 2004-09-29 at 00:47, Patrick Coleman wrote:
> Thanks for those links. I'm building this server from scratch, so
> kernel recompiling is fine. But will these tools be able to monitor
> the bandwidth of individual users on the server itself? I realise you
> grab the apache logs to monitor bandwidth for web servers, but what
> about other services, say ssh, scp or wget? Is there an way to log the
> total amount of data passing through an interface? Is this possible?
> Thanks for your help,
> Patrick

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Re: Bandwidth Metering

[Patrick Coleman]

I ran across one called culprit
(http://freshmeat.net/projects/culprit.pl/), which does kinda what I
want. However, by the looks of things it doesnt look like its going to
be easy to maintain a database of user bandwidth usage using it. I
might see what I can do about modifying it, but I'm shocking at perl :)

Netfilter sounds like a much saner idea - how would you grab the user
a packet belongs to when you parse the logfiles?
Thanks,
Patrick

 
> On Wed, 29 Sep 2004 12:49:34 +0200, Daniel Frederiksen
> <cyberdoc@cyberdoc.dk> wrote:
> > Hej Patrick
> >
> > Again there might be to ways I think. You could log the traffic via
> > libpcap or netfilter.
> >
> > Netfilter could be set up to log specific traffic and afterwords you
> > could parse the logfile and flush it. The collected data could then be
> > put into a RRD base and graphed. The parsing of the logfiles is quite
> > simple.
> >
> > Libpcap can do pretty much the same. However it might use up more CPU
> > and memory in the process. I guess I could whip up a perlscript that
> > uses libpcap to do that, if you don't know how.
> >
> > There are other projects out there resembling all this, but with a lot
> > of other features included. Although I can't remember them at the time.
> > *ponders*. If it comes back to me I'll post it.
> >
> > Best regards
> > Daniel Frederiksen, Cyberdoc.dk
> >
> >
> >
> > On Wed, 2004-09-29 at 00:47, Patrick Coleman wrote:
> > > Thanks for those links. I'm building this server from scratch, so
> > > kernel recompiling is fine. But will these tools be able to monitor
> > > the bandwidth of individual users on the server itself? I realise you
> > > grab the apache logs to monitor bandwidth for web servers, but what
> > > about other services, say ssh, scp or wget? Is there an way to log the
> > > total amount of data passing through an interface? Is this possible?
> > > Thanks for your help,
> > > Patrick
> >
> > _______________________________________________
> > LARTC mailing list / LARTC@mailman.ds9a.nl
> > http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
> >
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Re: Bandwidth Metering

[Daniel Frederiksen]

Hey Patrick

I seem to have forgotten the point that you want to relate the bandwidth
usages to a user. In the perl script you posted a link for, the author
uses lsof. This sollution is ok, if the connection is still in the list,
however if you accumulate in a log from netfilter, the probability of
the connection still being active is reduced and not reliable. Is there
a specific reason why you want to map the user accounts, and are these
accounts system or actual users?. If they are just system accounts
running daemons, there are no point in mapping them.

If you still need the mapping, I will help you with the perl script, if
not we can use some of the previously sugested ideas. 

Perhaps a more detailed description of the usage/problem would help
allot.

Daniel Frederiksen, Cyberdoc.dk

On Wed, 2004-09-29 at 13:14, Patrick Coleman wrote:
> I ran across one called culprit
> (http://freshmeat.net/projects/culprit.pl/), which does kinda what I
> want. However, by the looks of things it doesnt look like its going to
> be easy to maintain a database of user bandwidth usage using it. I
> might see what I can do about modifying it, but I'm shocking at perl :)
> 
> Netfilter sounds like a much saner idea - how would you grab the user
> a packet belongs to when you parse the logfiles?
> Thanks,
> Patrick

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Re: Bandwidth Metering

[Patrick Coleman]

Sure - what I want to do is set up a colocated webhost/shell server,
and sell people accounts. However, I only have 2000mb upstream/2000mb
downstream free bandwidth, after which I start getting charged extra.
Therefore, I want to give *each user* a bandwidth quota, which if they
go over they can choose to either pay extra or have their account
disabled. Normal webhosts would do this in Apache (I would imagine),
with mod_quota or similar.

However, as I'm offering shell accounts as well, an Apache-only quota
system doesnt do what I want. I'm looking to find a way to meter the
outgoing and incoming bandwidth used by any program, on any socket,
and link that back to a particular user so I can make sure they dont
go over their limit. So mapping to a user account is essential. The
accounts are actual users. If you could give me an example of how you
would implement it in perl, it would be brilliant.
Thanks,
Patrick

On Wed, 29 Sep 2004 19:09:58 +0200, Daniel Frederiksen
<cyberdoc@cyberdoc.dk> wrote:
> Hey Patrick
> 
> I seem to have forgotten the point that you want to relate the bandwidth
> usages to a user. In the perl script you posted a link for, the author
> uses lsof. This sollution is ok, if the connection is still in the list,
> however if you accumulate in a log from netfilter, the probability of
> the connection still being active is reduced and not reliable. Is there
> a specific reason why you want to map the user accounts, and are these
> accounts system or actual users?. If they are just system accounts
> running daemons, there are no point in mapping them.
> 
> If you still need the mapping, I will help you with the perl script, if
> not we can use some of the previously sugested ideas.
> 
> Perhaps a more detailed description of the usage/problem would help
> allot.
> 
> Daniel Frederiksen, Cyberdoc.dk
> 
> 
> 
> On Wed, 2004-09-29 at 13:14, Patrick Coleman wrote:
> > I ran across one called culprit
> > (http://freshmeat.net/projects/culprit.pl/), which does kinda what I
> > want. However, by the looks of things it doesnt look like its going to
> > be easy to maintain a database of user bandwidth usage using it. I
> > might see what I can do about modifying it, but I'm shocking at perl :)
> >
> > Netfilter sounds like a much saner idea - how would you grab the user
> > a packet belongs to when you parse the logfiles?
> > Thanks,
> > Patrick
> 
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
> 



-- 
ACHTUNG - ALLES LOOKENPEEPERS
Das Machine is nicht fur gefingerpoken und mittengrabben. Ist easy
schnappen der springenwerk, blowenfusen und poppencorken mit
spitzensparken. Ist nicht fur gewerken by das dummkopfen. Das
rubbernecken sightseeren musten keepen das cotten-pickenen hands in
das pockets - relaxen und watchen das blinkenlights.
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/