Howto get ulog-prefix with ulogd-1.24 + sqlite3 backend?

Howto get ulog-prefix with ulogd-1.24 + sqlite3 backend?

[Clemens Eisserer]

Hi,

We try to use ulog for traffic accounting in an university project.

First we tried ulogd-2.0beta4 + pgsql backend, but even under moderate
load the database couldn't keep pace with the amount of requests.
I thought about using ulog2 with sqlite, however in TODO theres an
entry about making ulogd2 compatible with squlite, so I guessed sqlite
is not yet supported with ulogd2?

Then we switched back to ulog-1.24 + squlite3, which works perfectly.
It can handle 100mbit/s torrent traffic at about ~50% CPU load.
However in sqlite's ulog table, the ulog-prefix is not stored. Is
there any way to get it stored - to be able to track which rule caused
the log enrty?

Also, conntrack is supported in ulogd2. Is this also true for
ulogd1.2? I couldn't find specific docs about that.


Thank you in advance, Clemens

Re: Howto get ulog-prefix with ulogd-1.24 + sqlite3 backend?

[Jan Engelhardt]

On Saturday 2011-04-09 14:13, Clemens Eisserer wrote:

>Hi,
>
>We try to use ulog for traffic accounting in an university project.
>
>First we tried ulogd-2.0beta4 + pgsql backend, but even under moderate
>load the database couldn't keep pace with the amount of requests.
>I thought about using ulog2 with sqlite, however in TODO theres an
>entry about making ulogd2 compatible with squlite, so I guessed sqlite
>is not yet supported with ulogd2?

ulogd2 does have a sqlite backend already.

Re: Howto get ulog-prefix with ulogd-1.24 + sqlite3 backend?

[Eric Leblond]

Hi,


Clemens Eisserer <linuxhippy@gmail.com> a écrit :

>Hi,
>
>We try to use ulog for traffic accounting in an university project.
>
>First we tried ulogd-2.0beta4 + pgsql backend, but even under moderate
>load the database couldn't keep pace with the amount of requests.

Interesting. Could you share some information?

Do you use flat or multitable SQL schemas?
What is the insert rate? 

>I thought about using ulog2 with sqlite, however in TODO theres an
>entry about making ulogd2 compatible with squlite, so I guessed sqlite
>is not yet supported with ulogd2?
>
>Then we switched back to ulog-1.24 + squlite3, which works perfectly.
>It can handle 100mbit/s torrent traffic at about ~50% CPU load.
>However in sqlite's ulog table, the ulog-prefix is not stored. Is
>there any way to get it stored - to be able to track which rule caused
>the log enrty?
>
>Also, conntrack is supported in ulogd2. Is this also true for
>ulogd1.2? I couldn't find specific docs about that.

No. Support for conntrack has been added in Ulogd2.

BR, 

-- 
Envoyé de mon téléphone Android avec K-9 Mail. Excusez la brièveté.

Re: Howto get ulog-prefix with ulogd-1.24 + sqlite3 backend?

[Pablo Neira Ayuso]

On 09/04/11 14:13, Clemens Eisserer wrote:
> Hi,
> 
> We try to use ulog for traffic accounting in an university project.
> 
> First we tried ulogd-2.0beta4 + pgsql backend, but even under moderate
> load the database couldn't keep pace with the amount of requests.
> I thought about using ulog2 with sqlite, however in TODO theres an
> entry about making ulogd2 compatible with squlite, so I guessed sqlite
> is not yet supported with ulogd2?

The support for sqlite3 was recently added in the git tree. Get a git
snapshot. I have tested it here. I'd appreciate if you let me know how
it goes.

> Then we switched back to ulog-1.24 + squlite3, which works perfectly.
> It can handle 100mbit/s torrent traffic at about ~50% CPU load.
> However in sqlite's ulog table, the ulog-prefix is not stored. Is
> there any way to get it stored - to be able to track which rule caused
> the log enrty?

You mean the log prefix? The ulogd2 version includes it for sqlite3.

> Also, conntrack is supported in ulogd2. Is this also true for
> ulogd1.2? I couldn't find specific docs about that.

No ulogd1.2 does not support conntrack, only packet-based logging.

Re: Howto get ulog-prefix with ulogd-1.24 + sqlite3 backend?

[Peter Phaal]

You can significantly reduce the accounting load and improve
scalability by using packet sampling in the netfilter statistic
module. The Host sFlow daemon can be used as a ULOG monitor,
forwarding the packet samples as sFlow datagrams to a remote sFlow
collector, allowing centralized monitoring of large numbers of
servers.

For traffic accounting you might want to look at pmacct. pmacct can
receive sFlow from remote servers, or monitor ULOG locally, storing
the data in MySQL, PostgreSQL, SQLite or BerkeleyDB

See:
http://blog.sflow.com/2010/12/ulog.html
http://www.pmacct.net/

On Sat, Apr 9, 2011 at 5:13 AM, Clemens Eisserer <linuxhippy@gmail.com> wrote:
> Hi,
>
> We try to use ulog for traffic accounting in an university project.
>
> First we tried ulogd-2.0beta4 + pgsql backend, but even under moderate
> load the database couldn't keep pace with the amount of requests.
> I thought about using ulog2 with sqlite, however in TODO theres an
> entry about making ulogd2 compatible with squlite, so I guessed sqlite
> is not yet supported with ulogd2?
>
> Then we switched back to ulog-1.24 + squlite3, which works perfectly.
> It can handle 100mbit/s torrent traffic at about ~50% CPU load.
> However in sqlite's ulog table, the ulog-prefix is not stored. Is
> there any way to get it stored - to be able to track which rule caused
> the log enrty?
>
> Also, conntrack is supported in ulogd2. Is this also true for
> ulogd1.2? I couldn't find specific docs about that.
>
>
> Thank you in advance, Clemens
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>