faster way to read conntrack state than /proc ?

faster way to read conntrack state than /proc ?

[Willy Tarreau]

Hello,

On LKML, to someone complaining about the slowness of /proc/net/tcp, Davem
suggested to use the netlink interface instead which was really faster. BTW,
the guy confirmed that 'ss' took 6 seconds where 'netstat' took 15 min. So
I was wondering if there is such an equivalent for /proc/net/ip_conntrack,
and if not, what would be needed to implement it, because it's sometimes
annoying to eat 100% system on production machines during several minutes
just to get some statistics about connections status.

Anyone has a clue on this ?

Cheers,
Willy

Re: faster way to read conntrack state than /proc ?

[Martin Josefsson]

[-- Attachment #1: Type: text/plain, Size: 2413 bytes --]

On Sat, 2003-10-18 at 09:42, Willy Tarreau wrote:
> Hello,
> 
> On LKML, to someone complaining about the slowness of /proc/net/tcp, Davem
> suggested to use the netlink interface instead which was really faster. BTW,
> the guy confirmed that 'ss' took 6 seconds where 'netstat' took 15 min. So
> I was wondering if there is such an equivalent for /proc/net/ip_conntrack,
> and if not, what would be needed to implement it, because it's sometimes
> annoying to eat 100% system on production machines during several minutes
> just to get some statistics about connections status.
> 
> Anyone has a clue on this ?

We have something called nfnetlink and ctnetlink. ctnetlink has the
ability to dump all connections via netlink (unless someone, possibly
me, broke it).

see extra/nfnetlink-ctnetlink-0.11.patch in patch-o-matic

This patch is very experimental and it may crash your machine.
I have a few quick fixes at
http://kashyyyk.netfilter.org/~gandalf/patches/nfnetlink_conntrack-fix.diff
Some of them has been applied to cvs so you'll have to hand-merge.

This only gives very limited functionality (lookup, dump table, delete).
If you want more there's a patch in the netfilter-ha repository that
adds a lot of things.

To be able to use ctnetlink you'll probably want to use libctnetlink
which you can find in netfilter/iptables2 in cvs. And possibly apply
http://kashyyyk.netfilter.org/~gandalf/patches/libctnetlink-020809-2.diff

That's just a small patch I made to be able to perform lookups in a
reasonable manner. I use that in my patch to oidentd (availiable at the
same place) to make it use ctnetlink for connection-lookups instead of
using /proc/net/ip_conntrack. Using /p/n/ip_conntrack can bring the
machine to it's knees with just one ident request every few seconds if
you have enough connections for it to go through.
With ctnetlink I can perform over 100k lookups per cpu per second on a
2xpIII 800 (it has a bug where lookups can stall until one of the
programs performing the lookups is restarted, only seen in
stresstesting).

nfnetlink and ctnetlink will be part of pkttables which Harald is
working on.

As you may nave noticed, this is code that's not widely used (mainly
because noone's finished it, but the patch in netfilter-ha by Patrick
McHardy adds a great deal of the missing stuff), so don't expect too
much :)

-- 
/Martin

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: faster way to read conntrack state than /proc ?

[Willy TARREAU]

On Sat, Oct 18, 2003 at 10:55:19AM +0200, Martin Josefsson wrote:
 
> We have something called nfnetlink and ctnetlink. ctnetlink has the
> ability to dump all connections via netlink (unless someone, possibly
> me, broke it).

Thank you very much Martin for all these details. I will take a look at all
this soon. Don't worry, I won't complain about the bugs (though I may report
them), and I understand that this is not production-ready. My first goal was to
be able to report statistics about number of established, syn_recv, close_wait
and time_wait connections without killing the machine.

Cheers,
Willy

Re: faster way to read conntrack state than /proc ?

[Martin Josefsson]

[-- Attachment #1: Type: text/plain, Size: 781 bytes --]

On Sat, 2003-10-18 at 11:04, Willy TARREAU wrote:
> On Sat, Oct 18, 2003 at 10:55:19AM +0200, Martin Josefsson wrote:
>  
> > We have something called nfnetlink and ctnetlink. ctnetlink has the
> > ability to dump all connections via netlink (unless someone, possibly
> > me, broke it).
> 
> Thank you very much Martin for all these details. I will take a look at all
> this soon. Don't worry, I won't complain about the bugs (though I may report
> them)

With patches? :)

> , and I understand that this is not production-ready. My first goal was to
> be able to report statistics about number of established, syn_recv, close_wait
> and time_wait connections without killing the machine.

Yes, that would kill throughput using /p/n/ip_conntrack.

-- 
/Martin

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: faster way to read conntrack state than /proc ?

[Willy TARREAU]

On Sat, Oct 18, 2003 at 11:09:08AM +0200, Martin Josefsson wrote:
> > Thank you very much Martin for all these details. I will take a look at all
> > this soon. Don't worry, I won't complain about the bugs (though I may report
> > them)
> 
> With patches? :)

If I understand how it works, why something goes wrong and how to fix it, yes
of course ! 
But don't expect much at first :-)

Willy

Re: faster way to read conntrack state than /proc ?

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 1013 bytes --]

On Sat, Oct 18, 2003 at 10:55:19AM +0200, Martin Josefsson wrote:

> We have something called nfnetlink and ctnetlink. ctnetlink has the
> ability to dump all connections via netlink (unless someone, possibly
> me, broke it).
> 
> see extra/nfnetlink-ctnetlink-0.11.patch in patch-o-matic
> 
> This patch is very experimental and it may crash your machine.
> I have a few quick fixes at
> http://kashyyyk.netfilter.org/~gandalf/patches/nfnetlink_conntrack-fix.diff
> Some of them has been applied to cvs so you'll have to hand-merge.

I've now merged those fixes with nfnetlink-ctnetlink-0.11.patch in CVS

> -- 
> /Martin

-- 
- Harald Welte <laforge@netfilter.org>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]