[RFC] ctnetlink events drop benchmark

[RFC] ctnetlink events drop benchmark

[Pablo Neira Ayuso]

Hi,

Finally, I've got some results in my test enviroment about the ctnetlink 
event drop issue. The machine used as firewall is a PIII 866 256 Mbytes 
RAM, two NIC realtek 8139 100Mbits running debian unstable with linux 
kernel 2.6.16.

I have run the same test with three different socket queue sizes:
107.520 (default)
215.040 (x2)
430.080 (x4)

I used netperf to create the connections with a shell script that loops 
calling netperf -H ip -l 3 &.

To get the number of events dropped I've used the following:

$ conntrackd & (run as daemon)
$ conntrackd -s (request statistics)

Results are available in:

http://people.netfilter.org/pablo/ctnetlink/events.eps

Currently nlmsg_seq is not set in ctnetlink events, if we set it, we can 
do some kind of sequence tracking on netlink. If an event gets lost, we 
can request a resync with the conntrack via dump_conntrack.

Another choice could be relax conntrackd states transitions, currently 
the valid transition sequence is: NEW -> UPDATE -> DESTROY, if an UPDATE 
event is received but no NEW was seen previously, then the event is 
ignored. Maybe this is too tight, but since I'm validating the whole 
thing I prefer remaining "tight" at the moment.

Comments welcome.

-- 
The dawn of the fourth age of Linux firewalling is coming; a time of 
great struggle and heroic deeds -- J.Kadlecsik got inspired by J.Morris

Re: [RFC] ctnetlink events drop benchmark

[Jozsef Kadlecsik]

Hi Pablo,

On Tue, 13 Jun 2006, Pablo Neira Ayuso wrote:

> Finally, I've got some results in my test enviroment about the ctnetlink
> event drop issue. The machine used as firewall is a PIII 866 256 Mbytes
> RAM, two NIC realtek 8139 100Mbits running debian unstable with linux
> kernel 2.6.16.
>
> I have run the same test with three different socket queue sizes:
> 107.520 (default)
> 215.040 (x2)
> 430.080 (x4)
>
> I used netperf to create the connections with a shell script that loops
> calling netperf -H ip -l 3 &.

How can you control the connection rate (i.e NEW events) by netperf?

> To get the number of events dropped I've used the following:
>
> $ conntrackd & (run as daemon)
> $ conntrackd -s (request statistics)
>
> Results are available in:
>
> http://people.netfilter.org/pablo/ctnetlink/events.eps

The results are pretty shocking! One would expect (hope) higher event rate
without event dropping :-(.

> Currently nlmsg_seq is not set in ctnetlink events, if we set it, we can
> do some kind of sequence tracking on netlink. If an event gets lost, we
> can request a resync with the conntrack via dump_conntrack.

Isn't that overkill? I mean, it'd result even more traffic and thus more
packet loss.

> Another choice could be relax conntrackd states transitions, currently
> the valid transition sequence is: NEW -> UPDATE -> DESTROY, if an UPDATE
> event is received but no NEW was seen previously, then the event is
> ignored. Maybe this is too tight, but since I'm validating the whole
> thing I prefer remaining "tight" at the moment.

As I see it does not improve the performance over netlink: the message is
already received by conntrackd.

Can we generalize your results and state that (ct)netlink can handle (on
the given hardvare) max 500-1000 messages/s?

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [RFC] ctnetlink events drop benchmark

[Jozsef Kadlecsik]

On Wed, 14 Jun 2006, Jozsef Kadlecsik wrote:

> Can we generalize your results and state that (ct)netlink can handle (on
> the given hardvare) max 500-1000 messages/s?

Oops, that's false: 500-1000 NEW messages/s. But what is the total number
of messages/s?

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [RFC] ctnetlink events drop benchmark

[Pablo Neira Ayuso]

Hi Jozsef,

Jozsef Kadlecsik wrote:
> On Tue, 13 Jun 2006, Pablo Neira Ayuso wrote:
> 
> 
>>Finally, I've got some results in my test enviroment about the ctnetlink
>>event drop issue. The machine used as firewall is a PIII 866 256 Mbytes
>>RAM, two NIC realtek 8139 100Mbits running debian unstable with linux
>>kernel 2.6.16.
>>
>>I have run the same test with three different socket queue sizes:
>>107.520 (default)
>>215.040 (x2)
>>430.080 (x4)
>>
>>I used netperf to create the connections with a shell script that loops
>>calling netperf -H ip -l 3 &.
> 
> 
> How can you control the connection rate (i.e NEW events) by netperf?

I can't, but I can measure the time required to create N connections

>>To get the number of events dropped I've used the following:
>>
>>$ conntrackd & (run as daemon)
>>$ conntrackd -s (request statistics)
>>
>>Results are available in:
>>
>>http://people.netfilter.org/pablo/ctnetlink/events.eps
> 
> 
> The results are pretty shocking! One would expect (hope) higher event rate
> without event dropping :-(.
> 
> 
>>Currently nlmsg_seq is not set in ctnetlink events, if we set it, we can
>>do some kind of sequence tracking on netlink. If an event gets lost, we
>>can request a resync with the conntrack via dump_conntrack.
> 
> Isn't that overkill? I mean, it'd result even more traffic and thus more
> packet loss.

AFAIK, dumping in done in data chunks of one memory page long. A data 
chunk isn't enqueued until the process calls recv(), therefore a new 
chunk is enqueued iff the process has processed the previous chunk. So, 
dumping really doesn't consumes too much space in the queue but you are 
right, if the dumping happens at the time that the queue is full, then 
the packet will get lost.

>>Another choice could be relax conntrackd states transitions, currently
>>the valid transition sequence is: NEW -> UPDATE -> DESTROY, if an UPDATE
>>event is received but no NEW was seen previously, then the event is
>>ignored. Maybe this is too tight, but since I'm validating the whole
>>thing I prefer remaining "tight" at the moment.
> 
> As I see it does not improve the performance over netlink: the message is
> already received by conntrackd.
> 
> Can we generalize your results and state that (ct)netlink can handle (on
> the given hardvare) max 500-1000 messages/s?

No, the graph doesn't talk about messages/s, just messages: after 
reading your comments, it seems that the graph can be misleading.

Please, have a look at these results in text format:

http://people.netfilter.org/pablo/ctnetlink/107520.txt

I think that it can be better to get some conclusions.

-- 
The dawn of the fourth age of Linux firewalling is coming; a time of 
great struggle and heroic deeds -- J.Kadlecsik got inspired by J.Morris

Re: [RFC] ctnetlink events drop benchmark

[Pablo Neira Ayuso]

Jozsef Kadlecsik wrote:
> On Wed, 14 Jun 2006, Jozsef Kadlecsik wrote:
> 
>>Can we generalize your results and state that (ct)netlink can handle (on
>>the given hardvare) max 500-1000 messages/s?
> 
> Oops, that's false: 500-1000 NEW messages/s. But what is the total number
> of messages/s?

Actually, my concern wasn't really to get the messages/s rate. I started 
the benchmark to know if ctnetlink can back off under big stress 
situation, in other words, if I could experiment messages loss. This is 
really important to validate conntrackd. And it could also be for other 
subsystems like nfqueue.

-- 
The dawn of the fourth age of Linux firewalling is coming; a time of 
great struggle and heroic deeds -- J.Kadlecsik got inspired by J.Morris

Re: [RFC] ctnetlink events drop benchmark

[Patrick McHardy]

Pablo Neira Ayuso wrote:
> Hi,
> 
> Finally, I've got some results in my test enviroment about the ctnetlink
> event drop issue. The machine used as firewall is a PIII 866 256 Mbytes
> RAM, two NIC realtek 8139 100Mbits running debian unstable with linux
> kernel 2.6.16.
> 
> I have run the same test with three different socket queue sizes:
> 107.520 (default)
> 215.040 (x2)
> 430.080 (x4)
> 
> I used netperf to create the connections with a shell script that loops
> calling netperf -H ip -l 3 &.
> 
> To get the number of events dropped I've used the following:
> 
> $ conntrackd & (run as daemon)
> $ conntrackd -s (request statistics)
> 
> Results are available in:
> 
> http://people.netfilter.org/pablo/ctnetlink/events.eps

Does "events" mean events/s?

> Currently nlmsg_seq is not set in ctnetlink events, if we set it, we can
> do some kind of sequence tracking on netlink. If an event gets lost, we
> can request a resync with the conntrack via dump_conntrack.

nlmsg_seq should only (but in that case it really should) be set for
responses to queries, so userspace can associate responses with queries.
Drops are reported as -ENOBUFS socket errors to all multicast listeners.

> Another choice could be relax conntrackd states transitions, currently
> the valid transition sequence is: NEW -> UPDATE -> DESTROY, if an UPDATE
> event is received but no NEW was seen previously, then the event is
> ignored. Maybe this is too tight, but since I'm validating the whole
> thing I prefer remaining "tight" at the moment.

I guess this could be made configurable, but I think for testing
purposes its better to be stricter so it doesn't shadow real
problems.

Re: [RFC] ctnetlink events drop benchmark

[Pablo Neira Ayuso]

Patrick McHardy wrote:
> Pablo Neira Ayuso wrote:
> 
>>Hi,
>>
>>Finally, I've got some results in my test enviroment about the ctnetlink
>>event drop issue. The machine used as firewall is a PIII 866 256 Mbytes
>>RAM, two NIC realtek 8139 100Mbits running debian unstable with linux
>>kernel 2.6.16.
>>
>>I have run the same test with three different socket queue sizes:
>>107.520 (default)
>>215.040 (x2)
>>430.080 (x4)
>>
>>I used netperf to create the connections with a shell script that loops
>>calling netperf -H ip -l 3 &.
>>
>>To get the number of events dropped I've used the following:
>>
>>$ conntrackd & (run as daemon)
>>$ conntrackd -s (request statistics)
>>
>>Results are available in:
>>
>>http://people.netfilter.org/pablo/ctnetlink/events.eps
> 
> 
> Does "events" mean events/s?

No, sorry, it seems that the graph is definitely misleading :(. Have a 
look at http://people.netfilter.org/pablo/ctnetlink/107520.txt

>>Currently nlmsg_seq is not set in ctnetlink events, if we set it, we can
>>do some kind of sequence tracking on netlink. If an event gets lost, we
>>can request a resync with the conntrack via dump_conntrack.
> 
> 
> nlmsg_seq should only (but in that case it really should) be set for
> responses to queries, so userspace can associate responses with queries.
> Drops are reported as -ENOBUFS socket errors to all multicast listeners.

Good point. Thanks.

>>Another choice could be relax conntrackd states transitions, currently
>>the valid transition sequence is: NEW -> UPDATE -> DESTROY, if an UPDATE
>>event is received but no NEW was seen previously, then the event is
>>ignored. Maybe this is too tight, but since I'm validating the whole
>>thing I prefer remaining "tight" at the moment.
> 
> I guess this could be made configurable, but I think for testing
> purposes its better to be stricter so it doesn't shadow real
> problems.

Indeed, I'm going to add a configurable parameter.

-- 
The dawn of the fourth age of Linux firewalling is coming; a time of 
great struggle and heroic deeds -- J.Kadlecsik got inspired by J.Morris

Re: [RFC] ctnetlink events drop benchmark

[Patrick McHardy]

Pablo Neira Ayuso wrote:
> Patrick McHardy wrote:
> 
>>> http://people.netfilter.org/pablo/ctnetlink/events.eps
>>
>>
>> Does "events" mean events/s?
> 
> 
> No, sorry, it seems that the graph is definitely misleading :(. Have a
> look at http://people.netfilter.org/pablo/ctnetlink/107520.txt


I can't seem to find the number of drops in that table, which I thought
this test was all about :) I did some plotting with the numbers myself,
comparing (1):(2)/(1) with (1):(3)/(1) (actually / 6.353 to scale the
initial UPDATE value down to NEW) shows that UPDATE events scale much
more linear than NEW events, which could either mean they get dropped
more for some reason or that the connections had a higher number of
packets in the runs with more connections (possibly because of more
retransmissions). I think we need the number of packets per connection
to make sense out of the data.