ipables and caching

ipables and caching

[Alex Bligh]

I have a legacy application which forwards lots of packets (router,
essentially) and uses a lot of sometimes badly written autogenerated
iptables rules (about 3,000 of them).

I am seeing on a good day high route cache efficiency. Do packets
which do not follow the slow path (i.e. cache hits) also cache
what iptables rules they hit? Nothing fancy in use bar conn_track.
If not, is there some magic wand I can use (other than a rewrite,
which has already been done but can't be used in some circumstances)
to speed up rule processing?

-- 
Alex Bligh

Re: ipables and caching

[Jan Engelhardt]

On Tuesday 2012-01-24 15:28, Alex Bligh wrote:

> I have a legacy application which forwards lots of packets (router,
> essentially) and uses a lot of sometimes badly written autogenerated
> iptables rules (about 3,000 of them).
>
> I am seeing on a good day high route cache efficiency. Do packets
> which do not follow the slow path (i.e. cache hits) also cache
> what iptables rules they hit? Nothing fancy in use bar conn_track.

Whether the route lookup was satisfied by cache or not  plays no role 
for Xtables execution.

Re: ipables and caching

[Alex Bligh]

Jan,

--On 24 January 2012 17:29:35 +0100 Jan Engelhardt <jengelh@medozas.de> 
wrote:

>> I have a legacy application which forwards lots of packets (router,
>> essentially) and uses a lot of sometimes badly written autogenerated
>> iptables rules (about 3,000 of them).
>>
>> I am seeing on a good day high route cache efficiency. Do packets
>> which do not follow the slow path (i.e. cache hits) also cache
>> what iptables rules they hit? Nothing fancy in use bar conn_track.
>
> Whether the route lookup was satisfied by cache or not  plays no role
> for Xtables execution.

Thanks. I don't suppose you know of any work on caching iptables lookups
(which I appreciate is non-trivial to impossible in the general case) or
non-linearising lookups? I am thinking of rules in the FORWARD chain which
either select by source prefix or interface (or the destination equivalent)
and if the criterion is met, jump to another rule. Currently that search in
linear (i.e. through every rule) which with a large number of
prefixes/interfaces is problematic, but with a btree or similar could be
O(log(n)), and with a hash O(1).

-- 
Alex Bligh

Re: ipables and caching

[Ed W]

On 27/01/2012 13:11, Alex Bligh wrote:
> Jan,
>
> --On 24 January 2012 17:29:35 +0100 Jan Engelhardt 
> <jengelh@medozas.de> wrote:
>
>>> I have a legacy application which forwards lots of packets (router,
>>> essentially) and uses a lot of sometimes badly written autogenerated
>>> iptables rules (about 3,000 of them).
>>>
>>> I am seeing on a good day high route cache efficiency. Do packets
>>> which do not follow the slow path (i.e. cache hits) also cache
>>> what iptables rules they hit? Nothing fancy in use bar conn_track.
>>
>> Whether the route lookup was satisfied by cache or not  plays no role
>> for Xtables execution.
>
> Thanks. I don't suppose you know of any work on caching iptables lookups
> (which I appreciate is non-trivial to impossible in the general case) or
> non-linearising lookups? I am thinking of rules in the FORWARD chain 
> which
> either select by source prefix or interface (or the destination 
> equivalent)
> and if the criterion is met, jump to another rule. Currently that 
> search in
> linear (i.e. through every rule) which with a large number of
> prefixes/interfaces is problematic, but with a btree or similar could be
> O(log(n)), and with a hash O(1).
>

I don't think so, but have you considered adding some optimisation into 
the auto generated rules? With so many rules there must surely be some 
low hanging fruit to add some conditional chains to reduce the number of 
rules hit?  ipsets are also a very useful option to collapse certain 
types of problem into fewer rules?

Good luck

Ed W

Re: ipables and caching

[Jan Engelhardt]

On Friday 2012-01-27 14:11, Alex Bligh wrote:
>
>>> I have a legacy application which forwards lots of packets (router,
>>> essentially) and uses a lot of sometimes badly written autogenerated
>>> iptables rules (about 3,000 of them).
>>>
>>> I am seeing on a good day high route cache efficiency. Do packets
>>> which do not follow the slow path (i.e. cache hits) also cache
>>> what iptables rules they hit? Nothing fancy in use bar conn_track.
>>
>> Whether the route lookup was satisfied by cache or not  plays no role
>> for Xtables execution.
>
>Thanks. I don't suppose you know of any work on caching iptables lookups

That would not quite work with e.g. matches that change depending on the
moonphase, such as -m statistic --mode nth.

>or non-linearising lookups? I am thinking of rules in the FORWARD chain which
>either select by source prefix or interface (or the destination equivalent)
>and if the criterion is met, jump to another rule.

Partly by use of ipset.