iptables modules efficiency, caching in conntrack

iptables modules efficiency, caching in conntrack

[Amin Azez]

I've done work with a few iptables modules, conntrack, ipt_recent,
ipt_account (recently) and others.

The design usually consists of a module specific storage using some
aspects of the connection or packet as a key to the storage.

For each packet, each module calculates the key, fetches and locks the
storage record, makes decisions and updates, unlocks and continues.

I also notice a similarity between many modules, ipt_set, ipt_recent,
ipt_account, and think how much space and time is used less efficiently
when more than one of these are used together.

This is tending me to merge functionality. I have just merged connrate
with ipt_account. (Expect patches soon.)

Arguably rate limiting is a function of per-ip accounting, arguably it
is not; however my point is this:

The one main storage element which is already looked up is conntrack.
If the conntrack record had an array of extra (ref-counted pointers)
then other modules would just need calculate keys and create/lookup
records from their own storage once, and keep a reference in the conntrack.

Slots in this conntrack array would be allocated, so a module might get
to keep slot 0, and the next module to request a slot would get slot 1, etc.

Let these slots be dynamicly growable, or use some other means, it
doesn't matter as long as it is faster than each module looking up its
storage the long way around.

This gives us faster storage lookups for other modules, which is important.

Its a small step from there to provide callbacks for each module when
the conntrack is destroyed, in case the associated per-module structure
needs destroying.

If beneficial, its also a small step for one module to tag extra data on
to another modules storage too; ipt_account is practically a superset of
ipt_set anyway (except in the meaningful way of actually saying if an ip
is in the set, but its nearly there...)

Currently the closest we get to this is hard-wired if-def's to add slots
to a conntrack so that they CAN be used at runtime if needed.

So this suggestion would speed up many modules, as well as potentially
reduce the size of the conntrack struct.

Comments?

Sam

Re: iptables modules efficiency, caching in conntrack

[Carl-Daniel Hailfinger]

Hi Amin,

Amin Azez wrote:
> I've done work with a few iptables modules, conntrack, ipt_recent,
> ipt_account (recently) and others.
> 
> The design usually consists of a module specific storage using some
> aspects of the connection or packet as a key to the storage.
> 
> For each packet, each module calculates the key, fetches and locks the
> storage record, makes decisions and updates, unlocks and continues.
> 
> I also notice a similarity between many modules, ipt_set, ipt_recent,
> ipt_account, and think how much space and time is used less efficiently
> when more than one of these are used together.

Have you looked at ipt_ACCOUNT (the capitalized one)? IIRC their storage
model is quite efficient and works fine even under memory pressure and
requires only 0-order-allocations.

If you want, I can supply you with a 64-bit-clean kernel-2.6 version
of ipt_ACCOUNT.

Regards,
Carl-Daniel
-- 
http://www.hailfinger.org/