early_drop() not working correctly?

early_drop() not working correctly?

[Menno Smits]

Hi list,

At one customer site we have a situation where a netfilter box sees half 
  of some connections due to asymmetric routing. This causes lots of 
ESTABLISHED but unreplied connections to fill the conntrack table with 
long (5 day) timeouts. Eventually the table completely fills up. The 
"nf_conntrack: table full, dropping packet" message is reported, packets 
  are dropped and the customer starts complaining.

According to discussions with Patrick McHardy off list and my own 
examinations of the code, early_drop() should free up some of the 
conntrack table when the table is full by removing unreplied 
connections. In practice, this doesn't actually happen.

I've experimented with a similar setup in the office and I can 
consistently replicate the problem. I'm happy to describe my test setup 
if anyone is interested (it's a little complicated).

Can someone explain why the unreplied connections aren't being removed 
from the conntrack table? Is there a bug here?

Regards,
Menno

ps. We have worked around the issue at the client's site by using the 
NOTRACK target for the asymmetrically routed traffic. Turning off TCP 
connection pickup also works around the problem.

Re: early_drop() not working correctly?

[Martijn Lievaart]

<citaat van="Menno Smits">
>
> ps. We have worked around the issue at the client's site by using the
> NOTRACK target for the asymmetrically routed traffic. Turning off TCP
> connection pickup also works around the problem.

IMHO that's not the workaround, that's how it should be setup. That does
not mean the bug is not real.

HTH,
M4

Re: early_drop() not working correctly?

[Patrick McHardy]

Menno Smits wrote:
> Hi list,
> 
> At one customer site we have a situation where a netfilter box sees half
>  of some connections due to asymmetric routing. This causes lots of
> ESTABLISHED but unreplied connections to fill the conntrack table with
> long (5 day) timeouts. Eventually the table completely fills up. The
> "nf_conntrack: table full, dropping packet" message is reported, packets
>  are dropped and the customer starts complaining.
> 
> According to discussions with Patrick McHardy off list and my own
> examinations of the code, early_drop() should free up some of the
> conntrack table when the table is full by removing unreplied
> connections. In practice, this doesn't actually happen.
> 
> I've experimented with a similar setup in the office and I can
> consistently replicate the problem. I'm happy to describe my test setup
> if anyone is interested (it's a little complicated).

I suggest adding some debugging printks to early_drop that dump the
other members of the hash chain and their flag values (epecially
IPS_ASSURED). Or simply to unreplied(). That should exlain what's
going on.

Re: early_drop() not working correctly?

[Jozsef Kadlecsik]

On Thu, 6 Jul 2006, Menno Smits wrote:

> At one customer site we have a situation where a netfilter box sees half
>   of some connections due to asymmetric routing.

As it was already written, conntrack cannot reliably work in such setups.

> This causes lots of ESTABLISHED but unreplied connections to fill the
> conntrack table with long (5 day) timeouts. Eventually the table
> completely fills up. The "nf_conntrack: table full, dropping packet"
> message is reported, packets
>   are dropped and the customer starts complaining.
>
> According to discussions with Patrick McHardy off list and my own
> examinations of the code, early_drop() should free up some of the
> conntrack table when the table is full by removing unreplied
> connections.

If you have got a pretty flat hash table (large hashsize parameter), then
early_drop has got bad chance to find any unreplied connection in the
given bucket. If you set hashsize relatively small (compared to the usual
traffic) then early_drop has got better chances but it hurts the
performance. I believe there is only bad and worse solutions to your
problem.

> In practice, this doesn't actually happen.

Sorry, but I think there should be more facts which supports your
assumption.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary