Re: DDoS attack causing bad effect on conntrack searches

[Eric Dumazet <eric.dumazet@gmail.com>]

Le jeudi 22 avril 2010 à 22:38 +0200, Jesper Dangaard Brouer a écrit :

> 
> I think its plausable, there is a lot of modification going on.
> Approx 40.000 deletes/sec and 40.000 inserts/sec.
> The hash bucket size is 300032, and with 80000 modifications/sec, we are 
> (potentially) changing 26.6% of the hash chains each second.
> 
> As can be seen from the graphs:
>   http://people.netfilter.org/hawk/DDoS/2010-04-12__001/list.html
> 
> Notice that primarily CPU2 is doing the 40k deletes/sec, while CPU1 is 
> caught searching...
> 
> 
> > maybe hash table has one slot :)
> 
> Guess I have to reproduce the DoS attack in a testlab (I will first have 
> time Tuesday).  So we can determine if its bad hashing or restart of the 
> search loop.
> 
> 
> The traffic pattern was fairly simple:
> 
> 200 bytes UDP packets, comming from approx 60 source IPs, going to one 
> destination IP.  The UDP destination port number was varied in the range 
> of 1 to 6000.   The source UDP port was varied a bit more, some ranging 
> from 32768 to 61000, and some from 1028 to 5000.
> 
> 

Re-reading this, I am not sure there is a real problem on RCU as you
pointed out.

With 800.000 entries, in a 300.032 buckets hash table, each lookup hit
about 3 entries (aka searches in conntrack stats)

300.000 packets/second -> 900.000 'searches' per second.

If you have four cpus all trying to insert/delete entries in //, they
all hit the central conntrack lock.

On a DDOS scenario, every packet needs to take this lock twice,
once to free an old conntrack (early drop), once to insert a new entry.

To scale this, only way would be to have an array of locks, like we have
for TCP/UDP hash tables.

I did some tests here, with a multiqueue card, flooded with 300.000
pack/second, 65.536 source IP, millions of flows, and nothing wrong
happened (but packets drops, of course)

My two cpus were busy 100%, after tweaking smp_affinities, because on
first try, irqbalance put "01" mask on both queues, so only one ksoftirq
was working, other cpu was idle :(