increase in time to delete an interface with 4.x kernels

increase in time to delete an interface with 4.x kernels

[David Ahern]

Hi Alex:

I believe you did the recent overhaul to the fib implementation. I am 
seeing dramatically higher times to delete an interface with an ipv4 
address in 4.2-rc3. perf-top points to update_suffix:

    PerfTop:   15834 irqs/sec  kernel:97.3%  exact:  0.0% [4000Hz 
cpu-clock],  (all, 4 CPUs)
-------------------------------------------------------------------------------------------

     74.69%  [kernel]       [k] update_suffix
      2.38%  [kernel]       [k] fib_table_flush
      2.20%  [kernel]       [k] fib6_walk_continue
      2.03%  [kernel]       [k] fib6_ifdown
      1.31%  [kernel]       [k] fib6_age


I have a simple script to create and assign an ipv4 address to 10k dummy 
interfaces:

l=0
for (( j = 1; j <= 40; j += 1))
do
	for (( k = 1 ; k <= 250  ; k += 1 ))
	do
		l=$((l + 1))
		ip link add dev dummy${l} type dummy
	  	ip addr add  72.$j.$k.1/24 dev dummy${l}
	  	ifconfig dummy${l} up
	done
done


and a counter script to delete them all:

k=$(ip link show | grep dummy | wc -l)
for (( j = 1; j <= k; j += 1))
do
	ip link del dev dummy${j}
done


Looking at v3.19:

# time ./tadd-dummy.sh

real    3m8.896s
user    0m7.104s
sys     0m22.020s


# time ./tdel-dummy.sh

real    7m18.207s
user    0m3.824s
sys     3m15.672s


And the time to delete 1 interface after all 10k have been created:
# time ip link del dev dummy6666

real    0m0.064s
user    0m0.000s
sys     0m0.020s


Contrast those times with 4.2.0-rc3+ running the exact same scripts

# time ./tadd-dummy.sh

real	2m51.044s
user	0m7.220s
sys	0m29.520s

#  time ip link del dev dummy6666

real	0m0.441s
user	0m0.000s
sys	0m0.416s

so here the time to delete 1 interface has gone up by more than 10x.


# time ./tdel-dummy.sh
^C

real	14m10.000s
user	0m0.528s
sys	13m14.728s

I killed the delete; after 14 minutes only ~2k+ interfaces had been deleted:

# ip link show | grep dummy | wc -l
7822

In 4.2.0-rc3 it seems to take about 60 seconds to delete 150 interfaces 
which is inline with the 1 interface time of 0.4 seconds.

David

Re: increase in time to delete an interface with 4.x kernels

[Alexander Duyck]

On 07/27/2015 09:49 AM, David Ahern wrote:
> Hi Alex:
>
> I believe you did the recent overhaul to the fib implementation. I am
> seeing dramatically higher times to delete an interface with an ipv4
> address in 4.2-rc3. perf-top points to update_suffix:
>
>     PerfTop:   15834 irqs/sec  kernel:97.3%  exact:  0.0% [4000Hz
> cpu-clock],  (all, 4 CPUs)
> -------------------------------------------------------------------------------------------
>
>
>      74.69%  [kernel]       [k] update_suffix
>       2.38%  [kernel]       [k] fib_table_flush
>       2.20%  [kernel]       [k] fib6_walk_continue
>       2.03%  [kernel]       [k] fib6_ifdown
>       1.31%  [kernel]       [k] fib6_age
>
>
> I have a simple script to create and assign an ipv4 address to 10k dummy
> interfaces:
>
> l=0
> for (( j = 1; j <= 40; j += 1))
> do
>      for (( k = 1 ; k <= 250  ; k += 1 ))
>      do
>          l=$((l + 1))
>          ip link add dev dummy${l} type dummy
>            ip addr add  72.$j.$k.1/24 dev dummy${l}
>            ifconfig dummy${l} up
>      done
> done
>
>
> and a counter script to delete them all:
>
> k=$(ip link show | grep dummy | wc -l)
> for (( j = 1; j <= k; j += 1))
> do
>      ip link del dev dummy${j}
> done
>

Okay so looking over what this script does it looks like it really 
exposes the worst case scenerio for update_suffix.  You have a monstrous 
tnode that is 15 bits ins size.  That is roughly 32K entries, and 
unfortunately the suffix is 8 bits long with a position of 7.

The result is that for every removal the code is scanning 16K entries in 
order to relevel things after an entry is removed.

Let me try a couple of quick things and I should have a patch for you in 
the next couple of hours.

Thanks.

- Alex

[net PATCH] fib_trie: Drop unnecessary calls to leaf_pull_suffix

[Alexander Duyck]

It was reported that update_suffix was taking a long time on systems where
a large number of leaves were attached to a single node.  As it turns out
fib_table_flush was calling update_suffix for each leaf that didn't have all
of the aliases stripped from it.  As a result, on this large node removing
one leaf would result in us calling update_suffix for every other leaf on
the node.

The fix is to just remove the calls to leaf_pull_suffix since they are
redundant as we already have a call in resize that will go through and
update the suffix length for the node before we exit out of
fib_table_flush or fib_table_flush_external.

Reported-by: David Ahern <dsa@cumulusnetworks.com>
Signed-off-by: Alexander Duyck <alexander.h.duyck@redhat.com>
---

This patch should apply to linux-4.1.y and newer kernels.

I've done a bit of testing on my system and I no longer see update_suffix
dominating the performance traces.  David if you can test with this patch
to see if you still see the issue I would appreciate it.

 net/ipv4/fib_trie.c |    4 ----
 1 file changed, 4 deletions(-)

diff --git a/net/ipv4/fib_trie.c b/net/ipv4/fib_trie.c
index ef90d73911de..70168ca4716b 100644
--- a/net/ipv4/fib_trie.c
+++ b/net/ipv4/fib_trie.c
@@ -1791,8 +1791,6 @@ void fib_table_flush_external(struct fib_table *tb)
 		if (hlist_empty(&n->leaf)) {
 			put_child_root(pn, n->key, NULL);
 			node_free(n);
-		} else {
-			leaf_pull_suffix(pn, n);
 		}
 	}
 }
@@ -1862,8 +1860,6 @@ int fib_table_flush(struct fib_table *tb)
 		if (hlist_empty(&n->leaf)) {
 			put_child_root(pn, n->key, NULL);
 			node_free(n);
-		} else {
-			leaf_pull_suffix(pn, n);
 		}
 	}
 

Re: [net PATCH] fib_trie: Drop unnecessary calls to leaf_pull_suffix

[David Ahern]

On 7/27/15 2:08 PM, Alexander Duyck wrote:
> It was reported that update_suffix was taking a long time on systems where
> a large number of leaves were attached to a single node.  As it turns out
> fib_table_flush was calling update_suffix for each leaf that didn't have all
> of the aliases stripped from it.  As a result, on this large node removing
> one leaf would result in us calling update_suffix for every other leaf on
> the node.
>
> The fix is to just remove the calls to leaf_pull_suffix since they are
> redundant as we already have a call in resize that will go through and
> update the suffix length for the node before we exit out of
> fib_table_flush or fib_table_flush_external.
>
> Reported-by: David Ahern<dsa@cumulusnetworks.com>
> Signed-off-by: Alexander Duyck<alexander.h.duyck@redhat.com>
> ---
>
> This patch should apply to linux-4.1.y and newer kernels.
>
> I've done a bit of testing on my system and I no longer see update_suffix
> dominating the performance traces.  David if you can test with this patch
> to see if you still see the issue I would appreciate it.
>

Works for me. Thanks.

Tested-by: David Ahern <dsa@cumulusnetworks.com>

Re: [net PATCH] fib_trie: Drop unnecessary calls to leaf_pull_suffix

[David Miller]

From: Alexander Duyck <alexander.h.duyck@redhat.com>
Date: Mon, 27 Jul 2015 13:08:06 -0700

> It was reported that update_suffix was taking a long time on systems where
> a large number of leaves were attached to a single node.  As it turns out
> fib_table_flush was calling update_suffix for each leaf that didn't have all
> of the aliases stripped from it.  As a result, on this large node removing
> one leaf would result in us calling update_suffix for every other leaf on
> the node.
> 
> The fix is to just remove the calls to leaf_pull_suffix since they are
> redundant as we already have a call in resize that will go through and
> update the suffix length for the node before we exit out of
> fib_table_flush or fib_table_flush_external.
> 
> Reported-by: David Ahern <dsa@cumulusnetworks.com>
> Signed-off-by: Alexander Duyck <alexander.h.duyck@redhat.com>

Applied and queued up for -stable, thanks.