netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Scalability of interface creation and deletion
@ 2011-05-07 11:08 Alex Bligh
  2011-05-07 12:22 ` Eric Dumazet
  0 siblings, 1 reply; 53+ messages in thread
From: Alex Bligh @ 2011-05-07 11:08 UTC (permalink / raw)
  To: netdev; +Cc: Alex Bligh

I am trying to track down why interface creation slows down badly with
large numbers of interfaces (~1,000 interfaces) and why deletion is so
slow. Use case: restarting routers needs to be fast; some failover methods
require interface up/down; some routers need lots of interfaces.

I have written a small shell script to create and delete a number of
interfaces supplied on the command line (script appended below). It
is important to run this with udev, udev-bridge etc. disabled. In
my environment
(Ubuntu 2.6.32-28-generic, Lucid). I did this by
 * service upstart-udev-bridge stop
 * service udev stop
 * unshare -n bash
If you don't do this, you are simply timing your distro's interface
scripts.

Note the "-n" parameter creates the supplied number of veth pair
interfaces. As these are pairs, there are twice as many interfaces actually
created.

So, the results which are pretty repeatable are as follows:

                            100 pairs      500 pairs
Interface creation               14ms          110ms
Interface deletion              160ms          148ms

Now I don't think interface deletion has in fact got faster: simply
the overhead of loading the script is spread over more processes.
But there are two obvious conclusions:

1. Interface creation slows down hugely with more interfaces
2. Interface deletion is normally much slower than interface creation

strace -T -ttt on the "ip" command used to do this does not show the delay
where I thought it would be - cataloguing the existing interfaces. Instead,
it's the final send() to the netlink socket which does the relevant action
which appears to be slow, for both addition and detion. Adding the last
interface takes 200ms in that syscall, the first is quick (symptomatic of a
slowdown); for deletion the last send syscall is quick.

Poking about in net/core/dev.c, I see that interface names are hashed using
a hash with a maximum of 256 entries. However, these seem to be hash
buckets supporting multiple entries so I can't imagine a chain of 4 entries
is problematic.

I am having difficulty seeing what might be the issue in interface
creation. Any ideas?

In interface deletion, my attention is drawn to netdev_wait_allrefs,
which does this:
        refcnt = netdev_refcnt_read(dev);

        while (refcnt != 0) {
                ...
                msleep(250);

                refcnt = netdev_refcnt_read(dev);
		....
        }

I am guessing that this is going to do the msleep 50% of the time,
explaining 125ms of the observed time. How would people react to
exponential backoff instead (untested):

	int backoff = 10;
        refcnt = netdev_refcnt_read(dev);

        while (refcnt != 0) {
                ...
                msleep(backoff);
                if ((backoff *= 2) > 250)
                  backoff = 250;
		
                refcnt = netdev_refcnt_read(dev);
		....
        }


-- 
Alex Bligh



#!/bin/bash

# Usage:
#   ifaceseq [options]
#
# Options:
#   -n NUM : use NUM interfaces
#   -t TYPE : use TYPE of interfaces (supported: veth, vlan)

numifs=10
itype=veth

while getopts n:t: flag; do
    case ${flag} in
	n) numifs=${OPTARG} ;;
	t) itype=${OPTARG} ;;
    esac
done

shift $((OPTIND-1))

createifs ()
{
    echo `date` creating $numifs interfaces
    case ${itype} in
	vlan)
	    for i in `seq 1 $numifs` ; do
		ip link add link eth0 name vlan${i} type vlan id ${i}
	    done
	    ;;
	*)
	    for i in `seq 1 $numifs` ; do
		ip link add testa${i} type veth peer name testb${i}
	    done
    esac
    echo `date` done
}

deleteifs ()
{
    echo `date` deleting $numifs interfaces
    case ${itype} in
	vlan)
	    for i in `seq 1 $numifs` ; do
		ip link delete dev vlan${i}
	    done
	    ;;
	*)
	    for i in `seq 1 $numifs` ; do
		ip link delete testa${i}
	    done
    esac
    echo `date` done
}

time createifs;
time deleteifs;




^ permalink raw reply	[flat|nested] 53+ messages in thread
end of thread, other threads:[~2011-05-10 19:23 UTC | newest]

Thread overview: 53+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-05-07 11:08 Scalability of interface creation and deletion Alex Bligh
2011-05-07 12:22 ` Eric Dumazet
2011-05-07 15:26   ` Alex Bligh
2011-05-07 15:54     ` Eric Dumazet
2011-05-07 16:23       ` Ben Greear
2011-05-07 16:37         ` Eric Dumazet
2011-05-07 16:44           ` Ben Greear
2011-05-07 16:51             ` Eric Dumazet
2011-05-08  3:45               ` Ben Greear
2011-05-08  8:08                 ` Alex Bligh
2011-05-09 21:46       ` Octavian Purdila
2011-05-07 16:26     ` Eric Dumazet
2011-05-07 18:24       ` Alex Bligh
2011-05-07 18:32         ` Eric Dumazet
2011-05-07 18:39           ` Eric Dumazet
2011-05-08 10:09             ` Alex Bligh
2011-05-07 18:42           ` Eric Dumazet
2011-05-07 18:50             ` Alex Bligh
2011-05-08  7:12             ` Eric Dumazet
2011-05-08  8:06               ` Alex Bligh
2011-05-08  9:35               ` Alex Bligh
2011-05-08 12:18                 ` Alex Bligh
2011-05-08 12:50                   ` Paul E. McKenney
2011-05-08 13:13                     ` Alex Bligh
2011-05-08 13:44                       ` Paul E. McKenney
2011-05-08 14:27                         ` Alex Bligh
2011-05-08 14:47                           ` Paul E. McKenney
2011-05-08 15:17                             ` Alex Bligh
2011-05-08 15:48                               ` Paul E. McKenney
2011-05-08 21:00                                 ` Eric Dumazet
2011-05-09  4:44                                   ` [PATCH] veth: use batched device unregister Eric Dumazet
2011-05-09  6:56                                     ` Michał Mirosław
2011-05-09  8:20                                       ` Eric Dumazet
2011-05-09  9:17                                         ` [PATCH net-next-2.6] net: use batched device unregister in veth and macvlan Eric Dumazet
2011-05-09 18:42                                           ` David Miller
2011-05-09 19:05                                             ` Eric Dumazet
2011-05-09 20:17                                               ` Eric Dumazet
2011-05-10  6:40                                                 ` [PATCH net-2.6] vlan: fix GVRP at dismantle time Eric Dumazet
2011-05-10 19:23                                                   ` David Miller
2011-05-09  7:45                                     ` [PATCH v2 net-next-2.6] veth: use batched device unregister Eric Dumazet
2011-05-09  9:22                                       ` Eric Dumazet
2011-05-09  5:37                                   ` Scalability of interface creation and deletion Alex Bligh
2011-05-09  6:37                                     ` Eric Dumazet
2011-05-09  7:11                                   ` Paul E. McKenney
2011-05-09 17:30                                   ` Jesse Gross
2011-05-08 12:44                 ` Paul E. McKenney
2011-05-08 13:06                   ` Alex Bligh
2011-05-08 13:14                     ` Alex Bligh
2011-05-08 12:32               ` Paul E. McKenney
2011-05-07 18:51           ` Alex Bligh
2011-05-07 19:24             ` Eric Dumazet
2011-05-07 18:38       ` Alex Bligh
2011-05-07 18:44         ` Eric Dumazet

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).