Re: [Lxc-users] Containers slow to start after 1600

[ebiederm@xmission.com (Eric W. Biederman)]

Benoit Lourdelet <blourdel@juniper.net> writes:

> Hello,
>
> I tried multiple kernels with similar results.
>
> I ran the following on 3.8.2
>
> $ cat > test-script.sh << 'EOF'
> #!/bin/bash
> for i in $(seq 1 2000) ; do
> 	ip link add a$i type veth peer name b$i
> done
> EOF
>
> $ perf record -a test-script.sh
> $ perf report
>

Interesting.  And the snmp_fold_field is an interesting hint.

It looks like something is listing all of your interfaces probably
between interface creation.  And that will definitely take this
process from O(NlogN) to O(N^2).

Although I find the find_next_bit call also concerning.

snmp_fold_field is used when reading /proc/net/snmp /proc/net/netstat
and when running ip link to show all of the interfaces.

I suspect find_next_bit is coming out of the per cpu allocator, and it
might be partially responsible.  But that doesn't feel right.

Regardless from this trace it looks like the performance problem on
creation is something in userspace is calling /sbin/ip to list all of
the interfaces and that is  causing the interface creation to slow down.


Oh I see what is happening.  In iproute.
do_iplink
  iplink_modify
     ll_init_map

So the limitation at this point is in iproute, and not in the kernel.

And with the following patch the time to create 5000 devices drops from
120 to 11 seconds for me.  So it looks like the kernel is fine it is
whatever userspace tools that are of interest that are the bottleneck.

Although a quicker way to do a name to ifindex mapping might be
interesting.

Eric


diff --git a/ip/iplink.c b/ip/iplink.c
index ad33611..58af369 100644
--- a/ip/iplink.c
+++ b/ip/iplink.c
@@ -533,8 +533,6 @@ static int iplink_modify(int cmd, unsigned int flags, int argc, char **argv)
                }
        }
 
-       ll_init_map(&rth);
-
        if (!(flags & NLM_F_CREATE)) {
                if (!dev) {
                        fprintf(stderr, "Not enough information: \"dev\" "
@@ -542,6 +540,7 @@ static int iplink_modify(int cmd, unsigned int flags, int argc, char **argv)
                        exit(-1);
                }
 
+               ll_init_map(&rth);
                req.i.ifi_index = ll_name_to_index(dev);
                if (req.i.ifi_index == 0) {
                        fprintf(stderr, "Cannot find device \"%s\"\n", dev);
@@ -555,6 +554,7 @@ static int iplink_modify(int cmd, unsigned int flags, int argc, char **argv)
                if (link) {
                        int ifindex;
 
+                       ll_init_map(&rth);
                        ifindex = ll_name_to_index(link);
                        if (ifindex == 0) {
                                fprintf(stderr, "Cannot find device \"%s\"\n",