Re: [Lxc-users] Containers slow to start after 1600

[Serge Hallyn <serge.hallyn@ubuntu.com>]

Hi,

Benoit was kind enough to follow up on some scalability issues with
larger (but not huge imo) numbers of containers.  Running a script
to simply time the creation of veth pairs on a rather large (iiuc)
machine, he got the following numbers (time is for creation of the
full number, not latest increment - so 1123 seconds to create 5000
veth pairs)

> >Quoting Benoit Lourdelet (blourdel@juniper.net):
> >> Hello Serge,
> >> 
> >> I put together a small table, running your script for various values :
> >> 
> >> Time are in seconds,
> >> 
> >> Number of veth, time to create, time to delete:
> >> 
> >> 500  18  26
> >> 
> >> 1000 57 70
> >> 
> >> 2000 193 250
> >> 
> >> 3000 435 510
> >> 
> >> 4000 752  824
> >> 
> >> 5000 1123  1185

> 
> Benoit

Ok.  Ran some tests on a tiny cloud instance.  When I simply run 2k tasks in
unshared new network namespaces, it flies by.

#!/bin/sh
rm -f /tmp/timings3
date | tee -a /tmp/timings3
for i in `seq 1 2000`; do
    nsexec -n -- /bin/sleep 1000 &
    if [ $((i % 100)) -eq 0 ]; then
           echo $i | tee -a /tmp/timings3
           date | tee -a /tmp/timings3
    fi
done

(all scripts run under sudo, and nsexec can be found at
https://code.launchpad.net/~serge-hallyn/+junk/nsexec))

So that isn't an issue.

When I run a script to just time veth pair creations like Benoit ran,
creating 2000 veth pairs and timing the results for each 100, the time
does degrade, from 1 second for the first 100 up to 8 seconds for the
last 100.

(that script for me is:

#!/bin/sh
rm -f /tmp/timings
for i in `seq 1 2000`; do
       ip link add type veth
       if [ $((i % 100)) -eq 0 ]; then
               echo $i | tee -a /tmp/timings
               date | tee -a /tmp/timings
               ls /sys/class/net > /dev/null
       fi
done
)

But when I actually pass veth instances to those unshared network
namespaces:

#!/bin/sh
rm -f /tmp/timings2
echo 0 | tee -a /tmp/timings2
date | tee -a /tmp/timings2
for i in `seq 1 2000`; do
    nsexec -n -P /tmp/pid.$i -- /bin/sleep 1000 &
    ip link add type veth
    dev2=`ls -d /sys/class/net/veth* | tail -1`
    dev=`basename $dev2`
    pid=`cat /tmp/pid.$i`
    ip link set $dev netns $pid
    if [ $((i % 100)) -eq 0 ]; then
           echo $i | tee -a /tmp/timings2
           date | tee -a /tmp/timings2
    fi
    rm -f /tmp/pid.*
done

it goes from 4 seconds for the first hundred to 16 seconds for
the last hundred - a worse regression than simply creating the
veths.  Though I guess that could be accounted for simply by
sysfs actions when a veth is moved from the old netns to the
new?

0
Tue Mar 19 20:15:26 UTC 2013
100
Tue Mar 19 20:15:30 UTC 2013
200
Tue Mar 19 20:15:35 UTC 2013
300
Tue Mar 19 20:15:41 UTC 2013
400
Tue Mar 19 20:15:47 UTC 2013
500
Tue Mar 19 20:15:54 UTC 2013
600
Tue Mar 19 20:16:02 UTC 2013
700
Tue Mar 19 20:16:09 UTC 2013
800
Tue Mar 19 20:16:17 UTC 2013
900
Tue Mar 19 20:16:26 UTC 2013
1000
Tue Mar 19 20:16:35 UTC 2013
1100
Tue Mar 19 20:16:46 UTC 2013
1200
Tue Mar 19 20:16:57 UTC 2013
1300
Tue Mar 19 20:17:08 UTC 2013
1400
Tue Mar 19 20:17:21 UTC 2013
1500
Tue Mar 19 20:17:33 UTC 2013
1600
Tue Mar 19 20:17:46 UTC 2013
1700
Tue Mar 19 20:17:59 UTC 2013
1800
Tue Mar 19 20:18:13 UTC 2013
1900
Tue Mar 19 20:18:29 UTC 2013
2000
Tue Mar 19 20:18:48 UTC 2013

-serge