Re: [Lxc-users] Containers slow to start after 1600

[ebiederm@xmission.com (Eric W. Biederman)]

Benoit Lourdelet <blourdel@juniper.net> writes:

> Hello,
>
> The measurement has been done with kernel 3.8.2.
>
> Linux ieng-serv06 3.7.9 #3 SMP Wed Feb 27 02:38:58 PST 2013 x86_64 x86_64
> x86_64 GNU/Linux

Two different kernel versions?

> What information would you like to see on the kernel ?

The question is where is the kernel spending it's time.  So profiling
information should help us see that.  Something like.

$ cat > test-script.sh << 'EOF'
#!/bin/bash
for i in $(seq 1 2000) ; do
	ip link add a$i type veth peer name b$i
done
EOF

$ perf record -a -g test-script.sh
$ perf report 

I don't do anywhere enough work with perf to remember what good options
are.

You definititely don't want to time anything you are doing something
silly like asking ip link add to generate device names which is O(N^2)
when you create one device at a time.

And of course there is the interesting discrepency.  Why can I add 5000
veth pairs in 120 seconds and it takes you 1123 seconds.  Do you have a very
slow cpu in your test environment?  Or was your test asking the kernel
to generate names.

Once we know where the kernel is spending it's time we can look to see
if there is anything that is easy to fix, and where to point you.

Both my timing and yours indicates that there is something taking O(N^2)
time in there.  So it would at least be interesting to see what that
something is.

Eric