Re: Scalability of interface creation and deletion

["Paul E. McKenney" <paulmck@linux.vnet.ibm.com>]

On Sun, May 08, 2011 at 03:27:07PM +0100, Alex Bligh wrote:
> Paul,
> 
> >>Yes, really 20-49us and 50-99us, not ms. Raw data attached :-)
> >>
> >>I'm guessing there are circumstances where there is an early exit.
> >
> >Well, if you were onlining and offlining CPUs, then if there was only
> >one CPU online, this could happen.
> 
> No, I wasn't doing that.

OK.

> > And there really is only one CPU
> >online during boot, so if your measurements included early boot time,
> >this could easily explain these very short timings.
> 
> No, I waited a few minutes after boot for the system to stabilize, and
> all CPUs were definitely online.
> 
> The patch to the kernel I am running is below.

OK, interesting...

My guess is that you need to be using ktime_get_ts().  Isn't ktime_get()
subject to various sorts of adjustment?

> >>There is nothing much going on these systems (idle, no other users,
> >>just normal system daemons).
> >
> >And normal system daemons might cause this, right?
> 
> Yes. Everything is normal, except I did
> service udev stop
> unshare -n bash
> which together stop the system running interface scripts when
> interfaces are created (as upstart and upstart-udev-bridge are
> now integrated, you can't kill upstart, so you have to rely on
> unshare -n to stop the events being propagated). That's just
> to avoid measuring the time it takes to execute the scripts.

OK, so you really could be seeing grace periods started by these system
daemons.

							Thanx, Paul

> -- 
> Alex Bligh
> 
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index dd4aea8..e401018 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -1518,6 +1518,7 @@ EXPORT_SYMBOL_GPL(call_rcu_bh);
> void synchronize_sched(void)
> {
>        struct rcu_synchronize rcu;
> +       ktime_t time_start = ktime_get();
> 
>        if (rcu_blocking_is_gp())
>                return;
> @@ -1529,6 +1530,7 @@ void synchronize_sched(void)
>        /* Wait for it. */
>        wait_for_completion(&rcu.completion);
>        destroy_rcu_head_on_stack(&rcu.head);
> +       pr_err("synchronize_sched() in %lld us\n",
> ktime_us_delta(ktime_get(), time_start));
> }
> EXPORT_SYMBOL_GPL(synchronize_sched);
> 
> diff --git a/net/core/dev.c b/net/core/dev.c
> index 856b6ee..013f627 100644
> --- a/net/core/dev.c
> +++ b/net/core/dev.c
> @@ -5164,7 +5164,9 @@ static void rollback_registered_many(struct
> list_head *head)
>        dev = list_first_entry(head, struct net_device, unreg_list);
>        call_netdevice_notifiers(NETDEV_UNREGISTER_BATCH, dev);
> 
> +       pr_err("begin rcu_barrier()\n");
>        rcu_barrier();
> +       pr_err("end rcu_barrier()\n");
> 
>        list_for_each_entry(dev, head, unreg_list)
>                dev_put(dev);
> @@ -5915,8 +5917,10 @@ EXPORT_SYMBOL(free_netdev);
>  */
> void synchronize_net(void)
> {
> +       pr_err("begin synchronize_net()\n");
>        might_sleep();
>        synchronize_rcu();
> +       pr_err("end synchronize_net()\n");
> }
> EXPORT_SYMBOL(synchronize_net);
> 
>