Re: RCU related performance regression in 3.3

[Pascal Chapperon <pascal.chapperon@wanadoo.fr>]

Le 18/04/2012 17:23, Paul E. McKenney a écrit :
> On Wed, Apr 18, 2012 at 05:00:14PM +0200, Pascal Chapperon wrote:
>> Le 18/04/2012 16:01, Paul E. McKenney a écrit :
>>> On Wed, Apr 18, 2012 at 11:37:28AM +0200, Pascal Chapperon wrote:
>>>> Mount and umount operations are not slower with RCU_FAST_NO_HZ during
>>>> runtime; systemctl start and stop operations are also not slower. In
>>>> fact, i couldn't find a single operation slower during runtime with
>>>> RCU_FAST_NO_HZ.
>>>
>>> Your boot-time setup is such that all CPUs are online before the
>>> boot-time mount operations take place, right?
>> Yes :
>> [ 0.242697] Brought up 8 CPUs
>> [ 0.242699] Total of 8 processors activated (35118.33 BogoMIPS).
>>
>>> Struggling to understand
>>> how RCU can tell the difference between post-CPU-bringup boot time
>>> and run time...
>>>
>> systemd is controlling the whole boot process including mount
>> operation (apart root filesystem) and as I can see, uses heavily
>> sockets to do it (not to mention cpu-affinity). It controls also the
>> major part of umount operations. Is it possible that your patch hits
>> a systemd bug ?
> 
> Is it possible that systemd is using network operations that include
> synchronize_rcu()? Then if you did the same operation from the
> command line at runtime, you might not see the slowdown.
> 
> Is it possible for you to convince systemd to collect RCU event tracing
> during the slow operation? RCU event tracing is available under
> /sys/kernel/debug/tracing/rcu.
>
.
I have collected the RCU event tracing during a slow boot with
FAST_NO_HZ (and the same without FAST_NO_HZ, same kernel config).
The full logs and associated "systemd-analyze plot" can be found
(in comment 32) at :

https://bugzilla.redhat.com/show_bug.cgi?id=806548

With FAST_NO_HZ, almost each rcu_prep_idle is followed by ksoftirqd
(75000 ksoftirqd lines with FAST_NO_HZ, 4000 without).

Sorry, the logs are very huge, but I can't figure where are the 
plots of some interest. 

Pascal